Secrets in AWS ECS Fargate

There are many ways to use ‘secrets’ in ECS and ECS Fargate. Most of the time they are exposed using Environment variables, because a few years ago this was the only easy option. Today we have two improved options. You could add a tool to your docker container which retrieves and decrypts your secrets before parsing them to the application. The other one is to embed the retrieval and decryption in the application. Both approaches are covered in this blog post. This is what we are going to do:

  1. Put a secret in the SSM Parameter Store
  2. Add the ssm-env tool to your Dockerfile to replace secrets at ‘boot time’
  3. Add some logic to your application to retrieve secrets
  4. Add a reference to the secret in container environment variables using CloudFormation

1. Put a secret in the SSM Parameter Store

This time we add the secret manually, for example in case of a third party api key shared with you. You could also generate a secret using the CloudFormation extension, for example when you want to generate a secret which nobody should know, even you. (Do not forget to restrict access to the SSM Parameter Store.)

aws ssm put-parameter \
  --name my.little.secret \
  --type SecureString \
  --value "th15-1s-53cur3"

2. Add the ssm-env tool to your app

Now add the following code to your Dockerfile. It will download and validate the ssm-env tool written in golang. To access the AWS APIs like SSM, you need to install certificates. Check out the complete dockerfile here.

<span style="font-weight:bold">FROM</span><span style="color:#b84"> alpine</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#998;font-style:italic"># SSM Secret Sauce</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="font-weight:bold">RUN</span> wget -O /usr/local/bin/ssm-env https://github.com/remind101/ssm-env/releases/download/v0.0.3/ssm-env<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="font-weight:bold">RUN</span> <span style="color:#999">echo</span> <span style="color:#b84">"c944fc169d860a1079e90b03a8ea2c71f749e1265e3c5b66f57b2dc6e0ef84f8  /usr/local/bin/ssm-env"</span> | sha256sum -c -<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="font-weight:bold">RUN</span> chmod +x /usr/local/bin/ssm-env<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="font-weight:bold">RUN</span> apk add —no-cache ca-certificates<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="font-weight:bold">ENTRYPOINT</span> [<span style="color:#b84">"/usr/local/bin/ssm-env"</span>, <span style="color:#b84">"-with-decryption"</span>]<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#998;font-style:italic"># /SSM Secret Sauce</span><span style="color:#a61717;background-color:#e3d2d2">
</span>

It will iterate all environment variables and search for the ssm:// prefix. When found, it will use the IAM role of the container to retrieve the secret and replaces the value for the secret. For example:

MY_LITTLE_SECRET=ssm://my.secret
# becomes within the container
MY_LITTLE_SECRET=th15-1s-53cur3

3. Add some logic to your application to retrieve secrets

In this example the public Docker image: mvandongen/alittlesecret is used, which is just a very simple Python Flask application. The first section will retrieve the SSM secret using the SDK. The secret in the second part is already decrypted. This is probably used for (legacy) applications or applications that you cannot easily change.

<span style="font-weight:bold">from</span> <span style="color:#555">flask</span> <span style="font-weight:bold">import</span> Flask
<span style="font-weight:bold">import</span> <span style="color:#555">boto3</span>
<span style="font-weight:bold">import</span> <span style="color:#555">os</span>

app <span style="font-weight:bold">=</span> Flask(__name__)

<span style="font-weight:bold">def</span> <span style="color:#900;font-weight:bold">ssm_secret</span>(path, prefix <span style="font-weight:bold">=</span> <span style="color:#b84"></span><span style="color:#b84">"</span><span style="color:#b84">ssm_sdk://</span><span style="color:#b84">"</span>):
    client <span style="font-weight:bold">=</span> boto3<span style="font-weight:bold">.</span>client(<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">ssm</span><span style="color:#b84">'</span>) <span style="color:#998;font-style:italic"># uses the container role</span>
    prefix_len <span style="font-weight:bold">=</span> <span style="color:#999">len</span>(prefix)
    <span style="font-weight:bold">if</span> os<span style="font-weight:bold">.</span>environ[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">SECRET_SDK</span><span style="color:#b84">'</span>][<span style="color:#099">0</span>:prefix_len] <span style="font-weight:bold">==</span> prefix:
        result <span style="font-weight:bold">=</span> client<span style="font-weight:bold">.</span>get_parameter(
            Name<span style="font-weight:bold">=</span>os<span style="font-weight:bold">.</span>environ[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">SECRET_SDK</span><span style="color:#b84">'</span>][prefix_len:],
            WithDecryption<span style="font-weight:bold">=</span>True
        )
        secret <span style="font-weight:bold">=</span> result[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">Parameter</span><span style="color:#b84">'</span>][<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">Value</span><span style="color:#b84">'</span>]
    <span style="font-weight:bold">else</span>: 
        secret <span style="font-weight:bold">=</span> os<span style="font-weight:bold">.</span>environ[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">SECRET_SDK</span><span style="color:#b84">'</span>]
    <span style="font-weight:bold">return</span> secret

@app.route(<span style="color:#b84"></span><span style="color:#b84">"</span><span style="color:#b84">/</span><span style="color:#b84">"</span>)
<span style="font-weight:bold">def</span> <span style="color:#900;font-weight:bold">hello</span>():
    <span style="color:#998;font-style:italic"># section 1: get the secret and decrypt it using the sdk</span>
    secret <span style="font-weight:bold">=</span> ssm_secret(
        os<span style="font-weight:bold">.</span>environ[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">SECRET_SDK</span><span style="color:#b84">'</span>],
        os<span style="font-weight:bold">.</span>environ[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">SECRET_SDK_PREFIX</span><span style="color:#b84">'</span>]
    )
    section1 <span style="font-weight:bold">=</span> <span style="color:#b84"></span><span style="color:#b84">"</span><span style="color:#b84">SDK decrypted: </span><span style="color:#b84">"</span> <span style="font-weight:bold">+</span> secret
    <span style="color:#998;font-style:italic"># section 2: get the already decrypted secret</span>
    section2 <span style="font-weight:bold">=</span> <span style="color:#b84"></span><span style="color:#b84">"</span><span style="color:#b84">ssn-env decrypted: </span><span style="color:#b84">"</span> <span style="font-weight:bold">+</span> os<span style="font-weight:bold">.</span>environ[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">SECRET</span><span style="color:#b84">'</span>]
    <span style="font-weight:bold">return</span> section1 <span style="font-weight:bold">+</span> section2

app<span style="font-weight:bold">.</span>run(host<span style="font-weight:bold">=</span><span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">0.0.0.0</span><span style="color:#b84">'</span>, port<span style="font-weight:bold">=</span><span style="color:#099">80</span>)

4. Add a reference to the secret in container environment variables using CloudFormation

Let’s deploy this Fargate container using CloudFormation. It works well in any public VPC, also the default VPC.

<span style="font-weight:bold">AWSTemplateFormatVersion</span>:<span style="color:#bbb"> </span><span style="color:#b84">"2010-09-09"</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="font-weight:bold">content</span>:<span style="color:#bbb"> </span><span style="color:#b84">>
</span><span style="color:#b84"> </span><span style="color:#b84"> </span><span style="color:#b84">"AWS Example Setup For Secrets in ECS / ECS Fargate"</span><span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="font-weight:bold">Parameters</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">VpcId</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::EC2::VPC::Id<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">content</span>:<span style="color:#bbb"> </span><span style="color:#b84">>
</span><span style="color:#b84">     </span><span style="color:#b84"> </span><span style="color:#b84">"ID of the VPC to deploy the containers to."</span><span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">Subnets</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>List<aws::EC2::Subnet::Id<span style="color:#b84">>
</span><span style="color:#b84">   </span><span style="color:#b84"> </span><span style="color:#b84">content: ></span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="color:#b84">"IDs of the Subnets to deploy the containers to."</span><span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="font-weight:bold">Resources</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">Cluster</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::ECS::Cluster<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb"> </span>{}<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">Service</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::ECS::Service<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">Cluster</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span>Cluster<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">TaskDefinition</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span><span style="color:#b84">'TaskDefinition'</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">DesiredCount</span>:<span style="color:#bbb"> </span><span style="color:#099">1</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">LaunchType</span>:<span style="color:#bbb"> </span><span style="color:#b84">'FARGATE'</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">NetworkConfiguration</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span><span style="font-weight:bold">AwsvpcConfiguration</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">          </span><span style="font-weight:bold">Subnets</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span><span style="color:#b84">'Subnets'</span><span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">SecurityGroups</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span>- !Ref<span style="color:#bbb"> </span>SecurityGroup<span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">AssignPublicIp</span>:<span style="color:#bbb"> </span><span style="color:#b84">'ENABLED'</span><span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">TaskDefinition</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::ECS::TaskDefinition<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">NetworkMode</span>:<span style="color:#bbb"> </span><span style="color:#b84">'awsvpc'</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">Cpu</span>:<span style="color:#bbb"> </span><span style="color:#b84">'256'</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">Memory</span>:<span style="color:#bbb"> </span><span style="color:#b84">'512'</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">ExecutionRoleArn</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span>TaskRole<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">TaskRoleArn</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span>TaskRole<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">RequiresCompatibilities</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span>- FARGATE<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">ContainerDefinitions</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span><span style="color:#b84">-
</span><span style="color:#b84">         </span><span style="color:#b84"> </span><span style="color:#b84">Name: "asecretservice"</span><span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">Image</span>:<span style="color:#bbb"> </span><span style="color:#b84">"mvandongen/asecretservice:latest"</span><span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">PortMappings</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span>- <span style="font-weight:bold">ContainerPort</span>:<span style="color:#bbb"> </span><span style="color:#099">80</span><span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">Environment</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span>- <span style="font-weight:bold">Name</span>:<span style="color:#bbb"> </span><span style="color:#b84">"SECRET"</span><span style="color:#bbb">
</span><span style="color:#bbb">              </span><span style="font-weight:bold">Value</span>:<span style="color:#bbb"> </span><span style="color:#b84">"ssm://my.little.secret"</span><span style="color:#bbb">
</span><span style="color:#bbb">            </span>- <span style="font-weight:bold">Name</span>:<span style="color:#bbb"> </span><span style="color:#b84">"SECRET_SDK"</span><span style="color:#bbb">
</span><span style="color:#bbb">              </span><span style="font-weight:bold">Value</span>:<span style="color:#bbb"> </span><span style="color:#b84">"ssm_sdk://my.little.secret"</span><span style="color:#bbb">
</span><span style="color:#bbb">            </span>- <span style="font-weight:bold">Name</span>:<span style="color:#bbb"> </span><span style="color:#b84">"SECRET_SDK_PREFIX"</span><span style="color:#bbb">
</span><span style="color:#bbb">              </span><span style="font-weight:bold">Value</span>:<span style="color:#bbb"> </span><span style="color:#b84">"ssm_sdk://"</span><span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">LogConfiguration</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span><span style="font-weight:bold">LogDriver</span>:<span style="color:#bbb"> </span>awslogs<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="font-weight:bold">Options</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">              </span><span style="font-weight:bold">awslogs-region</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span>AWS::Region<span style="color:#bbb">
</span><span style="color:#bbb">              </span><span style="font-weight:bold">awslogs-group</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span>LogGroup<span style="color:#bbb">
</span><span style="color:#bbb">              </span><span style="font-weight:bold">awslogs-stream-prefix</span>:<span style="color:#bbb"> </span>ecs<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">TaskRole</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::IAM::Role<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">AssumeRolePolicyDocument</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span><span style="font-weight:bold">Statement</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">          </span>- <span style="font-weight:bold">Effect</span>:<span style="color:#bbb"> </span>Allow<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="font-weight:bold">Principal</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">              </span><span style="font-weight:bold">Service</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">                </span>- ecs-tasks.amazonaws.com<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="font-weight:bold">Action</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">              </span>- sts:AssumeRole<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">Path</span>:<span style="color:#bbb"> </span>/<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">ManagedPolicyArns</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span>- <span style="color:#b84">'arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess'</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span>- <span style="color:#b84">'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'</span><span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">SecurityGroup</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::EC2::SecurityGroup<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">VpcId</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span><span style="color:#b84">'VpcId'</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">Groupcontent</span>:<span style="color:#bbb"> </span><span style="color:#b84">"Public access to container"</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">SecurityGroupIngress</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span>- <span style="font-weight:bold">FromPort</span>:<span style="color:#bbb"> </span><span style="color:#099">80</span><span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">ToPort</span>:<span style="color:#bbb"> </span><span style="color:#099">80</span><span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">IpProtocol</span>:<span style="color:#bbb"> </span><span style="color:#b84">'tcp'</span><span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">CidrIp</span>:<span style="color:#bbb"> </span><span style="color:#b84">"0.0.0.0/0"</span><span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">LogGroup</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::Logs::LogGroup<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">LogGroupName</span>:<span style="color:#bbb"> </span>!Sub<span style="color:#bbb"> </span><span style="color:#b84">"${AWS::StackName}"</span><span style="color:#bbb">
</span>

Because of the few parameters which are easy to select, we deploy the stack using the console. You could also use the cli if you want.

Secrets in AWS ECS Fargate

Now find the public IP address of the Docker container running in Fargate, and use curl or a web browser to check out the decrypted secrets. Also try to find the secrets in the AWS Management Console. It is not visible in the CloudFormation console, not in the ECS Fargate console. It’s only visible in the SSM Parameter Store.

Conclusion

In this blog post we have created a secret in the AWS SSM parameter store and retrieved it in a Docker container, without exposing it anywhere in the Management Console. By using the SDK, the programmer makes sure the secret is only retrieved and used where needed, this is most secure.

In the second example, from the application perspective, it’s just an environment variable within the container. When you leak all env vars, you also leak your secret, making this less secure.

It now becomes really easy to rotate secrets. It even doesn’t require a deployment.

  1. Update the secret in the SSM parameter store
  2. Gracefully terminate containers
  3. New containers with new secrets are recovered

If you have come this far, you might also be interested safely deploying or storing secrets in CloudFormation.

Share this article: Tweet this post / Post on LinkedIn