Description

Ensure that Amazon Elastic Container Service (ECS) task definitions do not contain secrets, such as passwords, access keys, API tokens or private keys, as plaintext values in the environment variables of their container definitions. Sensitive values should instead be stored in AWS Secrets Manager or AWS Systems Manager Parameter Store and injected into the container at runtime.

Rationale

Plaintext environment variables in a task definition are visible to anyone who can describe the task definition, and they can also appear in logs, task metadata and container inspection output. An attacker with read-only API access or access to a container can reuse these credentials to read or modify databases, move to other services or cause outages and unexpected costs. Storing secrets in Secrets Manager or Parameter Store, with least-privilege access and regular rotation, keeps them out of the task definition.

Impact

  • Removes plaintext credentials from task definitions, logs and API responses

  • Allows secrets to be rotated centrally without registering new task definitions

  • The task execution role must be allowed to read the referenced secrets, otherwise tasks fail to start

  • Services must be updated to the new task definition revision, which starts a new deployment

Default Value

ECS does not prevent sensitive values from being placed in plaintext environment variables. Any value entered as an environment variable is stored in the task definition as plaintext.

Pre-Requisites

  • IAM permissions: ecs:ListTaskDefinitions, ecs:DescribeTaskDefinition, ecs:RegisterTaskDefinition, ecs:DeregisterTaskDefinition and ecs:UpdateService

  • Permissions to create secrets in AWS Secrets Manager (secretsmanager:CreateSecret) or parameters in Parameter Store (ssm:PutParameter)

  • A task execution role that can read the secrets (secretsmanager:GetSecretValue or ssm:GetParameters, plus kms:Decrypt if a customer managed KMS key is used)

Remediation

Test Plan

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon ECS console and select Task definitions in the left menu.

  3. Select a task definition family, then an active revision.

  4. Open the JSON tab and review the environment list of each container definition.

  5. Check whether any variable value looks like a secret, for example a password, access key, API token, private key or a connection string that includes credentials.

  6. Repeat for each active task definition revision in each region in use.

  7. If a secret is found, follow the implementation steps.

CLI:

List the active task definition revisions:

aws ecs list-task-definitions --region <REGION> --status ACTIVE

Show the environment variables of each container in a revision:

aws ecs describe-task-definition --region <REGION> --task-definition <TASK_DEFINITION_FAMILY>:<REVISION> --query "taskDefinition.containerDefinitions[].[name,environment]"

A task definition revision fails if any container environment variable contains a value that looks like a secret, such as a password, access key, token or private key. Only active task definition revisions are evaluated.

Implementation Steps

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Store each sensitive value in AWS Secrets Manager or in Systems Manager Parameter Store as a SecureString parameter, and note its ARN.

  3. Make sure the task execution role of the task definition is allowed to read these secrets.

  4. Open the Amazon ECS console, select Task definitions and select the affected task definition.

  5. Choose Create new revision.

  6. For each container:

    • remove the sensitive values from the Environment variables,

    • add each one again with the value type ValueFrom and the ARN of the secret or parameter as the value.

  7. Choose Create to register the new revision.

  8. Update each service that uses the task definition to the new revision and wait for the deployment to complete.

  9. Deregister the old revisions that contain the plaintext secrets.

  10. Rotate the exposed credentials, because they were stored in plaintext.

  11. Repeat for each affected task definition in each region in use.

Backout Plan

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon ECS console and select the cluster and service.

  3. Choose Update service and select the previous task definition revision.

  4. Choose Update and wait for the deployment to complete.

Note: roll back before the previous revision is deregistered. Rolling back exposes the plaintext secrets again, and credentials that were already rotated will no longer work in the previous revision.

Reference