Description

Ensure that the user data of Amazon EC2 instances does not contain secrets such as passwords, access keys, tokens or private keys. User data is commonly used for startup scripts, and both plain and gzip-compressed user data are inspected for values that look like secrets. Terminated instances are not evaluated.

Rationale

User data is not encrypted and can be read by anyone with permission to view instance attributes, and by any process on the instance through the instance metadata service. Secrets stored in user data can therefore be reused to call services, exfiltrate data or move laterally, and they can persist in AMIs, snapshots, templates and backups. Secrets should be stored in a managed service such as AWS Secrets Manager or AWS Systems Manager Parameter Store and fetched at runtime through a least-privilege instance role.

Impact

  • Removes secrets from a location that many users and processes can read

  • Secrets found in user data must be treated as exposed and rotated

  • Changing user data requires the instance to be stopped, which interrupts the workload

  • Startup scripts must be updated to fetch secrets at runtime instead of reading them from user data

Default Value

An EC2 instance has no user data unless it is provided at launch or added later.

Pre-Requisites

  • IAM permissions: ec2:DescribeInstances, ec2:DescribeInstanceAttribute, ec2:ModifyInstanceAttribute, ec2:StopInstances and ec2:StartInstances

  • The secrets stored in AWS Secrets Manager or AWS Systems Manager Parameter Store, and an instance role that can read them

  • A maintenance window agreed with the application owner, because the instance must be stopped to change its user data

Remediation

Test Plan

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon EC2 console and select Instances.

  3. Select an instance and choose Actions, Instance settings, then Edit user data.

  4. Review the user data for passwords, access keys, tokens, private keys or connection strings.

  5. Repeat for each instance in each region in use.

  6. If user data contains a secret, follow the implementation steps.

CLI:

Show the user data of an instance (the value is base64-encoded and must be decoded to read it):

aws ec2 describe-instance-attribute --region <REGION> --instance-id <INSTANCE_ID> --attribute userData --query "UserData.Value" --output text

An instance that is not terminated fails if its decoded user data contains a value that looks like a secret, such as a password, access key, token or private key. Instances with empty user data pass.

Implementation Steps

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Store the secrets in AWS Secrets Manager or AWS Systems Manager Parameter Store, and update the startup script to read them at runtime through the instance role.

  3. Open the Amazon EC2 console and select Instances.

  4. Select the instance and choose Instance state, then Stop instance.

  5. When the instance is stopped, choose Actions, Instance settings, then Edit user data.

  6. Remove the secrets from the user data, or delete all of its content if it is no longer needed.

  7. Choose Save.

  8. Choose Instance state, then Start instance.

  9. Rotate every secret that was stored in the user data.

  10. Repeat for each affected instance in each region in use.

CLI:

The instance must be stopped before its user data can be changed, so the command below is preceded by a stop and followed by a start. Stop the instance:

aws ec2 stop-instances --instance-ids <INSTANCE_ID>

Clear the user data:

aws ec2 modify-instance-attribute --instance-id <INSTANCE_ID> --user-data Value=

Start the instance:

aws ec2 start-instances --instance-ids <INSTANCE_ID>

Backout Plan

Before making changes, save the current user data:

aws ec2 describe-instance-attribute --instance-id <INSTANCE_ID> --attribute userData > <INSTANCE_ID>-userdata-backup.json

To roll back:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon EC2 console, select the instance and stop it.

  3. Choose Actions, Instance settings, then Edit user data.

  4. Paste the previous user data from the saved backup (decoded from base64) and choose Save.

  5. Start the instance.

Note: restoring the previous user data puts the secrets back in a readable location. Use it only as a temporary measure, and do not restore secrets that have already been rotated.

Reference