Description

Ensure that every Amazon SageMaker training job encrypts the ML storage volumes attached to its training instances with an AWS Key Management Service (KMS) key that you specify, set in the job's VolumeKmsKeyId setting. The model artifacts that the job writes to Amazon S3 should also be encrypted with a KMS key, set in the job's output KmsKeyId setting.

Rationale

Training volumes hold training data, checkpoints and logs. Without encryption under a KMS key that you control, this data can be exposed through snapshots or privileged access, and you cannot restrict, rotate or revoke access to it through key policies. Encrypting training volumes and outputs with customer-managed KMS keys, with least-privilege key policies and key rotation, protects the data at rest and gives you control and an audit trail of its use.

Impact

  • Protects training data, checkpoints and model artifacts at rest with a key you control

  • Existing training jobs cannot be changed, so the setting applies only to training jobs created after the change; existing failing jobs keep failing and must be accepted as exceptions

  • Instance types with local instance storage encrypt that storage with a hardware module and do not accept a volume KMS key

  • The training job's execution role needs permission to use the KMS key, and KMS usage incurs standard AWS KMS charges

Default Value

No volume KMS key or output KMS key is set on a training job unless one is specified when the job is created.

Pre-Requisites

  • IAM permissions: sagemaker:ListTrainingJobs, sagemaker:DescribeTrainingJob, sagemaker:CreateTrainingJob and iam:PassRole

  • A customer-managed KMS key in the same region whose key policy allows the training job's execution role to use it

  • Access to the pipelines, notebooks or scripts that create training jobs, so the setting can be added to them

Remediation

Test Plan

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and select Training jobs under Training.

  3. Select a training job and review its resource configuration and output settings.

  4. Check that a volume encryption key is configured.

  5. Repeat for each training job in each region in use.

  6. If any training job has no volume encryption key, follow the implementation steps.

CLI:

List the training jobs in a region:

aws sagemaker list-training-jobs --region <REGION> --query "TrainingJobSummaries[].[TrainingJobName,TrainingJobStatus]" --output table

Show the volume and output KMS keys of a training job:

aws sagemaker describe-training-job --region <REGION> --training-job-name <TRAINING_JOB_NAME> --query "[ResourceConfig.VolumeKmsKeyId,OutputDataConfig.KmsKeyId]"

A training job fails if the first value (VolumeKmsKeyId) is null. All listed training jobs are evaluated, whatever their status. The second value (output key) should also be set, as a best practice.

Implementation Steps

Existing training jobs cannot be modified, and every training job that SageMaker still lists is evaluated. The original failing job therefore keeps failing even after a compliant clone is created. These steps prevent new failures; existing failing jobs must be accepted as exceptions, and they stop being reported only when SageMaker no longer returns them.

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and select Training jobs under Training.

  3. Select the failing training job, choose Actions, then Clone (or choose Create training job).

  4. In the resource configuration section, select your KMS key as the volume encryption key.

  5. In the output data configuration section, select your KMS key as the output encryption key.

  6. Complete the remaining settings and choose Create training job.

  7. Update the pipelines, notebooks or scripts that create training jobs so that new jobs set both keys:

    • in the CreateTrainingJob API, set ResourceConfig.VolumeKmsKeyId and OutputDataConfig.KmsKeyId,

    • in the SageMaker Python SDK, set volume_kms_key and output_kms_key on the estimator.

  8. Repeat for each affected training job in each region in use.

Backout Plan

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and select Training jobs under Training.

  3. Create new training jobs without the KMS keys, or remove the keys from the pipelines, notebooks or scripts that create them.

Note: rolling back leaves training volumes and outputs without encryption under a key you control. Do not disable or delete the KMS key while model artifacts encrypted with it are still needed.

Reference