# Ensure Amazon SageMaker training jobs have volume KMS encryption enabled - Source: https://support.icompaas.com/support/solutions/articles/a_1793292225038544-ensure-amazon-sagemaker-training-jobs-have-volume-kms-encryption-enabled - Section: [Solutions](https://support.icompaas.com/support/solutions) › [AWS_Checks_v5.39.1](https://support.icompaas.com/support/solutions/c_1793285913477877) › [AWS Checks](https://support.icompaas.com/support/solutions/folders/f_1793287194371224) - Last updated: 2026-10-29 - Publisher: iCompaas, Inc. (https://www.icompaas.com) **Description** Ensure that every Amazon SageMaker training job encrypts the ML storage volumes attached to its training instances with an AWS Key Management Service (KMS) key that you specify, set in the job's VolumeKmsKeyId setting. The model artifacts that the job writes to Amazon S3 should also be encrypted with a KMS key, set in the job's output KmsKeyId setting. **Rationale** Training volumes hold training data, checkpoints and logs. Without encryption under a KMS key that you control, this data can be exposed through snapshots or privileged access, and you cannot restrict, rotate or revoke access to it through key policies. Encrypting training volumes and outputs with customer-managed KMS keys, with least-privilege key policies and key rotation, protects the data at rest and gives you control and an audit trail of its use. **Impact** - Protects training data, checkpoints and model artifacts at rest with a key you control - Existing training jobs cannot be changed, so the setting applies only to training jobs created after the change; existing failing jobs keep failing and must be accepted as exceptions - Instance types with local instance storage encrypt that storage with a hardware module and do not accept a volume KMS key - The training job's execution role needs permission to use the KMS key, and KMS usage incurs standard AWS KMS charges **Default Value** No volume KMS key or output KMS key is set on a training job unless one is specified when the job is created. **Pre-Requisites** - IAM permissions: sagemaker:ListTrainingJobs, sagemaker:DescribeTrainingJob, sagemaker:CreateTrainingJob and iam:PassRole - A customer-managed KMS key in the same region whose key policy allows the training job's execution role to use it - Access to the pipelines, notebooks or scripts that create training jobs, so the setting can be added to them **Remediation** **Test Plan** **Console:** 1. Sign in to the AWS Management Console at [https://console.aws.amazon.com](https://console.aws.amazon.com/). 2. Open the Amazon SageMaker AI console and select **Training jobs** under **Training**. 3. Select a training job and review its resource configuration and output settings. 4. Check that a volume encryption key is configured. 5. Repeat for each training job in each region in use. 6. If any training job has no volume encryption key, follow the implementation steps. **CLI:** List the training jobs in a region: aws sagemaker list-training-jobs --region --query "TrainingJobSummaries\[\].\[TrainingJobName,TrainingJobStatus\]" --output table Show the volume and output KMS keys of a training job: aws sagemaker describe-training-job --region --training-job-name --query "\[ResourceConfig.VolumeKmsKeyId,OutputDataConfig.KmsKeyId\]" A training job fails if the first value (VolumeKmsKeyId) is null. All listed training jobs are evaluated, whatever their status. The second value (output key) should also be set, as a best practice. **Implementation Steps** Existing training jobs cannot be modified, and every training job that SageMaker still lists is evaluated. The original failing job therefore keeps failing even after a compliant clone is created. These steps prevent new failures; existing failing jobs must be accepted as exceptions, and they stop being reported only when SageMaker no longer returns them. **Console:** 1. Sign in to the AWS Management Console at [https://console.aws.amazon.com](https://console.aws.amazon.com/). 2. Open the Amazon SageMaker AI console and select **Training jobs** under **Training**. 3. Select the failing training job, choose **Actions**, then **Clone** (or choose **Create training job**). 4. In the resource configuration section, select your KMS key as the volume encryption key. 5. In the output data configuration section, select your KMS key as the output encryption key. 6. Complete the remaining settings and choose **Create training job**. 7. Update the pipelines, notebooks or scripts that create training jobs so that new jobs set both keys: - in the CreateTrainingJob API, set ResourceConfig.VolumeKmsKeyId and OutputDataConfig.KmsKeyId, - in the SageMaker Python SDK, set volume\_kms\_key and output\_kms\_key on the estimator. 8. Repeat for each affected training job in each region in use. **Backout Plan** 1. Sign in to the AWS Management Console at [https://console.aws.amazon.com](https://console.aws.amazon.com/). 2. Open the Amazon SageMaker AI console and select **Training jobs** under **Training**. 3. Create new training jobs without the KMS keys, or remove the keys from the pipelines, notebooks or scripts that create them. Note: rolling back leaves training volumes and outputs without encryption under a key you control. Do not disable or delete the KMS key while model artifacts encrypted with it are still needed. **Reference** - [https://docs.aws.amazon.com/sagemaker/latest/dg/key-management.html](https://docs.aws.amazon.com/sagemaker/latest/dg/key-management.html) - [https://docs.aws.amazon.com/sagemaker/latest/dg/encryption-at-rest.html](https://docs.aws.amazon.com/sagemaker/latest/dg/encryption-at-rest.html) - [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API\_ResourceConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ResourceConfig.html) - [https://docs.aws.amazon.com/cli/latest/reference/sagemaker/describe-training-job.html](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/describe-training-job.html) ## Related compliance guidance - [SOC 2 compliance: a practical implementation guide for cloud-native SaaS teams](https://support.icompaas.com/compliance/soc-2-trust-services-criteria): Configuration checks like this one are the technical half of a SOC 2 / ISO 27001 programme — the guide shows where they fit, what else an auditor expects, and how to evidence them. - [ISO/IEC 27001:2022 compliance: a practical implementation guide](https://support.icompaas.com/compliance/iso-27001-2022): ISO 27001 Annex A.8 (technological controls) is where cloud hardening evidence lands; the guide shows how to run the ISMS around it.