Profile Applicability:
- Level 1
Description:
Amazon SageMaker is a fully managed service for building, training, and deploying machine learning models. When running training jobs, SageMaker can store data, logs, and models in EBS volumes and S3 buckets. It is important to ensure that these volumes and outputs are encrypted using AWS Key Management Service (KMS) to protect sensitive data during the training process. This SOP ensures that Amazon SageMaker training jobs have KMS encryption enabled for both EBS volumes used during the job and the output data stored in S3.
Rationale:
Security: Encrypting the training data with KMS ensures that sensitive data is protected both at rest (in storage) and during the training process. This prevents unauthorized access and ensures data confidentiality.
Compliance: Many regulatory frameworks and standards (e.g., PCI-DSS, HIPAA) require encryption for data at rest and in transit. Enabling KMS encryption helps meet these requirements.
Data Integrity: Ensures that no one, except authorized users with the correct permissions, can read or modify the training data, models, or results.
Impact:
Pros:
Enhanced Security: Protects data by ensuring that it is encrypted with KMS keys during the training process.
Compliance: Helps organizations comply with data protection standards that mandate encryption for sensitive data.
Access Control: Provides control over who can decrypt the data by using KMS policies.
Cons:
Overhead: Enabling KMS encryption could introduce slight overhead in terms of performance, as the encryption and decryption processes require computational resources.
Configuration Complexity: Setting up KMS encryption might require additional configuration of the KMS keys and IAM roles/policies to ensure appropriate access control.
Default Value:
By default, Amazon SageMaker training jobs do not have KMS encryption enabled for both the EBS volumes and S3 output. These settings must be explicitly configured during job creation.
Pre-requisite:
AWS IAM Permissions:
sagemaker:DescribeTrainingJob
kms:DescribeKey
s3:GetBucketEncryption
AWS CLI installed and configured.
Amazon SageMaker Training Job is set up and operational.
KMS Key is created and accessible for encryption.
Test Plan:
Using AWS Console:
Sign in to the AWS Management Console.
Navigate to Amazon SageMaker under Services.
In the SageMaker Dashboard, select Training jobs.
Choose the Training job you want to review.
Under the Job details section, verify if KMS encryption is enabled for both:
Volume Encryption: Check if EBS volume encryption is enabled.
Output Encryption: Check if S3 output encryption is enabled, and that it is using the desired KMS key.
If KMS encryption is not enabled, proceed with enabling it as described in the Implementation Steps below.
Using AWS CLI:
To describe the SageMaker Training Job and check if KMS encryption is enabled, run:
aws sagemaker describe-training-job --training-job-name <training-job-name> --query 'TrainingJob.SecurityConfig'
Review the output and ensure that both:
Volume encryption is enabled (EbsVolumeEncryption field).
Output encryption is enabled (S3OutputEncryptionKmsKeyId field).
If the EbsVolumeEncryption and S3OutputEncryptionKmsKeyId fields are not set or configured, proceed with the next steps to enable them.
Implementation Steps:
Using AWS Console:
Sign in to the AWS Management Console and navigate to Amazon SageMaker.
In the SageMaker Dashboard, go to Training jobs.
Create a new Training Job or modify an existing job:
Under Security configuration, enable Volume Encryption and select the KMS key you wish to use for EBS volumes.
Enable S3 output encryption and select the KMS key for encrypting the output data stored in S3.
Save and start the training job. This will ensure both the EBS volumes and S3 output are encrypted with the chosen KMS key.
Using AWS CLI:
To create or update the training job with KMS encryption enabled, run the following command:
aws sagemaker create-training-job \ --training-job-name <training-job-name> \ --algorithm-specification TrainingImage=<image-uri>,TrainingInputMode=File \ --input-data-config <input-config> \ --output-data-config S3OutputPath=<s3-output-path>,S3OutputEncryptionKmsKeyId=<kms-key-id> \ --resource-config <resource-config> \ --security-config 'SecurityConfig={EbsVolumeEncryption=ENABLED,S3OutputEncryptionKmsKeyId=<kms-key-id>}'
To update an existing training job to enable KMS encryption, modify the security configuration:
aws sagemaker update-training-job --training-job-name <training-job-name> --security-config '{"EbsVolumeEncryption":"ENABLED","S3OutputEncryptionKmsKeyId":"<kms-key-id>"}'
Verify the update by running:
aws sagemaker describe-training-job --training-job-name <training-job-name> --query 'TrainingJob.SecurityConfig'
Ensure that the encryption settings are properly applied.
Backout Plan:
Using AWS Console:
If enabling encryption causes issues, sign in to the AWS Management Console.
Navigate to Amazon SageMaker, select the training job, and go to Edit.
Disable KMS encryption for both EBS volumes and output data.
Save the changes and ensure the job continues without encryption.
Using AWS CLI:
To disable KMS encryption, run the following command:
aws sagemaker update-training-job --training-job-name <TRAINING_JOB_NAME> --volume-encryption KmsKeyId="" --region <REGION>
Revert the output data configuration to remove KMS encryption:
aws sagemaker update-training-job --training-job-name <TRAINING_JOB_NAME> --output-data-config S3Ou