Description
Ensure that every Amazon SageMaker training job has inter-container traffic encryption enabled. When the EnableInterContainerTrafficEncryption setting is turned on, the communication between the containers of a distributed training job is encrypted in transit.
Rationale
Without inter-container encryption, the traffic exchanged between training containers can be sent in plain text and captured by a compromised host or workload. This exposes training data, model parameters and credentials, and allows an attacker to tamper with intermediate results or steal tokens to move to other resources. Enabling inter-container traffic encryption on all training jobs, together with network isolation and least-privilege roles, protects the confidentiality and integrity of the training process.
Impact
Encrypts data exchanged between training containers in transit
Protects training data and model parameters from interception and tampering
Encryption can increase the training time of distributed training jobs, and therefore their cost
Existing training jobs cannot be changed, so the setting applies only to training jobs created after the change; existing failing jobs keep failing and must be accepted as exceptions
Default Value
Inter-container traffic encryption is disabled by default for new training jobs.
Pre-Requisites
IAM permissions: sagemaker:ListTrainingJobs, sagemaker:DescribeTrainingJob, sagemaker:CreateTrainingJob and iam:PassRole
Access to the pipelines, notebooks or scripts that create training jobs, so the setting can be added to them
Remediation
Test Plan
Console:
Sign in to the AWS Management Console at https://console.aws.amazon.com.
Open the Amazon SageMaker AI console and select Training jobs under Training.
Select a training job and review its network settings.
Check that inter-container traffic encryption is enabled.
Repeat for each training job in each region in use.
If any training job has inter-container traffic encryption disabled, follow the implementation steps.
CLI:
List the training jobs in a region:
aws sagemaker list-training-jobs --region <REGION> --query "TrainingJobSummaries[].[TrainingJobName,TrainingJobStatus]" --output table
Show the inter-container encryption setting of a training job:
aws sagemaker describe-training-job --region <REGION> --training-job-name <TRAINING_JOB_NAME> --query "EnableInterContainerTrafficEncryption"
A training job fails if the output is false or null. All listed training jobs are evaluated, whatever their status.
Implementation Steps
Existing training jobs cannot be modified, and every training job that SageMaker still lists is evaluated. The original failing job therefore keeps failing even after a compliant clone is created. These steps prevent new failures; existing failing jobs must be accepted as exceptions, and they stop being reported only when SageMaker no longer returns them.
Console:
Sign in to the AWS Management Console at https://console.aws.amazon.com.
Open the Amazon SageMaker AI console and select Training jobs under Training.
Select the failing training job, choose Actions, then Clone (or choose Create training job).
Select Enable inter-container traffic encryption.
Complete the remaining settings and choose Create training job.
Update the pipelines, notebooks or scripts that create training jobs so that new jobs enable this setting:
in the CreateTrainingJob API, set EnableInterContainerTrafficEncryption to true,
in the SageMaker Python SDK, set encrypt_inter_container_traffic=True on the estimator.
Repeat for each affected training job in each region in use.
Backout Plan
Sign in to the AWS Management Console at https://console.aws.amazon.com.
Open the Amazon SageMaker AI console and select Training jobs under Training.
Create new training jobs with Enable inter-container traffic encryption cleared, or remove the setting from the pipelines, notebooks or scripts that create them.
Note: rolling back sends traffic between training containers without encryption again.
Reference