Description

Ensure that every Amazon SageMaker training job has VPC settings configured, with at least one subnet and security group in its VpcConfig. With VPC settings, SageMaker places network interfaces for the training job in your VPC, so the traffic of the training containers is controlled by your security groups, route tables and VPC endpoints.

Rationale

Without VPC settings, training containers rely on public networking with broad outbound access. This makes it easier to exfiltrate datasets or model artifacts, download malicious code, and bypass security group controls and VPC-based monitoring. Running training jobs in specific subnets with tightly scoped security groups, VPC endpoints for Amazon S3, Amazon ECR and SageMaker, and outbound internet blocked by default keeps training traffic private.

Impact

  • Places training traffic under your VPC security groups, routing and monitoring

  • Reduces the risk of data exfiltration from training containers

  • Existing training jobs cannot be changed, so the setting applies only to training jobs created after the change; existing failing jobs keep failing and must be accepted as exceptions

  • The subnets must provide access to Amazon S3 and Amazon ECR (through VPC endpoints or a NAT gateway), or the training job will fail to load its data and container image

Default Value

A training job has no VPC configuration unless subnets and security groups are specified when it is created.

Pre-Requisites

  • IAM permissions: sagemaker:ListTrainingJobs, sagemaker:DescribeTrainingJob, sagemaker:CreateTrainingJob and iam:PassRole

  • A VPC with private subnets and a security group for the training containers

  • VPC endpoints (or a NAT gateway) so that the training containers can reach Amazon S3 and Amazon ECR

  • The execution role must allow the network interface actions SageMaker needs, such as ec2:CreateNetworkInterface and ec2:DescribeSubnets

Remediation

Test Plan

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and select Training jobs under Training.

  3. Select a training job and review its network settings.

  4. Check that a VPC with at least one subnet is configured.

  5. Repeat for each training job in each region in use.

  6. If any training job has no VPC subnets, follow the implementation steps.

CLI:

List the training jobs in a region:

aws sagemaker list-training-jobs --region <REGION> --query "TrainingJobSummaries[].[TrainingJobName,TrainingJobStatus]" --output table

Show the VPC settings of a training job:

aws sagemaker describe-training-job --region <REGION> --training-job-name <TRAINING_JOB_NAME> --query "VpcConfig"

A training job fails if the output is null or contains no Subnets. All listed training jobs are evaluated, whatever their status.

Implementation Steps

Existing training jobs cannot be modified, and every training job that SageMaker still lists is evaluated. The original failing job therefore keeps failing even after a compliant clone is created. These steps prevent new failures; existing failing jobs must be accepted as exceptions, and they stop being reported only when SageMaker no longer returns them.

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and select Training jobs under Training.

  3. Select the failing training job, choose Actions, then Clone (or choose Create training job).

  4. In the network section, select a VPC, then choose at least one subnet and one security group.

  5. Complete the remaining settings and choose Create training job.

  6. Update the pipelines, notebooks or scripts that create training jobs so that new jobs include VPC settings:

    • in the CreateTrainingJob API, set VpcConfig with Subnets and SecurityGroupIds,

    • in the SageMaker Python SDK, set subnets and security_group_ids on the estimator.

  7. Repeat for each affected training job in each region in use.

Backout Plan

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and select Training jobs under Training.

  3. Create new training jobs without VPC settings, or remove the VPC settings from the pipelines, notebooks or scripts that create them.

Note: rolling back removes the network isolation of the training containers.

Reference