Description

Ensure that every Amazon SageMaker notebook instance is launched in a VPC, with a subnet configured. A notebook instance with VPC settings has a network interface in your VPC, so its traffic can be controlled with security groups, route tables and VPC endpoints. A notebook instance without VPC settings uses networking managed by SageMaker with direct internet access.

Rationale

Without a VPC, a notebook instance has no network isolation from your environment's controls. Traffic to AWS services may go over the public internet, and outbound access cannot be restricted with your security groups or private endpoints, which makes data exfiltration and interception easier. Running notebooks in private subnets with least-privilege security groups, VPC endpoints and restricted outbound traffic applies defense in depth to the notebook environment.

Impact

  • Places the notebook instance's network traffic under your VPC security groups, routing and monitoring

  • Reduces the risk of data exfiltration from the notebook instance

  • VPC settings cannot be added to an existing notebook instance, so a new instance must be created and the notebooks and data migrated

  • If direct internet access is disabled, the subnet needs VPC endpoints or a NAT gateway to reach required services and package repositories

Default Value

A notebook instance is not attached to a VPC unless a VPC and subnet are selected when it is created. The VPC settings cannot be changed after creation.

Pre-Requisites

  • IAM permissions: sagemaker:ListNotebookInstances, sagemaker:DescribeNotebookInstance, sagemaker:CreateNotebookInstance, sagemaker:StopNotebookInstance, sagemaker:DeleteNotebookInstance and iam:PassRole

  • A VPC with a subnet and a security group for the notebook instance

  • VPC endpoints (or a NAT gateway) for the services the notebook needs, such as Amazon S3 and the SageMaker API

  • A plan to copy notebooks and data from the old instance (for example, to Amazon S3 or a Git repository) before it is deleted

Remediation

Test Plan

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and go to the list of notebook instances.

  3. Select a notebook instance and review its network settings.

  4. Check that a VPC and subnet are configured.

  5. Repeat for each notebook instance in each region in use.

  6. If any notebook instance has no subnet, follow the implementation steps.

CLI:

List the notebook instances in a region:

aws sagemaker list-notebook-instances --region <REGION> --query "NotebookInstances[].NotebookInstanceName"

Show the network settings of a notebook instance:

aws sagemaker describe-notebook-instance --region <REGION> --notebook-instance-name <NOTEBOOK_INSTANCE_NAME> --query "[SubnetId,SecurityGroups,DirectInternetAccess]"

A notebook instance fails if the first value (SubnetId) is null (no VPC subnet is configured).

Implementation Steps

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and go to the list of notebook instances.

  3. Copy the notebooks and data of the failing instance to Amazon S3 or a Git repository.

  4. Choose Create notebook instance and enter a name, instance type and IAM role.

  5. In the Network section, select your VPC, a subnet and the required security groups.

  6. Choose Create notebook instance and wait until the status is InService.

  7. Copy the notebooks and data to the new instance.

  8. Select the old notebook instance, choose Actions, then Stop, and after it stops choose Actions, then Delete.

  9. Repeat for each affected notebook instance in each region in use.

Backout Plan

Before making changes, save the current notebook instance settings:

aws sagemaker describe-notebook-instance --region <REGION> --notebook-instance-name <NOTEBOOK_INSTANCE_NAME> > <NOTEBOOK_INSTANCE_NAME>-backup.json

To roll back:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and go to the list of notebook instances.

  3. If the old notebook instance was not deleted yet, start it again and continue using it.

  4. Otherwise, create a notebook instance with the settings from the saved backup, without VPC settings, and copy the notebooks and data to it.

  5. Stop and delete the new instance if it is no longer needed.

Note: rolling back removes the network isolation of the notebook instance.

Reference