Description

Ensure that every Amazon SageMaker notebook instance encrypts its ML storage volume with an AWS Key Management Service (KMS) key that you specify. The key is set in the notebook instance's KmsKeyId setting when the instance is created and is used to encrypt the data stored on the attached volume.

Rationale

Notebook storage volumes often hold training data, credentials and code. Without encryption under a KMS key that you control, this data and its snapshots can be exposed through storage access, copied backups or host compromise, and you cannot limit, rotate or revoke access to it through key policies. Using a customer-managed KMS key, with least-privilege key policies and key rotation, protects the data at rest and gives you control and an audit trail of its use.

Impact

  • Protects notebook data at rest with a key you control and can audit, rotate or disable

  • The encryption key cannot be added to an existing notebook instance, so a new instance must be created and the data migrated

  • Users of the notebook instance need permission to use the KMS key

  • KMS key usage incurs standard AWS KMS charges

Default Value

A notebook instance has no KMS key in its KmsKeyId setting unless one is chosen when the instance is created. The key cannot be changed after creation.

Pre-Requisites

  • IAM permissions: sagemaker:ListNotebookInstances, sagemaker:DescribeNotebookInstance, sagemaker:CreateNotebookInstance, sagemaker:StopNotebookInstance, sagemaker:DeleteNotebookInstance and iam:PassRole

  • A customer-managed KMS key in the same region whose key policy allows the users and role that create and use the notebook instance

  • A plan to copy notebooks and data from the old instance (for example, to Amazon S3 or a Git repository) before it is deleted

Remediation

Test Plan

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and go to the list of notebook instances.

  3. Select a notebook instance and review its details.

  4. Check that an Encryption key is shown.

  5. Repeat for each notebook instance in each region in use.

  6. If any notebook instance has no encryption key, follow the implementation steps.

CLI:

List the notebook instances in a region:

aws sagemaker list-notebook-instances --region <REGION> --query "NotebookInstances[].NotebookInstanceName"

Show the KMS key of a notebook instance:

aws sagemaker describe-notebook-instance --region <REGION> --notebook-instance-name <NOTEBOOK_INSTANCE_NAME> --query "KmsKeyId"

A notebook instance fails if the output is null (no KMS key is configured).

Implementation Steps

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and go to the list of notebook instances.

  3. Copy the notebooks and data of the failing instance to Amazon S3 or a Git repository.

  4. Choose Create notebook instance and enter a name, instance type and IAM role.

  5. In the Permissions and encryption section, select your KMS key as the Encryption key.

  6. Choose Create notebook instance and wait until the status is InService.

  7. Copy the notebooks and data to the new instance.

  8. Select the old notebook instance, choose Actions, then Stop, and after it stops choose Actions, then Delete.

  9. Repeat for each affected notebook instance in each region in use.

Backout Plan

Before making changes, save the current notebook instance settings:

aws sagemaker describe-notebook-instance --region <REGION> --notebook-instance-name <NOTEBOOK_INSTANCE_NAME> > <NOTEBOOK_INSTANCE_NAME>-backup.json

To roll back:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon SageMaker AI console and go to the list of notebook instances.

  3. If the old notebook instance was not deleted yet, start it again and continue using it.

  4. Otherwise, create a notebook instance with the settings from the saved backup and copy the notebooks and data to it.

  5. Stop and delete the new encrypted instance if it is no longer needed.

Note: rolling back leaves the notebook data without encryption under a key you control.

Reference