iCompaas Support

Profile Applicability:

• Level 1

Description:

AWS Glue is a fully managed ETL (Extract, Transform, Load) service that enables the preparation and loading of data for analytics. Development endpoints in AWS Glue allow you to interact with data in a local development environment or an IDE, and execute Spark scripts.

Job bookmarks in AWS Glue allow you to maintain state between job runs, enabling the processing of only new or modified data, thereby reducing the amount of data that needs to be processed. Job bookmark encryption ensures that the metadata for job bookmarks is securely encrypted at rest using AWS Key Management Service (KMS).

This SOP ensures that job bookmark encryption is enabled for AWS Glue development endpoints, ensuring that sensitive metadata stored as part of the job state is protected.

Rationale:

Enabling job bookmark encryption for AWS Glue development endpoints is important for:

• Data Security: Protects sensitive job metadata, such as state information between runs, which could contain critical data transformation state.

• Compliance: Helps meet regulatory requirements like PCI-DSS, HIPAA, and SOC 2, which require encryption of sensitive data at rest.

• Best Practice: Aligns with industry best practices for securing metadata associated with data transformations.

Impact:

Pros:

• Enhanced Security: Ensures that job metadata (job bookmarks) is encrypted at rest, preventing unauthorized access to sensitive job state data.

• Compliance: Meets security and regulatory requirements for data encryption, supporting compliance with data protection laws.

• Data Integrity: Ensures the integrity and confidentiality of job metadata, particularly useful for long-running or incremental jobs.

Cons:

• Key Management Overhead: If using SSE-KMS encryption, managing the KMS keys adds complexity to key rotation and permissions management.

• Performance Overhead: While the encryption overhead is typically minimal, there may be a slight performance impact when reading and writing encrypted job bookmark data.

Default Value:

By default, job bookmark encryption is not enabled for AWS Glue development endpoints. You need to explicitly enable job bookmark encryption when creating or modifying the development endpoint.

Pre-requisite:

• AWS IAM Permissions:

  • glue:DescribeDevEndpoint

  • glue:UpdateDevEndpoint

  • kms:DescribeKey

• AWS CLI installed and configured.

• Basic knowledge of AWS Glue, job bookmarks, and KMS encryption.

Remediation:

Test Plan:

Using AWS Console:

1. Sign in to the AWS Management Console.

2. Navigate to AWS Glue under Services.

3. Go to Development Endpoints.

4. Select the development endpoint you want to check.

5. In the Job Details section, verify if Job Bookmark Encryption is enabled.

   • If enabled, the Job bookmark encryption mode should show AWS KMS encryption or similar.

   • If not enabled, it will not display an encryption setting, or it will show None.

Using AWS CLI:

To describe the development endpoint and check if Job Bookmark Encryption is enabled, run:

aws glue get-dev-endpoints --query 'DevEndpoints[*].{Name:EndpointName,JobBookmarkEncryption:SecurityConfiguration}'

1. Check the JobBookmarkEncryption field:
   

   • If JobBookmarkEncryption is enabled, it should show SSE-KMS or similar encryption information.

   • If enabled, it should also show the KMS key ID used for encryption.

Example output

[

    {
        "Name": "my-glue-endpoint",

        "JobBookmarkEncryption": "SSE-KMS",

        "KmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/abcd1234"

    }
]

3. If JobBookmarkEncryption is None or missing, it is not enabled.

Implementation Steps:

Using AWS Console:

1. Sign in to the AWS Management Console and navigate to AWS Glue.

2. In the AWS Glue Dashboard, go to Development Endpoints and select the endpoint to modify.

3. Under Security Configuration, enable Job Bookmark Encryption:

   • Ensure that Job Bookmark Encryption is set to SSE-KMS.

   • Choose the KMS Key for encryption (either AWS-managed or customer-managed key).

4. Save the changes to enable job bookmark encryption.

Using AWS CLI:

To enable Job Bookmark Encryption using AWS KMS, run:

aws glue update-dev-endpoint \
 --endpoint-name <dev-endpoint-name> \

  --security-configuration '{"JobBookmarkEncryption": "SSE-KMS", "KmsKeyId": "<kms-key-id>"}'

To verify that job bookmark encryption is enabled, run:

aws glue get-dev-endpoints --query 'DevEndpoints[*].{Name:EndpointName,JobBookmarkEncryption:SecurityConfiguration}'

1. Ensure that the JobBookmarkEncryption property is set to SSE-KMS and the KmsKeyId is provided.

Backout Plan:

If enabling job bookmark encryption causes issues, such as job failures or access issues:

1. Identify the affected development endpoint.

To disable job bookmark encryption, run:

aws glue update-dev-endpoint \

  --endpoint-name <dev-endpoint-name> \

  --security-configuration '{"JobBookmarkEncryption": "None"}'

2. Verify that job bookmark encryption has been disabled and that the endpoint is functioning as expected.
   

Note:

• Key Management: If using SSE-KMS, ensure that the KMS key used for encryption is properly managed, rotated, and only accessible to authorized users and roles.

• Job Testing: Test the development endpoint and related jobs to ensure they are working correctly after enabling job bookmark encryption.

• Performance Considerations: Monitor the performance of the job runs after enabling encryption to ensure no significant delays.

References:

• AWS Glue Documentation

• AWS CLI: update-dev-endpoint

• AWS CLI: get-dev-endpoints

• AWS Key Management Service (KMS) Documentation

CIS Controls Mapping: