Profile Applicability:
Level 1
Description:
AWS Glue is a fully managed ETL (Extract, Transform, Load) service that helps with preparing and loading data for analytics. Development endpoints in AWS Glue are used for running Spark scripts in local development environments or integrated IDEs. These endpoints allow you to interact with the data sources.
When using Amazon S3 as a data source or target in AWS Glue, it's essential to ensure that S3 encryption is enabled to protect the data at rest. Enabling S3 encryption ensures that all data written to or read from Amazon S3 is securely encrypted using either SSE-S3 or SSE-KMS encryption.
This SOP verifies that S3 encryption is enabled for AWS Glue development endpoints, protecting data stored in Amazon S3.
Rationale:
Enabling S3 encryption for AWS Glue development endpoints is crucial for:
Data Security: Protects sensitive data by ensuring that it is encrypted at rest in Amazon S3.
Compliance: Helps meet regulatory compliance standards such as PCI-DSS, HIPAA, and SOC 2, which require encryption of data.
Best Practice: Aligns with security best practices by ensuring that data is not stored in plaintext on Amazon S3.
Impact:
Pros:
Enhanced Data Protection: Ensures data is encrypted, preventing unauthorized access.
Compliance: Meets security requirements for data encryption, supporting compliance with various standards.
Risk Mitigation: Reduces the risk of data breaches and leaks of sensitive information.
Cons:
Performance Overhead: Enabling encryption could introduce minor performance overhead during read/write operations on encrypted data, though this is typically minimal.
Key Management: If using SSE-KMS, you will need to manage the KMS keys, which could introduce additional complexity.
Default Value:
By default, AWS Glue development endpoints do not have S3 encryption enabled unless explicitly configured during job setup or endpoint creation. S3 encryption must be manually configured to ensure data is encrypted at rest.
Pre-requisite:
AWS IAM Permissions:
glue:DescribeDevEndpoint
glue:UpdateDevEndpoint
s3:PutObject
kms:DescribeKey (if using SSE-KMS)
AWS CLI installed and configured.
Basic knowledge of AWS Glue, Amazon S3 encryption, and KMS.
Remediation:
Test Plan:
Using AWS Console:
Sign in to the AWS Management Console.
Navigate to AWS Glue under Services.
Go to Development Endpoints.
Select the development endpoint you want to inspect.
Check the S3 Encryption settings under the Security Configuration section:
If S3 encryption is enabled, it will show either SSE-S3 or SSE-KMS encryption settings.
If encryption is not enabled, it will show None or no encryption configured.
Using AWS CLI:
To describe the development endpoint and check if S3 encryption is enabled, run:
aws glue get-dev-endpoints --query 'DevEndpoints[*].{Name:EndpointName,S3EncryptionMode:SecurityConfiguration}'
Check if the S3EncryptionMode is SSE-S3 or SSE-KMS.
Example output
[
{
"Name": "my-glue-endpoint",
"S3EncryptionMode": "SSE-KMS",
"KmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/abcd-1234"
}
]If the S3EncryptionMode is NONE, then S3 encryption is not enabled.
Implementation Steps:
Using AWS Console:
Sign in to the AWS Management Console and navigate to AWS Glue.
In the AWS Glue Dashboard, go to Development Endpoints and select the endpoint to modify.
In the Security Configuration section, locate S3 Encryption settings.
Enable S3 encryption:
Select SSE-S3 for simple encryption using AWS-managed keys.
Select SSE-KMS for more granular control, using a custom KMS key for encryption.
If using SSE-KMS, select the KMS key from your KMS key list or create a new one.
Save the changes to enable S3 encryption for the development endpoint.
Using AWS CLI:
To enable S3 encryption using AWS KMS, run the following command:
aws glue update-dev-endpoint \
--endpoint-name <dev-endpoint-name> \
--security-configuration '{"S3EncryptionMode": "SSE-KMS", "KmsKeyId": "<kms-key-id>"}'To enable S3 encryption using SSE-S3, run:
aws glue update-dev-endpoint \
--endpoint-name <dev-endpoint-name> \
--security-configuration '{"S3EncryptionMode": "SSE-S3"}Verify the update by running:
aws glue get-dev-endpoints --query 'DevEndpoints[*].{Name:EndpointName,S3EncryptionMode:SecurityConfiguration}'
Ensure that S3EncryptionMode is set to SSE-KMS or SSE-S3.
Backout Plan:
If enabling S3 encryption causes issues (e.g., access issues or job failures):
Identify the affected development endpoint.
To disable S3 encryption, run:
aws glue update-dev-endpoint \
--endpoint-name <dev-endpoint-name> \
--security-configuration '{"S3EncryptionMode": "NONE"}'Verify that S3 encryption has been disabled, and the endpoint is functioning as expected.
Note:
KMS Key Management: If using SSE-KMS, ensure the KMS key is properly managed, rotated, and only accessible to authorized users and roles.
Performance Monitoring: After enabling encryption, monitor the performance of the ETL jobs to ensure that encryption does not significantly impact job execution time.
Log Monitoring: Ensure that the logs for the development endpoint (e.g., CloudWatch Logs or S3 logs) are encrypted as well.