Profile Applicability:
Level 1
Description:
AWS Glue is a fully managed ETL (Extract, Transform, Load) service that helps with preparing and loading data for analytics. CloudWatch Logs integration in AWS Glue allows you to capture logs for ETL job execution, which is critical for debugging, monitoring, and troubleshooting data processing tasks.
CloudWatch Logs encryption ensures that all logs generated by AWS Glue ETL jobs are encrypted when stored in Amazon CloudWatch Logs, preventing unauthorized access to sensitive job execution data. This encryption is essential to ensure that any log data, including job status, errors, and other information, is secure.
This SOP ensures that CloudWatch Logs encryption is enabled for AWS Glue ETL jobs, helping to protect job execution logs and sensitive information from unauthorized access.
Rationale:
Enabling CloudWatch Logs encryption for AWS Glue ETL jobs is essential for:
Data Security: Protects sensitive job logs and execution data from unauthorized access.
Compliance: Meets security and compliance standards like PCI-DSS, HIPAA, and SOC 2, which require encrypted logs.
Best Practice: Ensures that logs generated by data processing workflows are securely stored, protecting sensitive information from leaks or breaches.
Impact:
Pros:
Enhanced Data Security: Ensures that job execution logs are encrypted, preventing unauthorized access to sensitive log data.
Compliance: Helps meet regulatory compliance requirements by ensuring that logs are encrypted.
Audit and Monitoring: Encrypted logs are protected and provide a traceable audit trail for ETL job execution.
Cons:
Key Management Overhead: If using SSE-KMS for encryption, you need to manage KMS keys, including permissions and key rotation.
Performance Overhead: While the encryption process is usually minimal, it could introduce slight latency when writing and retrieving logs.
Default Value:
By default, CloudWatch Logs encryption is not enabled for AWS Glue ETL jobs unless explicitly configured. If encryption is not enabled, logs are stored in plaintext in CloudWatch.
Pre-requisite:
AWS IAM Permissions:
glue:DescribeJob
glue:GetJob
glue:UpdateJob
logs:CreateLogStream
logs:PutLogEvents
kms:DescribeKey (if using SSE-KMS)
AWS CLI installed and configured.
Basic knowledge of AWS Glue, CloudWatch Logs, and KMS encryption.
Remediation:
Test Plan:
Using AWS Console:
Sign in to the AWS Management Console.
Navigate to AWS Glue under Services.
In the AWS Glue Dashboard, go to Jobs and select the ETL job you want to inspect.
In the Job Details section, check the Logging settings:
Ensure that the CloudWatch Logs are enabled for the job.
If CloudWatch Logs encryption is enabled, it will indicate either SSE-KMS or SSE-S3 encryption.
If CloudWatch Logs encryption is not enabled, it will not show any encryption settings.
Using AWS CLI:
To describe the ETL job and check if CloudWatch Logs encryption is enabled, run:
aws glue get-job --job-name <job-name> --query 'Job.Logging'
The output will show the logging configuration for the job, including CloudWatch Logs settings:
If CloudWatch Logs encryption is enabled, it will show the encryption mode, either SSE-KMS or SSE-S3.
Example output:
{
"Job": {
"Logging": {
"CloudWatchLogs": {
"Enabled": true,
"LogGroupName": "/aws/glue/jobs",
"LogStreamName": "my-etl-job-logs",
"EncryptionMode": "SSE-KMS",
"KmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/abcd1234"
}
}
}
}If EncryptionMode is missing or set to None, then CloudWatch Logs encryption is not enabled.
Implementation Steps:
Using AWS Console:
Sign in to the AWS Management Console and navigate to AWS Glue.
In the AWS Glue Dashboard, go to Jobs and select the ETL job to modify.
In the Logging section, ensure that CloudWatch Logs are enabled:
Enable CloudWatch Logs if they are not enabled.
In the Encryption Settings, ensure that SSE-KMS or SSE-S3 is selected for CloudWatch Logs encryption.
If using SSE-KMS, choose the KMS key to encrypt the logs.
Save the changes to apply CloudWatch Logs encryption.
Using AWS CLI:
To enable CloudWatch Logs encryption for an existing job, run:
aws glue update-job \
--job-name <job-name> \
--job-update '{"Logging": {"CloudWatchLogs": {"Enabled": true, "LogGroupName": "/aws/glue/jobs", "EncryptionMode": "SSE-KMS", "KmsKeyId": "<kms-key-id>"}}}'To verify the CloudWatch Logs encryption setting, run:
aws glue get-job --job-name <job-name> --query 'Job.Logging'
Ensure that the EncryptionMode is set to SSE-KMS or SSE-S3.
Backout Plan:
If enabling CloudWatch Logs encryption causes issues (e.g., job failures or access issues):
Identify the affected ETL job.
To disable CloudWatch Logs encryption, run:
aws glue update-job \
--job-name <job-name> \
--job-update '{"Logging": {"CloudWatchLogs": {"Enabled": true, "LogGroupName": "/aws/glue/jobs", "EncryptionMode": "None"}}}'Verify that CloudWatch Logs encryption has been disabled and that the job is functioning correctly.
Note:
KMS Key Management: If using SSE-KMS, ensure that the KMS key is managed properly, including rotation and access control.
Performance Testing: After enabling CloudWatch Logs encryption, test the ETL job to ensure there is no significant impact on job performance due to encryption.