Profile Applicability:
Level 2
Description:
Configuring persistent logs to be stored in Amazon S3 ensures that logs generated by your AWS resources are stored securely and persistently for long-term retention, analysis, and auditing. By setting up log configuration to send logs to S3, you can maintain a centralized repository for logs, enabling easy access and compliance with data retention policies.
Rationale:
Storing logs in S3 offers several advantages, including scalability, durability, and security. By configuring persistent logs to be sent to S3, you can ensure that logs are safely stored, easily accessible, and durable for compliance, troubleshooting, and security monitoring. Additionally, S3 allows you to configure lifecycle policies for automatic log archival or deletion based on your organization’s retention requirements.
Impact:
Pros:
Long-Term Retention: Logs stored in S3 are durable and accessible, ensuring compliance with retention policies.
Scalable Storage: S3 provides scalable storage, handling large volumes of logs without performance degradation.
Security: S3 offers encryption and access control, ensuring logs are stored securely.
Cost-Effective: You can use S3 lifecycle policies to archive or delete logs automatically, optimizing storage costs.
Cons:
Management Overhead: Requires managing access policies and lifecycle rules to ensure logs are stored securely and comply with data retention policies.
Costs: While S3 is cost-effective, storing large amounts of log data for long periods may incur storage charges.
Default Value:
By default, logs may not be configured to persist in S3. Many AWS services use CloudWatch or other temporary storage solutions, but logs need to be manually configured to be sent to S3 for persistent storage.
Pre-requisites:
IAM permissions to configure logging for your AWS resources and access to Amazon S3.
Access to an S3 bucket where the logs will be stored.
Access to AWS services (e.g., EC2, CloudTrail, CloudWatch) generating logs that need to be stored.
Remediation:
Test Plan:
Using the AWS Console:
Log in to the AWS Management Console.
Navigate to S3 and create a new bucket (or use an existing one) where the logs will be stored.
Go to the CloudWatch console (or other services that generate logs, such as EC2 or CloudTrail).
For CloudWatch Logs:
In the CloudWatch Console, navigate to Log groups.
Select the log group for which you want to store logs persistently.
Click on Actions and choose Export to S3.
Configure the export settings, including the S3 bucket and log stream prefix.
Ensure that the correct IAM roles are assigned to allow CloudWatch (or other services) to write logs to the selected S3 bucket.
Verify that logs are being stored in the configured S3 bucket by checking the bucket’s contents.
Test by generating logs (e.g., from CloudTrail or CloudWatch) and verifying that the logs appear in the S3 bucket.
Using AWS CLI:
List available S3 buckets to find the one where you will store logs:
aws s3 ls
To create a new S3 bucket for storing logs:
aws s3 mb s3://<your-bucket-name>
Export CloudWatch logs to the S3 bucket:
aws logs create-export-task --task-name <task-name> --log-group-name <log-group-name> --from <start-time> --to <end-time> --destination <s3-bucket-name> --destination-prefix <log-prefix>
Verify that logs are being exported by checking the S3 bucket:
aws s3 ls s3://<your-bucket-name>
Ensure proper IAM permissions are configured by checking the role policies:
aws iam get-role-policy --role-name <role-name> --policy-name <policy-name>
Implementation Plan:
Using the AWS Console:
Log in to the AWS Management Console.
Navigate to S3 and create a new bucket (if needed) for storing persistent logs.
Go to CloudWatch Logs (or the respective service generating logs) and select the Log group to export.
Click on Actions, choose Export to S3, and configure the export destination to the desired S3 bucket.
Review the configuration to ensure logs will be written securely to the S3 bucket.
Monitor the logs to confirm that data is being exported to the S3 bucket.
Using AWS CLI:
Create a new S3 bucket if needed:
aws s3 mb s3://<your-bucket-name>
Export CloudWatch logs to S3:
aws logs create-export-task --task-name <task-name> --log-group-name <log-group-name> --from <start-time> --to <end-time> --destination <s3-bucket-name> --destination-prefix <log-prefix>
Verify that logs are being written to the S3 bucket:
aws s3 ls s3://<your-bucket-name>
Backout Plan:
Using the AWS Console:
If logs are not being exported correctly or the configuration fails, go to the CloudWatch Logs console.
Delete the export task or modify the configuration.
Ensure that the IAM roles associated with CloudWatch have proper permissions to write to the S3 bucket.
Verify that the logs are either being re-exported or revert to a previous configuration if necessary.
Using AWS CLI:
If the export task fails, cancel the export task:
aws logs delete-export-task --task-id <task-id>
Reconfigure the export task or resolve any issues with IAM permissions to ensure that CloudWatch can export logs to S3 correctly.
Verify that logs are being written to the S3 bucket:
aws s3 ls s3://<your-bucket-name>