Profile Applicability:
Level 1
Description:
Amazon ECS (Elastic Container Service) task definitions define how containers should be run within ECS. To enhance observability and troubleshooting, it is crucial to configure logging for each container in an ECS task definition. This SOP ensures that ECS task definition containers are configured to use appropriate logging drivers, such as awslogs (CloudWatch Logs), to capture container logs for monitoring and analysis. Logging provides vital insights into container behavior and aids in troubleshooting issues like application crashes or performance degradation.
Rationale:
Having a logging configuration in ECS task definitions is essential for:
Monitoring: Captures container logs for real-time monitoring of the application’s behavior.
Troubleshooting: Logs provide detailed information to diagnose issues, such as application errors or performance bottlenecks.
Security: Helps track unauthorized access or behavior anomalies within containers.
Compliance: Ensures proper auditing and compliance for containerized applications as part of security and monitoring best practices.
By enabling logs, especially in CloudWatch Logs, ECS containers will allow easy access to logs for debugging, performance analysis, and security monitoring.
Impact:
Pros:
Improved Observability: Logging allows real-time monitoring and retrospective troubleshooting.
Faster Issue Resolution: Logs provide detailed insights to quickly identify and resolve issues.
Compliance: Meeting regulatory and compliance requirements for logging, such as PCI-DSS, SOC 2, etc.
Security and Auditing: Helps in detecting malicious activities or unauthorized access by logging security events.
Cons:
Storage Costs: Log data stored in CloudWatch Logs or other services may incur additional costs based on the volume of logs.
Overhead: Excessive logging or incorrect configuration may lead to performance overhead or unnecessary data collection.
Default Value:
By default, ECS containers are not configured with a logging configuration unless explicitly defined in the ECS task definition.
Pre-requisite:
AWS IAM Permissions:
ecs:DescribeTaskDefinitions
ecs:CreateTaskDefinition
ecs:UpdateTaskDefinition
logs:CreateLogGroup
logs:CreateLogStream
logs:PutLogEvents
AWS CLI installed and configured.
Basic knowledge of ECS task definitions, logging drivers, and CloudWatch Logs.
Remediation:
Test Plan:
Using AWS Console:
Sign in to the AWS Management Console.
Navigate to ECS under Services.
In the ECS Dashboard, go to Task Definitions.
Select a task definition and review the container definitions to check if a logging configuration is set.
Ensure that awslogs is defined under logConfiguration for each container.
The configuration should specify the log driver as awslogs and include log group and log stream configurations.
Using AWS CLI:
To describe the ECS task definition and check if the logging configuration is set, run:
aws ecs describe-task-definition --task-definition <task-definition-name> --query 'taskDefinition.containerDefinitions[*].logConfiguration'
The output should contain the logConfiguration section, where the logDriver is set to awslogs. Example output:
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-container-logs",
"awslogs-stream-prefix": "ecs",
"awslogs-region": "us-east-1"
}
}
}If logConfiguration is missing or the logDriver is not awslogs, the task definition needs to be updated.
Implementation Steps:
Using AWS Console:
Sign in to the AWS Management Console and navigate to ECS.
In the ECS Dashboard, go to Task Definitions and select the task definition you want to configure or create a new revision.
Under Container Definitions, find the container that requires logging and click to edit.
In the Log Configuration section:
Set Log Driver to awslogs.
Specify the Log Group name (e.g., /ecs/my-app-logs).
Optionally, configure a Log Stream Prefix (e.g., ecs) to categorize logs.
Set the awslogs-region to the appropriate AWS region (e.g., us-east-1).
Save the changes to update the task definition and register a new revision.
Using AWS CLI:
To create a new ECS task definition with a logging configuration, run:
aws ecs register-task-definition \
--family <task-family> \
--container-definitions '[{
"name": "my-container",
"image": "my-image",
"memory": 512,
"cpu": 256,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-app-logs",
"awslogs-stream-prefix": "ecs",
"awslogs-region": "us-east-1"
}
},
"essential": true
}]'To update an existing task definition to include logging, use:
aws ecs update-task-definition \
--task-definition <task-definition-name> \
--container-definitions '[{
"name": "my-container",
"image": "my-image",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-app-logs",
"awslogs-stream-prefix": "ecs",
"awslogs-region": "us-east-1"
}
}
}]'Backout Plan:
If enabling logging causes issues (e.g., excessive logging or performance overhead):
Identify the affected ECS task definition.
Remove the logConfiguration settings from the task definition by updating it.
To deregister the task definition, run:
aws ecs deregister-task-definition --task-definition <task-definition-name>
Re-deploy the task definition without the logging configuration.
Note:
Ensure that CloudWatch Logs is set up in the appropriate region and that the Log Group exists before configuring it in the ECS task definition.
It's important to properly manage log retention policies in CloudWatch Logs to avoid excessive storage costs due to high log volumes.