Profile Applicability:
Level 1
Description:
In Amazon ECS, task definitions define how containers should run within a cluster. By default, ECS containers can be configured to use logging drivers, such as awslogs (CloudWatch Logs), for capturing logs generated by the container. This SOP ensures that ECS task definitions are configured to use non-blocking logging mode. Non-blocking logging ensures that log entries are asynchronously sent to CloudWatch, preventing potential delays or disruptions in container execution due to log write operations.
Using non-blocking mode helps ensure better performance and scalability by preventing the container from waiting for log data to be written before continuing its operations. It also reduces the risk of performance degradation under heavy load, as logging operations are separated from container processing.
Rationale:
The primary benefits of configuring non-blocking logging mode for ECS containers are:
Performance: Asynchronous logging prevents the container from being blocked during log writes, which ensures better performance, particularly under heavy logging workloads.
Reliability: Non-blocking mode ensures that the container’s operations are not delayed due to network or storage bottlenecks when writing logs.
Scalability: Asynchronous logging allows containers to scale without being hindered by logging throughput, which is essential in high-traffic or resource-intensive environments.
Improved Troubleshooting: Logs are still collected and available for analysis, but without blocking the main container functionality, helping you maintain operational speed.
Impact:
Pros:
Better Performance: By using non-blocking mode, log writes don’t interrupt container operations, improving overall application performance.
Reduced Latency: Containers can continue processing without being delayed by the log system, especially under high load conditions.
Scalable Logging: Allows your ECS tasks to scale more effectively without the overhead of waiting for synchronous log writes.
Minimal Disruptions: Containers continue their work even if there is an issue writing logs, ensuring application uptime.
Cons:
Log Latency: In rare cases, there may be a delay in log delivery due to asynchronous logging. However, this does not significantly impact the overall performance of the container.
Possible Log Gaps: If the container is terminated unexpectedly before the logs are successfully delivered, there may be a loss of log data, though this is generally rare and mitigated by CloudWatch Logs' internal buffering.
Default Value:
By default, ECS containers are configured to use blocking mode for log delivery unless explicitly configured otherwise in the task definition.
Pre-requisite:
AWS IAM Permissions:
ecs:DescribeTaskDefinitions
ecs:UpdateTaskDefinition
ecs:CreateTaskDefinition
logs:CreateLogGroup
logs:CreateLogStream
logs:PutLogEvents
AWS CLI installed and configured.
Basic knowledge of ECS task definitions, logging drivers, and CloudWatch Logs.
Remediation:
Test Plan:
Using AWS Console:
Sign in to the AWS Management Console.
Navigate to ECS under Services.
In the ECS Dashboard, go to Task Definitions and select the task definition you wish to inspect.
In the Container Definitions section, check the Log Configuration.
Ensure that the awslogs driver is being used.
Verify that the logConfiguration contains the setting for non-blocking mode ("logDriver": "awslogs" with "awslogs-create-group": "true").
If the configuration is not set to non-blocking mode, update the task definition.
Using AWS CLI:
To describe the ECS task definition and check the logConfiguration, run:
aws ecs describe-task-definition --task-definition <task-definition-name> --query 'taskDefinition.containerDefinitions[*].logConfiguration'
Ensure the logDriver is set to awslogs and check the options: Example output:
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-app-logs",
"awslogs-stream-prefix": "ecs",
"awslogs-region": "us-east-1",
"awslogs-create-group": "true"
}
}
}If the logConfiguration does not specify non-blocking mode, update the ECS task definition accordingly.
Implementation Steps:
Using AWS Console:
Sign in to the AWS Management Console and navigate to ECS.
In the ECS Dashboard, go to Task Definitions and select the task definition you want to update or create a new revision.
Under Container Definitions, select the container to configure.
In the Log Configuration section:
Set Log Driver to awslogs.
Add or verify the options:
"awslogs-create-group": "true"
"awslogs-group": "/ecs/my-app-logs"
"awslogs-stream-prefix": "ecs"
"awslogs-region": "us-east-1"
Save the changes and register the updated task definition.
Using AWS CLI:
To create a new ECS task definition with non-blocking log configuration, run:
aws ecs register-task-definition \
--family <task-family> \
--container-definitions '[{
"name": "my-container",
"image": "my-image",
"memory": 512,
"cpu": 256,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-app-logs",
"awslogs-stream-prefix": "ecs",
"awslogs-region": "us-east-1",
"awslogs-create-group": "true"
}
},
"essential": true
}]'To update an existing ECS task definition with non-blocking log configuration, use:
aws ecs update-task-definition \
--task-definition <task-definition-name> \
--container-definitions '[{
"name": "my-container",
"image": "my-image",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-app-logs",
"awslogs-stream-prefix": "ecs",
"awslogs-region": "us-east-1",
"awslogs-create-group": "true"
}
}
}]'Backout Plan:
If enabling non-blocking logging mode causes issues:
Identify the affected task definition and container.
Revert the logConfiguration settings by switching to blocking mode or adjusting the settings.
To deregister the task definition, run:
aws ecs deregister-task-definition --task-definition <task-definition-name>
Re-deploy using the previous working task definition.
Note:
Ensure that the CloudWatch Log Group exists or is created automatically as part of the ECS task definition configuration. Logs from containers will be directed to this Log Group for central management.
If performance degradation or excessive log data collection occurs, review the log retention policies in CloudWatch Logs and adjust them as needed to optimize storage costs.