Description
Ensure that Amazon EC2 Auto Scaling groups attached to both a Classic Load Balancer and a target group use Elastic Load Balancing (ELB) health checks instead of EC2 instance status checks only. With ELB health checks, the group also uses the load balancer's view of each instance's health, not only the EC2 instance status checks, to decide when an instance must be replaced. Groups that have only one kind of attachment (only a Classic Load Balancer, or only a target group) are not evaluated.
Rationale
With EC2 status checks only, an instance that is running but whose application no longer responds is still considered healthy. The group keeps it in service, the load balancer stops sending it traffic, and the remaining instances must handle the load, which reduces availability, increases error rates and wastes capacity. Enabling ELB health checks, with suitable load balancer probes and grace periods, lets the group replace instances that fail the load balancer's checks.
Impact
Unhealthy instances that fail load balancer checks are replaced automatically, improving availability
Instances that fail the load balancer health check are terminated and replaced, so poorly tuned probes or a short grace period can cause unnecessary replacements
No change is needed in clients or the application
Default Value
New Auto Scaling groups use EC2 health checks only. ELB health checks must be turned on explicitly.
Pre-Requisites
IAM permissions: autoscaling:DescribeAutoScalingGroups and autoscaling:UpdateAutoScalingGroup
Load balancer health checks that correctly reflect application health
A health check grace period long enough for new instances to start the application
Remediation
Test Plan
Console:
Sign in to the AWS Management Console at https://console.aws.amazon.com.
Open the Amazon EC2 console and select Auto Scaling Groups under Auto Scaling.
Select a group that is attached to both a Classic Load Balancer and a target group, and open the Details tab.
In the Health checks section, check whether Elastic Load Balancing (ELB) health checks are turned on.
Repeat for each group in each region in use.
If ELB health checks are not turned on, follow the implementation steps.
CLI:
Show the health check type and load balancer attachments of each group:
aws autoscaling describe-auto-scaling-groups --region <REGION> --query "AutoScalingGroups[].{Name:AutoScalingGroupName,HealthCheckType:HealthCheckType,LoadBalancers:LoadBalancerNames,TargetGroups:TargetGroupARNs}" --output json
Only groups that have both a Classic Load Balancer (LoadBalancers) and a target group (TargetGroups) attached are evaluated. Such a group fails if HealthCheckType does not include ELB.
Implementation Steps
Console:
Sign in to the AWS Management Console at https://console.aws.amazon.com.
Open the Amazon EC2 console and select Auto Scaling Groups under Auto Scaling.
Select the group and open the Details tab.
In the Health checks section, choose Edit.
Under Additional health check types, select Turn on Elastic Load Balancing health checks.
Check that the Health check grace period is long enough for new instances to start.
Choose Update.
Repeat for each affected group in each region in use.
CLI:
Turn on ELB health checks for the group:
aws autoscaling update-auto-scaling-group --region <REGION> --auto-scaling-group-name <AUTO_SCALING_GROUP_NAME> --health-check-type ELB
Backout Plan
Sign in to the AWS Management Console at https://console.aws.amazon.com.
Open the Amazon EC2 console and select Auto Scaling Groups under Auto Scaling.
Select the group, open the Details tab and choose Edit in the Health checks section.
Clear the Elastic Load Balancing health check option and choose Update.
CLI:
aws autoscaling update-auto-scaling-group --region <REGION> --auto-scaling-group-name <AUTO_SCALING_GROUP_NAME> --health-check-type EC2
Note: rolling back means instances that fail load balancer health checks stay in service and are not replaced.
Reference