Description

Ensure that Amazon EC2 Auto Scaling groups that have both a Classic Load Balancer and a target group attached have Capacity Rebalancing enabled. Groups without both attachments are not evaluated. With Capacity Rebalancing, the group responds to EC2 rebalance recommendations by launching a replacement Spot Instance before an at-risk Spot Instance is interrupted, and terminates the old instance once the new one is healthy.

Rationale

Without Capacity Rebalancing, the group only reacts to the two-minute Spot interruption notice, which is often not enough time to launch and register a replacement. Load balancer targets can disappear, causing timeouts, error spikes, growing backlogs and reduced availability. Enabling Capacity Rebalancing for groups that use Spot Instances, together with a capacity-aware allocation strategy, headroom below the maximum size and lifecycle hooks for draining, keeps capacity stable during Spot interruptions.

Impact

  • Replaces at-risk Spot Instances before they are interrupted, reducing capacity loss and errors

  • The group may briefly run more instances than the desired capacity (up to 10% above it) while replacements launch, which can add cost

  • Has no effect on On-Demand Instances in the group

Default Value

Capacity Rebalancing is turned off for Auto Scaling groups created with the AWS CLI or API unless it is explicitly enabled.

Pre-Requisites

  • IAM permissions: autoscaling:DescribeAutoScalingGroups and autoscaling:UpdateAutoScalingGroup

  • Confirmation that the workload tolerates instances being replaced, for example with lifecycle hooks or connection draining to finish in-flight work

Remediation

Test Plan

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon EC2 console and select Auto Scaling Groups under Auto Scaling.

  3. Select a group that is attached to a load balancer and open the Details tab.

  4. In the Allocation strategies (or instance purchase options) section, check whether Capacity rebalancing is turned on.

  5. Repeat for each group in each region in use.

  6. If Capacity Rebalancing is turned off, follow the implementation steps.

CLI:

Show the Capacity Rebalancing setting and load balancer attachments of each group:

aws autoscaling describe-auto-scaling-groups --region <REGION> --query "AutoScalingGroups[].{Name:AutoScalingGroupName,CapacityRebalance:CapacityRebalance,LoadBalancers:LoadBalancerNames,TargetGroups:TargetGroupARNs}" --output json

Only groups that have both a Classic Load Balancer (LoadBalancers) and a target group (TargetGroups) attached are evaluated. Such a group fails if CapacityRebalance is false.

Implementation Steps

Console:

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon EC2 console and select Auto Scaling Groups under Auto Scaling.

  3. Select the group and open the Details tab.

  4. In the Allocation strategies section, choose Edit.

  5. Select Capacity rebalancing.

  6. Choose Update.

  7. Repeat for each affected group in each region in use.

CLI:

Enable Capacity Rebalancing for the group:

aws autoscaling update-auto-scaling-group --region <REGION> --auto-scaling-group-name <AUTO_SCALING_GROUP_NAME> --capacity-rebalance

Backout Plan

  1. Sign in to the AWS Management Console at https://console.aws.amazon.com.

  2. Open the Amazon EC2 console and select Auto Scaling Groups under Auto Scaling.

  3. Select the group, open the Details tab and choose Edit in the Allocation strategies section.

  4. Clear Capacity rebalancing and choose Update.

CLI:

aws autoscaling update-auto-scaling-group --region <REGION> --auto-scaling-group-name <AUTO_SCALING_GROUP_NAME> --no-capacity-rebalance

Note: rolling back means Spot Instances are only replaced after they are interrupted, which can reduce capacity and availability.

Reference