Lab · Task 3.2 · about 45 minutes
Target Tracking and Placement Limits
Services in this lab
- Amazon EC2in scope
- Amazon EC2 Auto Scalingin scope
- Amazon CloudWatchin scope
Lesson for this lab: High-Performing and Elastic Compute
Goal
Build an Auto Scaling group with a target tracking policy on average CPU, load its instances so it scales out until its maximum stops it, and see a cluster placement group refuse a burstable instance type.
Before you start
- An AWS account you can experiment in, ideally a sandbox account with a budget alert. Don't use the root user; sign in with an IAM Identity Center user or an IAM role that can manage EC2, EC2 Auto Scaling and CloudWatch.
- AWS CLI v2 configured for that identity, and one Region chosen for the whole lab (for example
us-east-1). Every command below uses the CLI's default Region. Run them all in one terminal session so the shell variables carry over. - The Region's default VPC, with a default subnet in at least two Availability Zones. The instances get the default security group and you never connect to them, so no inbound ports are opened.
Resources this lab creates (the clean-up list): one launch template, one Auto Scaling group with its instances, one target tracking scaling policy with the two CloudWatch alarms it manages, and one placement group.
Steps
0 of 7 steps done
VPC=$(aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query 'Vpcs[0].VpcId' --output text) SUBNETS=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC Name=default-for-az,Values=true \ --query 'sort_by(Subnets,&AvailabilityZone)[0:2].SubnetId' --output text | tr '\t' ',') echo $SUBNETSYou should see two subnet IDs separated by a comma.
The user data keeps every vCPU busy for 15 minutes after the instance starts, so each new instance adds load just like real traffic would. Detailed monitoring sends EC2 metrics every minute instead of every five, so the policy reacts faster.
cat > burn.sh <<'EOF' #!/bin/bash for i in $(seq $(nproc)); do timeout 900 sh -c 'while :; do :; done' & done EOF cat > lt.json <<EOF { "ImageId": "resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64", "InstanceType": "t3.micro", "Monitoring": { "Enabled": true }, "UserData": "$(base64 -w0 burn.sh)", "TagSpecifications": [{ "ResourceType": "instance", "Tags": [{ "Key": "Name", "Value": "lab32" }] }] } EOF aws ec2 create-launch-template --launch-template-name lab32-lt --launch-template-data file://lt.jsonOn macOS, use
base64 -i burn.shinstead ofbase64 -w0 burn.sh. Console: EC2 > Launch templates > Create launch template, choose Amazon Linux 2023 andt3.micro, and under Advanced details turn on detailed CloudWatch monitoring and paste the script into User data.Minimum 1, maximum 3, desired 1, across both subnets, with a 60-second default instance warmup so booting instances don't skew the group's metric.
aws autoscaling create-auto-scaling-group --auto-scaling-group-name lab32-asg \ --launch-template LaunchTemplateName=lab32-lt,Version='$Latest' \ --min-size 1 --max-size 3 --desired-capacity 1 \ --vpc-zone-identifier "$SUBNETS" \ --default-instance-warmup 60Console: EC2 > Auto Scaling groups > Create Auto Scaling group, choose
lab32-lt, the default VPC and two subnets, no load balancer, group size 1 / 1 / 3, and Enable default instance warmup at 60 seconds.cat > cpu50.json <<'EOF' { "TargetValue": 50.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "ASGAverageCPUUtilization" } } EOF aws autoscaling put-scaling-policy --auto-scaling-group-name lab32-asg \ --policy-name lab32-cpu50 --policy-type TargetTrackingScaling \ --target-tracking-configuration file://cpu50.jsonThe output lists two alarms whose names start with
TargetTracking-lab32-asg-AlarmHighand-AlarmLow. Auto Scaling created them for you; don't edit them.The first instance runs at about 100 percent CPU, double the target. Within a few minutes the high alarm fires and Auto Scaling raises the desired capacity in proportion. Each new instance also starts at 100 percent, so the group keeps asking for more.
while true; do aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names lab32-asg \ --query 'AutoScalingGroups[0].[DesiredCapacity,MaxSize,length(Instances)]' --output text sleep 30 doneThis loop works on Linux and macOS, where
watchisn't installed by default. Stop it with Ctrl-C.Then read the scaling history (console: the group's Activity tab):
aws autoscaling describe-scaling-activities --auto-scaling-group-name lab32-asg \ --query 'Activities[].[StartTime,StatusCode,Description]' --output tableDesired capacity climbs to 3 and stays there, even though average CPU is still far above 50 percent. That is the lesson's point about the maximum: target tracking never goes past the group's maximum size, so set the maximum from the peak you must serve, not from today's traffic.
EXPECTED FAILURE. Create a cluster placement group and try to launch the same
t3.microtype into it.aws ec2 create-placement-group --group-name lab32-cluster --strategy cluster FIRST_SUBNET=${SUBNETS%%,*} aws ec2 run-instances --launch-template LaunchTemplateName=lab32-lt \ --subnet-id $FIRST_SUBNET --placement GroupName=lab32-cluster \ --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=lab32-pg}]'The launch is refused with an error about the instance type and the placement group, and no instance is created. This is the trap from the lesson: cluster placement groups are for tightly coupled, high-throughput nodes and don't accept burstable T instances. A real HPC cluster would use one current-generation compute or accelerated type, launched in one request, in one Availability Zone.
If your Region does launch an instance here, note its ID and terminate it in clean-up step 2.
About 15 minutes after each instance started, its CPU load ends. Keep the polling loop from the previous step running: target tracking scales in more gradually than it scales out, so the desired capacity steps back toward 1 over the next 15 minutes or more. You can skip this and go straight to clean-up.
Verify
aws autoscaling describe-policies --auto-scaling-group-name lab32-asg --query 'ScalingPolicies[0].[PolicyType,TargetTrackingConfiguration.TargetValue]'returnsTargetTrackingScalingand50.0.aws cloudwatch describe-alarms --alarm-name-prefix TargetTracking-lab32-asg --query 'MetricAlarms[].AlarmName'lists the two managed alarms.describe-scaling-activitiesshows at least two successfulLaunching a new EC2 instanceactivities, and the group's desired capacity reached 3, its maximum.aws ec2 describe-instances --filters Name=tag:Name,Values=lab32-pg Name=instance-state-name,Values=pending,running --query 'Reservations[].Instances[].InstanceId'returns an empty list: nothing launched into the cluster placement group.
Clean up
Delete everything in this order.
-
Delete the scaling policy first; deleting a target tracking policy also deletes the two alarms it manages. Then delete the Auto Scaling group;
--force-deleteterminates its instances.aws autoscaling delete-policy --auto-scaling-group-name lab32-asg --policy-name lab32-cpu50 aws autoscaling delete-auto-scaling-group --auto-scaling-group-name lab32-asg --force-deleteConsole: EC2 > Auto Scaling groups, select
lab32-asg, Actions > Delete. -
If any instance launched in the placement-group step, terminate it and wait for it to finish. A placement group can't be deleted while it still holds instances.
aws ec2 terminate-instances --instance-ids <instance-id> aws ec2 wait instance-terminated --instance-ids <instance-id> -
Wait until the group is gone; this command returns an empty list once deletion finishes (it takes a few minutes):
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names lab32-asg --query 'AutoScalingGroups[].Status' -
Delete the placement group and the launch template:
aws ec2 delete-placement-group --group-name lab32-cluster aws ec2 delete-launch-template --launch-template-name lab32-lt -
Remove the local files:
rm burn.sh lt.json cpu50.json. -
Check that nothing billable remains. Each command should return an empty list (terminated instances may still be listed for a short while with state
terminated, which the filter excludes):aws ec2 describe-instances --filters Name=tag:Name,Values=lab32,lab32-pg Name=instance-state-name,Values=pending,running,stopping,stopped --query 'Reservations[].Instances[].InstanceId' aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names lab32-asg --query 'AutoScalingGroups[].AutoScalingGroupName' aws cloudwatch describe-alarms --alarm-name-prefix TargetTracking-lab32-asg --query 'MetricAlarms[].AlarmName' aws ec2 describe-launch-templates --filters Name=launch-template-name,Values=lab32-lt --query 'LaunchTemplates[].LaunchTemplateId'
Sources
- https://docs.aws.amazon.com/autoscaling/ec2/userguide/using-systems-manager-parameters.html
- https://docs.aws.amazon.com/autoscaling/ec2/userguide/enable-default-instance-warmup.html
- https://docs.aws.amazon.com/autoscaling/ec2/userguide/examples-scaling-policies.html
- https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scaling-target-tracking.html
- https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scale-based-on-demand.html
- https://docs.aws.amazon.com/autoscaling/ec2/userguide/enable-as-instance-metrics.html
- https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-process-shutdown.html
- https://docs.aws.amazon.com/autoscaling/ec2/userguide/deleting-scaling-policy.html
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-strategies.html
Last reviewed