Lab · Task 2.2 · about 40 minutes
Force a Multi-AZ Database Failover and Watch the Standby Take Over
Services in this lab
- Amazon RDSin scope
Lesson for this lab: High Availability and Disaster Recovery: Strategies, Failover and Single Points of Failure
Goal
Build a Multi-AZ DB instance, force a failover with a reboot, and observe that the standby replaces the primary behind the same endpoint name, while a DNS watch shows the address behind that name changing during the switch.
Before you start
Why this matters on the exam: questions about "survive an Availability Zone failure with no application change" point to a Multi-AZ DB instance deployment. The exam also tests what that standby is not: it cannot serve reads, and it does not protect against a Region outage. In this lab you check both.
You create one RDS DB instance with a Multi-AZ standby, plus a DB subnet group if your account has none. Nothing else is created. The lab assumes your Region has a default VPC; if it does not, pick two subnets in a VPC you own in step 1.
- Use a sandbox AWS account with an AWS Budgets alert, not production. Sign in through IAM Identity Center or another role-based identity, never the root user.
- Install the AWS CLI v2 and sign in (
aws sso loginor your usual role-based profile). Your identity needsrds:*permissions on resources namedlab-*andsecretsmanager:*for the managed master password. The DNS watch in step 6 usesdig, which comes with BIND utilities (dnsutilson Debian and Ubuntu,bind-utilson Amazon Linux and Fedora, preinstalled on macOS). - Work in one Region with at least two Availability Zones that support RDS. The commands use a Bash shell; set these variables first and keep the terminal open:
export AWS_REGION=<your-region> # for example us-east-1
export AWS_DEFAULT_REGION=$AWS_REGION
export DB_ID=lab-orders-db
export SUBNET_GROUP=lab-orders-subnets # or the name of an existing group that spans two zones
Steps
0 of 9 steps done
A Multi-AZ DB instance needs a DB subnet group that spans at least two Availability Zones. List the subnets in your default VPC and note two IDs in different zones.
Console: VPC > Subnets, filter by the default VPC, and note two subnets with different Availability Zone values.
aws ec2 describe-subnets \ --filters Name=default-for-az,Values=true \ --query 'Subnets[].[SubnetId,AvailabilityZone]' --output tableSkip this step if
aws rds describe-db-subnet-groupsalready lists a group with subnets in two zones, and setSUBNET_GROUPto that group's name in the variables above. Otherwise create one with the two subnet IDs from the previous step. This step creates$SUBNET_GROUP, and the rest of the lab uses that variable.aws rds create-db-subnet-group \ --db-subnet-group-name "$SUBNET_GROUP" \ --db-subnet-group-description "Lab subnets in two Availability Zones" \ --subnet-ids <subnet-id-az-a> <subnet-id-az-b>The
--multi-azflag tells RDS to create a synchronous standby in a second Availability Zone. The lab uses a PostgreSQL instance that is not publicly accessible, and RDS stores the master password in Secrets Manager so you never type it.Console: RDS > Create database, choose Standard create, engine PostgreSQL, Multi-AZ DB instance, template Dev/Test, identifier
lab-orders-db, Manage master credentials in AWS Secrets Manager, Public access No, and the subnet group you chose.aws rds create-db-instance \ --db-instance-identifier "$DB_ID" \ --db-instance-class db.t3.micro \ --engine postgres \ --allocated-storage 20 \ --master-username labadmin \ --manage-master-user-password \ --multi-az \ --no-publicly-accessible \ --db-subnet-group-name "$SUBNET_GROUP" aws rds wait db-instance-available --db-instance-identifier "$DB_ID"Creation takes several minutes. Wait for the command to return before continuing.
Ask RDS which zone holds the primary and which holds the standby, and what endpoint applications use.
aws rds describe-db-instances --db-instance-identifier "$DB_ID" \ --query 'DBInstances[0].[MultiAZ,AvailabilityZone,SecondaryAvailabilityZone,Endpoint.Address]' \ --output table export ENDPOINT=$(aws rds describe-db-instances --db-instance-identifier "$DB_ID" \ --query 'DBInstances[0].Endpoint.Address' --output text)Write down the values for
AvailabilityZone,SecondaryAvailabilityZoneandEndpoint.Address. Note that there is exactly one endpoint. There is no second endpoint for the standby, because the standby does not accept connections.The exam trap is treating the standby as extra read capacity. Ask RDS whether the instance has any reader you could connect to.
aws rds describe-db-instances --db-instance-identifier "$DB_ID" \ --query 'DBInstances[0].ReadReplicaDBInstanceIdentifiers'What you should see:
[]. A Multi-AZ DB instance has no reader that you can connect to, and the standby is kept only so RDS can fail over to it. If the stem needs read scaling, the answer is a read replica or a Multi-AZ DB cluster, not this deployment.Leave this running in the background. Every second it records the address the endpoint name resolves to, so you can see when the switch changes it. Run it in the same terminal as the other steps, because the variables are set there.
( for i in $(seq 1 900); do echo "$(date +%T) $(dig +short "$ENDPOINT" | tail -n 1)" sleep 1 done > "$HOME/lab-orders-dns-watch.log" ) & echo $! > "$HOME/lab-orders-dns-watch.pid" sleep 5 && cat "$HOME/lab-orders-dns-watch.log"Expected: a few lines, each with a time and one IP address that does not change yet. The address can be a private one, which is normal for an instance that is not publicly accessible.
EXPECTED FAILURE. This step deliberately interrupts the primary. Reboot the instance and choose failover. RDS stops the primary, promotes the standby in the other zone, and points the endpoint name at the new primary. Any application connection to the old primary is dropped, and the instance is unavailable for the length of the switch, which the lesson puts at 60 to 120 seconds.
Record the current zone first. Right after the reboot call, the instance can still report
availablefor a moment, so a plain wait could return before the switch starts. The loop polls until the zone changes, and only then waits for the instance to be available.export OLD_AZ=$(aws rds describe-db-instances --db-instance-identifier "$DB_ID" \ --query 'DBInstances[0].AvailabilityZone' --output text) aws rds reboot-db-instance --db-instance-identifier "$DB_ID" --force-failover for i in $(seq 1 90); do [ "$(aws rds describe-db-instances --db-instance-identifier "$DB_ID" \ --query 'DBInstances[0].AvailabilityZone' --output text)" != "$OLD_AZ" ] && break sleep 10 done aws rds wait db-instance-available --db-instance-identifier "$DB_ID"Console equivalent: select the instance, Actions > Reboot, and choose Reboot with failover.
What you should see: the loop ends once RDS reports the new zone (it gives up after 15 minutes; if it does, check the instance events in the console), and the wait returns when the instance is available again. In the DNS watch log from the previous step, the address changes at some point during the switch, while the endpoint name stays the same.
Run the zone query from step 4 again. The failover can take several minutes to show in the console, so repeat it if the output is not yet updated.
aws rds describe-db-instances --db-instance-identifier "$DB_ID" \ --query 'DBInstances[0].[MultiAZ,AvailabilityZone,SecondaryAvailabilityZone,Endpoint.Address]' \ --output tableAvailabilityZoneandSecondaryAvailabilityZonehave swapped, andEndpoint.Addressis the same value you wrote down. Applications did not need a new name, which is the "no application change" point in the lesson.RDS records why and when the failover happened. This is the log you check during a real incident.
aws rds describe-events \ --source-identifier "$DB_ID" --source-type db-instance \ --duration 60 \ --query 'Events[].[Date,Message]' --output table cat "$HOME/lab-orders-dns-watch.log"Look for the Multi-AZ failover entries in the events. The events show the failover reason and the time it completed.
In the watch log, find the second when the address changed. That shows when DNS switched to the new primary. It does not measure the outage, which starts earlier, when the old primary stops. To measure downtime, you would run a loop that opens a database connection every second.
Verify
- The instance is Multi-AZ and available:
aws rds describe-db-instances --db-instance-identifier "$DB_ID" \
--query 'DBInstances[0].[DBInstanceStatus,MultiAZ]' --output text
Expected: available and True.
-
The zones swapped during the failover:
AvailabilityZoneandSecondaryAvailabilityZonefrom step 8 are reversed compared with step 4. -
The endpoint is unchanged:
Endpoint.Addressmatches the value from step 4. -
The DNS watch log shows one address before the switch and a different one after it, and the endpoint name did not change. Check with
awk 'NF == 2 {print $2}' "$HOME/lab-orders-dns-watch.log" | uniq, which should print two addresses. TheNF == 2filter skips lines wheredigreturned nothing during the switch. -
The events include a failover entry from the forced reboot in step 7.
Clean up
Stop the DNS watch first, then delete the instance, then the subnet group only if you created it in this lab. The instance has deletion protection off unless you turned it on with --deletion-protection, so this command does not need extra steps. Skip the final snapshot because this is a lab database.
kill "$(cat "$HOME/lab-orders-dns-watch.pid")" 2>/dev/null
rm -f "$HOME/lab-orders-dns-watch.pid"
aws rds delete-db-instance \
--db-instance-identifier "$DB_ID" \
--skip-final-snapshot \
--delete-automated-backups
aws rds wait db-instance-deleted --db-instance-identifier "$DB_ID"
aws rds delete-db-subnet-group --db-subnet-group-name "$SUBNET_GROUP"
Delete the subnet group only if you created it in the "Create a DB subnet group" step. If the delete-db-subnet-group command reports that it is in use, wait a minute and run it again. The DNS watch log in your home directory holds no AWS resources, so you can keep it or delete it.
Console equivalent: RDS > Databases, select lab-orders-db, Actions > Delete, turn off Create final snapshot, and confirm. Then RDS > Subnet groups to delete lab-orders-subnets if you created it.
Confirm nothing is left:
aws rds describe-db-instances --query "DBInstances[?DBInstanceIdentifier=='$DB_ID']"
aws rds describe-db-subnet-groups --query "DBSubnetGroups[?DBSubnetGroupName=='$SUBNET_GROUP']"
Expected: [] for both. Also open Secrets Manager in this Region and check whether the secret for the master password (created because the lab used --manage-master-user-password) is still listed. If it is, delete it, so the lab leaves nothing behind. This lab created no other billable resources.
Sources
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.Failover.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RebootInstance.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html
- https://docs.aws.amazon.com/cli/latest/reference/rds/create-db-instance.html
- https://docs.aws.amazon.com/cli/latest/reference/rds/reboot-db-instance.html
- https://docs.aws.amazon.com/cli/latest/reference/rds/delete-db-instance.html
Last reviewed