Skip to content
Certification Guide

Lab · Task 2.2 · about 40 minutes

Force a Multi-AZ Database Failover and Watch the Standby Take Over

Services in this lab

  • Amazon RDSin scope

Lesson for this lab: High Availability and Disaster Recovery: Strategies, Failover and Single Points of Failure

Goal

Build a Multi-AZ DB instance, force a failover with a reboot, and observe that the standby replaces the primary behind the same endpoint name, while a DNS watch shows the address behind that name changing during the switch.

Before you start

Why this matters on the exam: questions about "survive an Availability Zone failure with no application change" point to a Multi-AZ DB instance deployment. The exam also tests what that standby is not: it cannot serve reads, and it does not protect against a Region outage. In this lab you check both.

You create one RDS DB instance with a Multi-AZ standby, plus a DB subnet group if your account has none. Nothing else is created. The lab assumes your Region has a default VPC; if it does not, pick two subnets in a VPC you own in step 1.

  • Use a sandbox AWS account with an AWS Budgets alert, not production. Sign in through IAM Identity Center or another role-based identity, never the root user.
  • Install the AWS CLI v2 and sign in (aws sso login or your usual role-based profile). Your identity needs rds:* permissions on resources named lab-* and secretsmanager:* for the managed master password. The DNS watch in step 6 uses dig, which comes with BIND utilities (dnsutils on Debian and Ubuntu, bind-utils on Amazon Linux and Fedora, preinstalled on macOS).
  • Work in one Region with at least two Availability Zones that support RDS. The commands use a Bash shell; set these variables first and keep the terminal open:
export AWS_REGION=<your-region>      # for example us-east-1
export AWS_DEFAULT_REGION=$AWS_REGION
export DB_ID=lab-orders-db
export SUBNET_GROUP=lab-orders-subnets   # or the name of an existing group that spans two zones

Steps

0 of 9 steps done

  1. A Multi-AZ DB instance needs a DB subnet group that spans at least two Availability Zones. List the subnets in your default VPC and note two IDs in different zones.

    Console: VPC > Subnets, filter by the default VPC, and note two subnets with different Availability Zone values.

    aws ec2 describe-subnets \
      --filters Name=default-for-az,Values=true \
      --query 'Subnets[].[SubnetId,AvailabilityZone]' --output table
    
  2. Skip this step if aws rds describe-db-subnet-groups already lists a group with subnets in two zones, and set SUBNET_GROUP to that group's name in the variables above. Otherwise create one with the two subnet IDs from the previous step. This step creates $SUBNET_GROUP, and the rest of the lab uses that variable.

    aws rds create-db-subnet-group \
      --db-subnet-group-name "$SUBNET_GROUP" \
      --db-subnet-group-description "Lab subnets in two Availability Zones" \
      --subnet-ids <subnet-id-az-a> <subnet-id-az-b>
    
  3. The --multi-az flag tells RDS to create a synchronous standby in a second Availability Zone. The lab uses a PostgreSQL instance that is not publicly accessible, and RDS stores the master password in Secrets Manager so you never type it.

    Console: RDS > Create database, choose Standard create, engine PostgreSQL, Multi-AZ DB instance, template Dev/Test, identifier lab-orders-db, Manage master credentials in AWS Secrets Manager, Public access No, and the subnet group you chose.

    aws rds create-db-instance \
      --db-instance-identifier "$DB_ID" \
      --db-instance-class db.t3.micro \
      --engine postgres \
      --allocated-storage 20 \
      --master-username labadmin \
      --manage-master-user-password \
      --multi-az \
      --no-publicly-accessible \
      --db-subnet-group-name "$SUBNET_GROUP"
    
    aws rds wait db-instance-available --db-instance-identifier "$DB_ID"
    

    Creation takes several minutes. Wait for the command to return before continuing.

  4. Ask RDS which zone holds the primary and which holds the standby, and what endpoint applications use.

    aws rds describe-db-instances --db-instance-identifier "$DB_ID" \
      --query 'DBInstances[0].[MultiAZ,AvailabilityZone,SecondaryAvailabilityZone,Endpoint.Address]' \
      --output table
    
    export ENDPOINT=$(aws rds describe-db-instances --db-instance-identifier "$DB_ID" \
      --query 'DBInstances[0].Endpoint.Address' --output text)
    

    Write down the values for AvailabilityZone, SecondaryAvailabilityZone and Endpoint.Address. Note that there is exactly one endpoint. There is no second endpoint for the standby, because the standby does not accept connections.

  5. The exam trap is treating the standby as extra read capacity. Ask RDS whether the instance has any reader you could connect to.

    aws rds describe-db-instances --db-instance-identifier "$DB_ID" \
      --query 'DBInstances[0].ReadReplicaDBInstanceIdentifiers'
    

    What you should see: []. A Multi-AZ DB instance has no reader that you can connect to, and the standby is kept only so RDS can fail over to it. If the stem needs read scaling, the answer is a read replica or a Multi-AZ DB cluster, not this deployment.

  6. Leave this running in the background. Every second it records the address the endpoint name resolves to, so you can see when the switch changes it. Run it in the same terminal as the other steps, because the variables are set there.

    ( for i in $(seq 1 900); do
        echo "$(date +%T) $(dig +short "$ENDPOINT" | tail -n 1)"
        sleep 1
      done > "$HOME/lab-orders-dns-watch.log" ) &
    echo $! > "$HOME/lab-orders-dns-watch.pid"
    
    sleep 5 && cat "$HOME/lab-orders-dns-watch.log"
    

    Expected: a few lines, each with a time and one IP address that does not change yet. The address can be a private one, which is normal for an instance that is not publicly accessible.

  7. EXPECTED FAILURE. This step deliberately interrupts the primary. Reboot the instance and choose failover. RDS stops the primary, promotes the standby in the other zone, and points the endpoint name at the new primary. Any application connection to the old primary is dropped, and the instance is unavailable for the length of the switch, which the lesson puts at 60 to 120 seconds.

    Record the current zone first. Right after the reboot call, the instance can still report available for a moment, so a plain wait could return before the switch starts. The loop polls until the zone changes, and only then waits for the instance to be available.

    export OLD_AZ=$(aws rds describe-db-instances --db-instance-identifier "$DB_ID" \
      --query 'DBInstances[0].AvailabilityZone' --output text)
    
    aws rds reboot-db-instance --db-instance-identifier "$DB_ID" --force-failover
    
    for i in $(seq 1 90); do
      [ "$(aws rds describe-db-instances --db-instance-identifier "$DB_ID" \
        --query 'DBInstances[0].AvailabilityZone' --output text)" != "$OLD_AZ" ] && break
      sleep 10
    done
    aws rds wait db-instance-available --db-instance-identifier "$DB_ID"
    

    Console equivalent: select the instance, Actions > Reboot, and choose Reboot with failover.

    What you should see: the loop ends once RDS reports the new zone (it gives up after 15 minutes; if it does, check the instance events in the console), and the wait returns when the instance is available again. In the DNS watch log from the previous step, the address changes at some point during the switch, while the endpoint name stays the same.

  8. Run the zone query from step 4 again. The failover can take several minutes to show in the console, so repeat it if the output is not yet updated.

    aws rds describe-db-instances --db-instance-identifier "$DB_ID" \
      --query 'DBInstances[0].[MultiAZ,AvailabilityZone,SecondaryAvailabilityZone,Endpoint.Address]' \
      --output table
    

    AvailabilityZone and SecondaryAvailabilityZone have swapped, and Endpoint.Address is the same value you wrote down. Applications did not need a new name, which is the "no application change" point in the lesson.

  9. RDS records why and when the failover happened. This is the log you check during a real incident.

    aws rds describe-events \
      --source-identifier "$DB_ID" --source-type db-instance \
      --duration 60 \
      --query 'Events[].[Date,Message]' --output table
    
    cat "$HOME/lab-orders-dns-watch.log"
    

    Look for the Multi-AZ failover entries in the events. The events show the failover reason and the time it completed.

    In the watch log, find the second when the address changed. That shows when DNS switched to the new primary. It does not measure the outage, which starts earlier, when the old primary stops. To measure downtime, you would run a loop that opens a database connection every second.

Verify

  1. The instance is Multi-AZ and available:
aws rds describe-db-instances --db-instance-identifier "$DB_ID" \
  --query 'DBInstances[0].[DBInstanceStatus,MultiAZ]' --output text

Expected: available and True.

  1. The zones swapped during the failover: AvailabilityZone and SecondaryAvailabilityZone from step 8 are reversed compared with step 4.

  2. The endpoint is unchanged: Endpoint.Address matches the value from step 4.

  3. The DNS watch log shows one address before the switch and a different one after it, and the endpoint name did not change. Check with awk 'NF == 2 {print $2}' "$HOME/lab-orders-dns-watch.log" | uniq, which should print two addresses. The NF == 2 filter skips lines where dig returned nothing during the switch.

  4. The events include a failover entry from the forced reboot in step 7.

Clean up

Stop the DNS watch first, then delete the instance, then the subnet group only if you created it in this lab. The instance has deletion protection off unless you turned it on with --deletion-protection, so this command does not need extra steps. Skip the final snapshot because this is a lab database.

kill "$(cat "$HOME/lab-orders-dns-watch.pid")" 2>/dev/null
rm -f "$HOME/lab-orders-dns-watch.pid"

aws rds delete-db-instance \
  --db-instance-identifier "$DB_ID" \
  --skip-final-snapshot \
  --delete-automated-backups

aws rds wait db-instance-deleted --db-instance-identifier "$DB_ID"

aws rds delete-db-subnet-group --db-subnet-group-name "$SUBNET_GROUP"

Delete the subnet group only if you created it in the "Create a DB subnet group" step. If the delete-db-subnet-group command reports that it is in use, wait a minute and run it again. The DNS watch log in your home directory holds no AWS resources, so you can keep it or delete it.

Console equivalent: RDS > Databases, select lab-orders-db, Actions > Delete, turn off Create final snapshot, and confirm. Then RDS > Subnet groups to delete lab-orders-subnets if you created it.

Confirm nothing is left:

aws rds describe-db-instances --query "DBInstances[?DBInstanceIdentifier=='$DB_ID']"
aws rds describe-db-subnet-groups --query "DBSubnetGroups[?DBSubnetGroupName=='$SUBNET_GROUP']"

Expected: [] for both. Also open Secrets Manager in this Region and check whether the secret for the master password (created because the lab used --manage-master-user-password) is still listed. If it is, delete it, so the lab leaves nothing behind. This lab created no other billable resources.

Sources

Last reviewed