Skip to content
Certification Guide

Task 2.2 · 22 min read

High Availability and Disaster Recovery: Strategies, Failover and Single Points of Failure

Services in this lesson

  • Amazon Route 53in scope
  • Elastic Load Balancing (ELB)in scope
  • Amazon EC2 Auto Scalingin scope
  • Amazon RDSin scope
  • Amazon Aurorain scope
  • Amazon S3in scope
  • Amazon EBSin scope
  • Amazon EFSin scope

What the exam tests

  • Matching a disaster recovery (DR) strategy to the stated RPO and RTO targets, and choosing the cheapest strategy that meets both.
  • Telling a data center failure (solved by Multi-AZ) from a Region failure (which needs a second Region), and spotting which one the stem asks about.
  • Choosing the failover mechanism: Multi-AZ failover for databases, and health-checked Route 53 failover records between Regions.
  • Removing single points of failure, and making a standby Region able to take production load: quotas raised, and servers rebuilt from immutable images rather than patched.
  • Choosing storage that survives failure, and picking RDS Proxy to handle connections during a database failover.

The core concept

Availability has two layers. Copies in several Availability Zones survive a data center failure, and a copy in a second Region survives losing the whole Region.

A Region is a geographic area, such as Northern Virginia. An Availability Zone (AZ) is one or more separate data centers inside a Region. The second Region, kept ready to take over, is the recovery Region.

An analogy: a bakery chain

Picture a bakery with two shops across town, each on its own power line. When one shop's power fails, the other keeps bread coming. That's Multi-AZ.

A storm can black out the whole town, though. That's the Region-level event. The owner's options for it, from cheapest to most expensive, are:

  • Recipes in a safe in the next town. Nothing is open there, so rebuilding a shop takes hours. That's backup and restore.
  • A rented shop in the next town with stocked fridges but cold ovens. Someone has to fire up the ovens first. That's pilot light.
  • A small shop in the next town, open with one baker. It sells bread right away and hires more staff as crowds arrive. That's warm standby.
  • Two full shops, both selling every day. Customers already go to both. That's multi-site active/active.

Where it breaks: the top three tiers replicate every change continuously and asynchronously, so RPO stays above zero. Only backup and restore copies now and then.

Walk one booking through it

The diagram shows both layers. The walkthrough follows one booking through a normal day, then through two different failures.

Enlarge
Multi-AZ protects against a data center failure inside the Region. The recovery Region protects against losing the whole Region.

Fernbrook Clinic runs a booking app on AWS. Hana is the engineer on call, and Diego is a patient booking a check-up.

  1. Diego opens the booking site. His browser asks Amazon Route 53 for the site's address. Route 53 is the AWS DNS (Domain Name System) service.
    • Route 53 holds a failover record: a primary answer, a secondary answer, and a health check that tests the primary.
    • Failover means switching traffic to the standby copy.
  2. The primary health check passes. Route 53 answers with the primary Region's Application Load Balancer (ALB), which spreads requests across servers.
  3. The ALB sends the request to an app copy. Copies run in AZ a and AZ b. They sit in an Auto Scaling group: a fleet of Amazon EC2 instances (virtual servers) that replaces failed ones.
  4. The app saves Diego's booking to an Amazon RDS (Relational Database Service) Multi-AZ DB instance deployment. RDS copies the write synchronously, before confirming it, to a standby in another AZ.
  5. The booking also travels to the recovery Region. Continuous replication copies it to a cross-Region replica, a read-only copy of the database. The recovery Region's app servers are defined in code but switched off: a pilot light.
  6. AZ a loses power. The app copies in AZ b keep serving. If the database primary was in AZ a, RDS fails over to the standby behind the same DNS name. Route 53 changes nothing, because the Region is still healthy.
  7. Later, the whole primary Region fails. The primary health check fails. Hana starts the recovery runbook, a written, tested list of recovery steps.
  8. The runbook brings the recovery Region up. It promotes the replica to a writable database and starts the app servers. Route 53 then answers with the secondary record, once the recovery load balancer reports healthy.
  9. Diego can book again. The time until the site works again is the RTO. Bookings not yet replicated are lost, and that gap is the RPO.

The wait while app servers start is why warm standby, with servers already running, gives a shorter RTO. Step 6 needed no second Region. Steps 7 to 9 did, and they cost a second copy of the data running all year.

The rules the exam tests

  • RPO (recovery point objective) is how much data you can lose, measured as time since the last good copy.
  • RTO (recovery time objective) is how long the service can be down.
  • Multi-AZ covers a data center. Most workloads need only this layer.
  • A second Region is a deliberate, expensive choice. The stem has to justify it with a Region-level event.
  • Four DR strategies, cheapest first: backup and restore, pilot light, warm standby, multi-site active/active. Each step up costs more and lowers RTO and RPO.
  • Pick the cheapest strategy that meets both stated targets. Don't build for the worst case you can imagine.
Deep diveControl plane and data plane during a failover

The control plane is the part of a service that creates and changes resources. The data plane is the part that does the running work, such as answering DNS queries.

Use only data plane operations during a failover. The data plane has higher availability goals than the control plane.

Scaling an Auto Scaling group is a control plane action. Changing the weight on a Route 53 weighted record is too. Each one adds a dependency you may not want during a disaster.

A warm standby that scales out through Auto Scaling depends on the control plane. Pre-scaling enough capacity before the disaster avoids that dependency.

Deep diveAutomatic or manual failover

Failover is either automatic or manual. Automatic failover, driven by health checks, is fast.

Automatic failover can also trigger on a false alarm. A failover you didn't need still interrupts the service.

Many teams choose a manual trigger with an automated runbook. One action by a person then starts a tested sequence of steps.

The walkthrough used a pilot light, but the stem's numbers decide which of the four strategies fits. Each one answers two questions: what already runs in the recovery Region, and how much must happen before it serves traffic.

The four strategies side by side

  • Backup and restore runs nothing in the recovery Region. You rebuild the infrastructure from code and restore the data.
  • Pilot light keeps only the data layer running. The application servers must start before it can take traffic.
  • Warm standby keeps a scaled-down but complete copy running. It serves reduced traffic at once, then scales out.
  • Multi-site active/active serves traffic from every Region at once. RTO is close to zero, but you must handle writes that land in two places.

An active/active design still needs backups. Replication can carry corruption or deletes to the other Region, so keep versioning or point-in-time backups.

Pilot light and warm standby are both active/passive: one Region serves, and the other waits.

Decision guide

Decision guide
If the question says…Reach for…Because…
"survive a data center or Availability Zone failure", no Region-level requirementMulti-AZ deployment (Auto Scaling group across AZs, Multi-AZ database)Copies in other zones keep serving. A second Region costs more for a risk not named
"RPO in hours, RTO up to 24 hours", "lowest cost", "data must survive a Region loss"Backup and restore, with backups copied to the recovery Region (for example with AWS Backup)Nothing runs there, so it's cheapest. Rebuilding from code and restoring is slowest
"RPO in minutes", "RTO in tens of minutes", "keep the core database always replicated"Pilot light: replicated data always on, application servers defined in code but switched offReplication keeps data current, so only compute has to start
"RPO in seconds", "RTO in minutes", "the standby must handle traffic right away at reduced capacity"Warm standby: scaled-down, fully running copy in the recovery RegionThe stack is already live, so recovery is a scale-out, not a rebuild
"RTO near zero", "every Region serves users", "global writes must stay consistent"Multi-site active/active with a data layer that handles cross-Region writesEvery Region already serves. The hard part is replication and write conflicts
"Region-level database failover with data loss measured in seconds"Amazon Aurora global database, with a secondary cluster promoted to primaryReplication lag is typically under a second. Plan for recovery in minutes
"failover from one Region to another, controlled by DNS, with health checks"Route 53 failover routing: a primary record and a secondary record, each tied to a health checkRoute 53 answers with the secondary when the primary fails, on the DNS data plane
"database failover with no application change, the endpoint must stay the same"RDS Multi-AZ DB instance deploymentRDS points the same DNS name at the standby, typically within 60 to 120 seconds
"many short-lived connections spike during failover", "too many connections", Lambda to databaseAmazon RDS Proxy in front of the databaseIt holds client connections during failover, and pooling shares few database connections among many clients
"recovery Region drifted from production after hand patches", "rebuild the same servers in another Region"Immutable infrastructure: launch replacements from a tested, versioned image and never patch in placeEvery replacement starts from one image, so the Regions match and drift can't build
"data must survive the loss of an Availability Zone", "shared files for a fleet across zones"S3 (except One Zone-IA) for objects, or a Regional EFS file system for shared filesBoth store data across multiple AZs, so there's no standby to build
"restore a failed EBS volume in another zone or Region"EBS snapshots, copied to the recovery RegionA volume lives in one AZ. Its snapshot can be copied to another Region
"copy data to another Region continuously", "recovery Region needs current objects"S3 Cross-Region Replication (versioning on both buckets)Objects copy as they're written. It isn't a backup: version-specific deletes don't copy

Terms in the table:

  • AWS Backup is the AWS service that schedules and copies backups.
  • AWS Lambda runs code without servers. Each Lambda execution environment opens its own database connections, so many concurrent invocations can exhaust the database's connection limit.
  • Connection pooling keeps a small set of database connections open and shares them among many clients.
  • Amazon S3 (Simple Storage Service) stores objects. S3 One Zone-IA is a storage class kept in a single AZ.
  • Amazon EFS (Elastic File System) is a shared file system. Amazon EBS (Elastic Block Store) is a disk that normally attaches to one instance.
Deep diveCaveats behind the decision guide rows

An Aurora global database spans Regions. The Aurora FAQ says a secondary can be promoted in under a minute. The user guide frames recovery time in minutes, so plan for minutes.

An EBS snapshot is stored durably. You can copy it to another Region and create new volumes from it there.

Correct pattern

REFERENCE PATTERN: Route 53 active-passive failover between Regions. Two alias records share one name. An alias record points at an AWS resource, such as a load balancer.

Each record has a SetIdentifier, and the primary record references a health check. Route 53 answers with the secondary record only when the primary's health check fails. On an alias record, Evaluate Target Health or a health check can drive failover.

The names and IDs below are placeholders.

PrimaryRecord:
  Type: AWS::Route53::RecordSet
  Properties:
    HostedZoneId: <your-hosted-zone-id>
    Name: app.<your-domain>
    Type: A
    SetIdentifier: primary-region
    Failover: PRIMARY
    HealthCheckId: !Ref PrimaryHealthCheck
    AliasTarget:
      DNSName: <primary-alb-dns-name>
      HostedZoneId: <primary-alb-hosted-zone-id>
      EvaluateTargetHealth: true

SecondaryRecord:
  Type: AWS::Route53::RecordSet
  Properties:
    HostedZoneId: <your-hosted-zone-id>
    Name: app.<your-domain>
    Type: A
    SetIdentifier: recovery-region
    Failover: SECONDARY
    AliasTarget:
      DNSName: <recovery-alb-dns-name>
      HostedZoneId: <recovery-alb-hosted-zone-id>
      EvaluateTargetHealth: true

REFERENCE PATTERN: the standby must be able to take load. Define the recovery Region in the same template as the primary, with a scaled-down desired capacity. Set the Auto Scaling group maximum to the production size so that scaling out is possible.

RecoveryAsg:
  Type: AWS::AutoScaling::AutoScalingGroup
  Properties:
    MinSize: "2" # scaled-down, but across two Availability Zones
    MaxSize: "12" # production capacity, which must fit within the Region's quotas
    DesiredCapacity: "2"
    VPCZoneIdentifier: [!Ref RecoverySubnetA, !Ref RecoverySubnetB]

Raise the recovery Region's service quotas (account limits, such as EC2 vCPU limits) to production size before you need them. A standby that can't scale past its quota is a warm standby in name only.

Traps

AWS CloudFormation builds resources from a template you write as code. The fourth trap relies on it.

TRAPExam traps
The trapWhy it failsCorrect pattern
Multi-Region active/active for a workload whose requirement is only "survive an AZ failure"Looks most resilient, but multiplies cost and data-sync work for a risk never raisedMulti-AZ inside one Region; add a recovery Region only when the stem names a Region-level event or a regulation that requires one
Counting a Multi-AZ standby as capacity for readsA Multi-AZ DB instance standby serves no reads. It exists only for failoverMulti-AZ DB instance for failover; read replicas or a Multi-AZ DB cluster (a cluster with two readable standbys) for read scaling (details in 3.3)
Assuming a pilot light can take traffic the moment it is neededNo app servers run. Starting them, deploying code and warming caches is your RTOPick warm standby if the RTO is minutes; keep pilot light for RTOs in the tens of minutes and rehearse the start-up steps
Backups in the recovery Region with no infrastructure codeData is half of recovery. Rebuilding the environment by hand can outlast the RTODefine the stack in CloudFormation, back up configuration and machine images, and test restores on a schedule
Route 53 weighted records changed by hand to move all traffic to the recovery RegionChanging weights is a control plane action. If that plane is impaired, nothing changesFailover routing records with health checks for automatic switching, plus a runbook that a person starts for a deliberate switch
Sizing the standby Region at production scale with no reason in the stemDoubles cost for an RTO nobody asked for. Over-building is the most common DR mistakeChoose the lowest strategy whose RPO and RTO fit the stated numbers; check the stem before you add capacity
Calling RDS Proxy a replacement for Multi-AZThe proxy pools connections but stores no data copy, so it has nothing to fail over toUse Multi-AZ for the copy and RDS Proxy for connection handling during failover

Compare

Compare
AspectBackup and restorePilot lightWarm standbyMulti-site active/active
Typical RPOHours (minutes with point-in-time recovery)MinutesSecondsNear zero
Typical RTOUp to 24 hoursTens of minutesMinutesPotentially zero
What runs in the recovery RegionNothing; backups onlyReplicated data storesFull stack at reduced capacityFull stack serving traffic
Application servers at failoverBuilt and deployed from codeStarted and scaledScaled outAlready serving
Relative costLowestLowModerate to highHighest
Compare
AspectMulti-AZ DB instanceAurora global databaseRead replica (cross-Region)
PurposeFailover within one RegionFailover across RegionsRead scaling, and a manual promotion target
ReplicationSynchronous to a standby in another AZStorage-level, typically under a second lagAsynchronous
Serves reads from standbyNoSecondary clusters serve readsYes, as a read replica
Typical failover time60 to 120 seconds (automatic)Under a minute to promote a secondary (Aurora FAQ); the user guide frames RTO in minutesManual promotion; takes several minutes or longer
Application changeNone; the DNS endpoint stays the sameNone with the global writer endpoint (Aurora FAQ)Endpoint change for reads

Worked exam scenario

Exam scenario

A claims company runs its web tier on Amazon EC2 instances in an Auto Scaling group behind an Application Load Balancer and stores claims in an Amazon RDS for PostgreSQL database. The business requires that the service recover from an Availability Zone outage within 5 minutes with no data loss, and that it stay available if the entire primary Region becomes unavailable, with an RTO of 10 minutes and an RPO of about 1 minute for the Region-level event. The standby Region must take traffic as soon as DNS fails over, even at reduced capacity. The team wants the lowest cost that meets these targets. Which solution meets these requirements MOST cost-effectively?

  1. A.Run the web tier and database at full capacity in two Regions with active/active writes and a global load balancer that splits traffic evenly.
  2. B.Deploy the web tier and the database across two Availability Zones in one Region, and back up the database to Amazon S3 once a day.
  3. C.Deploy the database as Multi-AZ in the primary Region, replicate it continuously to a cross-Region read replica that is promoted on failover, keep the web tier as a scaled-down Auto Scaling group in the recovery Region, and use a health-checked Route 53 failover record.
  4. D.Use a Multi-AZ database and Auto Scaling group in one Region, and restore a Region-level backup to a new Region after a disaster.
Show the walkthrough
  1. Split the requirements. The stem names two failure levels. An AZ outage needs recovery within 5 minutes with no data loss. A whole-Region event needs an RTO of 10 minutes and an RPO of about 1 minute.
  2. Note what that rules out. Each level needs a different mechanism, so a one-Region design can't cover both.
  3. Eliminate A. Full active/active in two Regions meets the targets, but at the highest cost. Its near-zero RTO is more than the business asked for.
  4. Eliminate B. Multi-AZ covers the AZ outage. A daily backup, though, can lose up to a day of data, and it doesn't cover a Region event.
  5. Eliminate D. Restoring a backup in a new Region is backup and restore. Its RPO and RTO are measured in hours, so it misses both Region targets.
  6. Choose C. Multi-AZ database failover covers the AZ outage within minutes. A continuously replicated cross-Region copy keeps data loss within about a minute while replication keeps up.
  7. Confirm the cost. The scaled-down web tier behind a health-checked Route 53 failover record is a warm standby. It is already serving, so it fits a 10-minute RTO at moderate cost.
  8. Check for anything cheaper. A pilot light would cost less, but its servers start from nothing, so recovery takes tens of minutes. That misses a 10-minute RTO, so C is the cheapest design that meets both targets.

Answer: C

Check your understanding

Question 1: A workload must keep serving requests in a second Region during a Region outage, with the standby taking traffic within minutes at reduced capacity. What is the best fit?
Question 2: Which TWO statements about Multi-AZ DB instance deployments are correct? (Choose TWO.)
Question 3: A company's failover runbook changes Route 53 weighted records to send all traffic to the recovery Region. During a regional disaster the change is slow. What should the architect change?

Build it

The lab for this task, Force a Multi-AZ database failover, builds a Multi-AZ DB instance and forces a failover. You confirm that the standby takes over behind the same endpoint. You also check this lesson's trap: the standby can't serve reads.

Key takeaways

  • Start from the stated RPO and RTO. Pick the cheapest DR strategy that meets both, and don't over-build.
  • Multi-AZ handles a data center failure in one Region. Add a recovery Region only when the requirement names a Region-level event.
  • Backup and restore is cheapest and slowest. Pilot light keeps data running, warm standby keeps the whole stack running, and active/active serves from every Region.
  • Automate failover with Route 53 failover records and health checks, and use only data plane operations during recovery.
  • Make the recovery Region able to take production load: define it as code, and raise its service quotas before you need them.
  • Match the recovery strategy to the stated RPO and RTO, not the worst case: Multi-AZ covers a data center, and only a Region-level loss needs a second Region.

Hands-on lab: Force a Multi-AZ Database Failover and Watch the Standby Take Over

Sources

Last reviewed