Task 2.2 · 22 min read
High Availability and Disaster Recovery: Strategies, Failover and Single Points of Failure
Services in this lesson
- Amazon Route 53in scope
- Elastic Load Balancing (ELB)in scope
- Amazon EC2 Auto Scalingin scope
- Amazon RDSin scope
- Amazon Aurorain scope
- Amazon S3in scope
- Amazon EBSin scope
- Amazon EFSin scope
What the exam tests
- Matching a disaster recovery (DR) strategy to the stated RPO and RTO targets, and choosing the cheapest strategy that meets both.
- Telling a data center failure (solved by Multi-AZ) from a Region failure (which needs a second Region), and spotting which one the stem asks about.
- Choosing the failover mechanism: Multi-AZ failover for databases, and health-checked Route 53 failover records between Regions.
- Removing single points of failure, and making a standby Region able to take production load: quotas raised, and servers rebuilt from immutable images rather than patched.
- Choosing storage that survives failure, and picking RDS Proxy to handle connections during a database failover.
The core concept
Availability has two layers. Copies in several Availability Zones survive a data center failure, and a copy in a second Region survives losing the whole Region.
A Region is a geographic area, such as Northern Virginia. An Availability Zone (AZ) is one or more separate data centers inside a Region. The second Region, kept ready to take over, is the recovery Region.
An analogy: a bakery chain
Picture a bakery with two shops across town, each on its own power line. When one shop's power fails, the other keeps bread coming. That's Multi-AZ.
A storm can black out the whole town, though. That's the Region-level event. The owner's options for it, from cheapest to most expensive, are:
- Recipes in a safe in the next town. Nothing is open there, so rebuilding a shop takes hours. That's backup and restore.
- A rented shop in the next town with stocked fridges but cold ovens. Someone has to fire up the ovens first. That's pilot light.
- A small shop in the next town, open with one baker. It sells bread right away and hires more staff as crowds arrive. That's warm standby.
- Two full shops, both selling every day. Customers already go to both. That's multi-site active/active.
Where it breaks: the top three tiers replicate every change continuously and asynchronously, so RPO stays above zero. Only backup and restore copies now and then.
Walk one booking through it
The diagram shows both layers. The walkthrough follows one booking through a normal day, then through two different failures.
Fernbrook Clinic runs a booking app on AWS. Hana is the engineer on call, and Diego is a patient booking a check-up.
- Diego opens the booking site. His browser asks Amazon Route 53 for the site's address. Route 53 is the AWS DNS (Domain Name System) service.
- Route 53 holds a failover record: a primary answer, a secondary answer, and a health check that tests the primary.
- Failover means switching traffic to the standby copy.
- The primary health check passes. Route 53 answers with the primary Region's Application Load Balancer (ALB), which spreads requests across servers.
- The ALB sends the request to an app copy. Copies run in AZ a and AZ b. They sit in an Auto Scaling group: a fleet of Amazon EC2 instances (virtual servers) that replaces failed ones.
- The app saves Diego's booking to an Amazon RDS (Relational Database Service) Multi-AZ DB instance deployment. RDS copies the write synchronously, before confirming it, to a standby in another AZ.
- The booking also travels to the recovery Region. Continuous replication copies it to a cross-Region replica, a read-only copy of the database. The recovery Region's app servers are defined in code but switched off: a pilot light.
- AZ a loses power. The app copies in AZ b keep serving. If the database primary was in AZ a, RDS fails over to the standby behind the same DNS name. Route 53 changes nothing, because the Region is still healthy.
- Later, the whole primary Region fails. The primary health check fails. Hana starts the recovery runbook, a written, tested list of recovery steps.
- The runbook brings the recovery Region up. It promotes the replica to a writable database and starts the app servers. Route 53 then answers with the secondary record, once the recovery load balancer reports healthy.
- Diego can book again. The time until the site works again is the RTO. Bookings not yet replicated are lost, and that gap is the RPO.
The wait while app servers start is why warm standby, with servers already running, gives a shorter RTO. Step 6 needed no second Region. Steps 7 to 9 did, and they cost a second copy of the data running all year.
The rules the exam tests
- RPO (recovery point objective) is how much data you can lose, measured as time since the last good copy.
- RTO (recovery time objective) is how long the service can be down.
- Multi-AZ covers a data center. Most workloads need only this layer.
- A second Region is a deliberate, expensive choice. The stem has to justify it with a Region-level event.
- Four DR strategies, cheapest first: backup and restore, pilot light, warm standby, multi-site active/active. Each step up costs more and lowers RTO and RPO.
- Pick the cheapest strategy that meets both stated targets. Don't build for the worst case you can imagine.
Deep diveControl plane and data plane during a failover
The control plane is the part of a service that creates and changes resources. The data plane is the part that does the running work, such as answering DNS queries.
Use only data plane operations during a failover. The data plane has higher availability goals than the control plane.
Scaling an Auto Scaling group is a control plane action. Changing the weight on a Route 53 weighted record is too. Each one adds a dependency you may not want during a disaster.
A warm standby that scales out through Auto Scaling depends on the control plane. Pre-scaling enough capacity before the disaster avoids that dependency.
Deep diveAutomatic or manual failover
Failover is either automatic or manual. Automatic failover, driven by health checks, is fast.
Automatic failover can also trigger on a false alarm. A failover you didn't need still interrupts the service.
Many teams choose a manual trigger with an automated runbook. One action by a person then starts a tested sequence of steps.
The walkthrough used a pilot light, but the stem's numbers decide which of the four strategies fits. Each one answers two questions: what already runs in the recovery Region, and how much must happen before it serves traffic.
The four strategies side by side
- Backup and restore runs nothing in the recovery Region. You rebuild the infrastructure from code and restore the data.
- Pilot light keeps only the data layer running. The application servers must start before it can take traffic.
- Warm standby keeps a scaled-down but complete copy running. It serves reduced traffic at once, then scales out.
- Multi-site active/active serves traffic from every Region at once. RTO is close to zero, but you must handle writes that land in two places.
An active/active design still needs backups. Replication can carry corruption or deletes to the other Region, so keep versioning or point-in-time backups.
Pilot light and warm standby are both active/passive: one Region serves, and the other waits.
Decision guide
| If the question says… | Reach for… | Because… |
|---|---|---|
| "survive a data center or Availability Zone failure", no Region-level requirement | Multi-AZ deployment (Auto Scaling group across AZs, Multi-AZ database) | Copies in other zones keep serving. A second Region costs more for a risk not named |
| "RPO in hours, RTO up to 24 hours", "lowest cost", "data must survive a Region loss" | Backup and restore, with backups copied to the recovery Region (for example with AWS Backup) | Nothing runs there, so it's cheapest. Rebuilding from code and restoring is slowest |
| "RPO in minutes", "RTO in tens of minutes", "keep the core database always replicated" | Pilot light: replicated data always on, application servers defined in code but switched off | Replication keeps data current, so only compute has to start |
| "RPO in seconds", "RTO in minutes", "the standby must handle traffic right away at reduced capacity" | Warm standby: scaled-down, fully running copy in the recovery Region | The stack is already live, so recovery is a scale-out, not a rebuild |
| "RTO near zero", "every Region serves users", "global writes must stay consistent" | Multi-site active/active with a data layer that handles cross-Region writes | Every Region already serves. The hard part is replication and write conflicts |
| "Region-level database failover with data loss measured in seconds" | Amazon Aurora global database, with a secondary cluster promoted to primary | Replication lag is typically under a second. Plan for recovery in minutes |
| "failover from one Region to another, controlled by DNS, with health checks" | Route 53 failover routing: a primary record and a secondary record, each tied to a health check | Route 53 answers with the secondary when the primary fails, on the DNS data plane |
| "database failover with no application change, the endpoint must stay the same" | RDS Multi-AZ DB instance deployment | RDS points the same DNS name at the standby, typically within 60 to 120 seconds |
| "many short-lived connections spike during failover", "too many connections", Lambda to database | Amazon RDS Proxy in front of the database | It holds client connections during failover, and pooling shares few database connections among many clients |
| "recovery Region drifted from production after hand patches", "rebuild the same servers in another Region" | Immutable infrastructure: launch replacements from a tested, versioned image and never patch in place | Every replacement starts from one image, so the Regions match and drift can't build |
| "data must survive the loss of an Availability Zone", "shared files for a fleet across zones" | S3 (except One Zone-IA) for objects, or a Regional EFS file system for shared files | Both store data across multiple AZs, so there's no standby to build |
| "restore a failed EBS volume in another zone or Region" | EBS snapshots, copied to the recovery Region | A volume lives in one AZ. Its snapshot can be copied to another Region |
| "copy data to another Region continuously", "recovery Region needs current objects" | S3 Cross-Region Replication (versioning on both buckets) | Objects copy as they're written. It isn't a backup: version-specific deletes don't copy |
Terms in the table:
- AWS Backup is the AWS service that schedules and copies backups.
- AWS Lambda runs code without servers. Each Lambda execution environment opens its own database connections, so many concurrent invocations can exhaust the database's connection limit.
- Connection pooling keeps a small set of database connections open and shares them among many clients.
- Amazon S3 (Simple Storage Service) stores objects. S3 One Zone-IA is a storage class kept in a single AZ.
- Amazon EFS (Elastic File System) is a shared file system. Amazon EBS (Elastic Block Store) is a disk that normally attaches to one instance.
Deep diveCaveats behind the decision guide rows
An Aurora global database spans Regions. The Aurora FAQ says a secondary can be promoted in under a minute. The user guide frames recovery time in minutes, so plan for minutes.
An EBS snapshot is stored durably. You can copy it to another Region and create new volumes from it there.
Correct pattern
REFERENCE PATTERN: Route 53 active-passive failover between Regions. Two alias records share one name. An alias record points at an AWS resource, such as a load balancer.
Each record has a SetIdentifier, and the primary record references a health check. Route 53 answers with the secondary record only when the primary's health check fails. On an alias record, Evaluate Target Health or a health check can drive failover.
The names and IDs below are placeholders.
PrimaryRecord:
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: <your-hosted-zone-id>
Name: app.<your-domain>
Type: A
SetIdentifier: primary-region
Failover: PRIMARY
HealthCheckId: !Ref PrimaryHealthCheck
AliasTarget:
DNSName: <primary-alb-dns-name>
HostedZoneId: <primary-alb-hosted-zone-id>
EvaluateTargetHealth: true
SecondaryRecord:
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: <your-hosted-zone-id>
Name: app.<your-domain>
Type: A
SetIdentifier: recovery-region
Failover: SECONDARY
AliasTarget:
DNSName: <recovery-alb-dns-name>
HostedZoneId: <recovery-alb-hosted-zone-id>
EvaluateTargetHealth: true
REFERENCE PATTERN: the standby must be able to take load. Define the recovery Region in the same template as the primary, with a scaled-down desired capacity. Set the Auto Scaling group maximum to the production size so that scaling out is possible.
RecoveryAsg:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
MinSize: "2" # scaled-down, but across two Availability Zones
MaxSize: "12" # production capacity, which must fit within the Region's quotas
DesiredCapacity: "2"
VPCZoneIdentifier: [!Ref RecoverySubnetA, !Ref RecoverySubnetB]
Raise the recovery Region's service quotas (account limits, such as EC2 vCPU limits) to production size before you need them. A standby that can't scale past its quota is a warm standby in name only.
Traps
AWS CloudFormation builds resources from a template you write as code. The fourth trap relies on it.
| The trap | Why it fails | Correct pattern |
|---|---|---|
| Multi-Region active/active for a workload whose requirement is only "survive an AZ failure" | Looks most resilient, but multiplies cost and data-sync work for a risk never raised | Multi-AZ inside one Region; add a recovery Region only when the stem names a Region-level event or a regulation that requires one |
| Counting a Multi-AZ standby as capacity for reads | A Multi-AZ DB instance standby serves no reads. It exists only for failover | Multi-AZ DB instance for failover; read replicas or a Multi-AZ DB cluster (a cluster with two readable standbys) for read scaling (details in 3.3) |
| Assuming a pilot light can take traffic the moment it is needed | No app servers run. Starting them, deploying code and warming caches is your RTO | Pick warm standby if the RTO is minutes; keep pilot light for RTOs in the tens of minutes and rehearse the start-up steps |
| Backups in the recovery Region with no infrastructure code | Data is half of recovery. Rebuilding the environment by hand can outlast the RTO | Define the stack in CloudFormation, back up configuration and machine images, and test restores on a schedule |
| Route 53 weighted records changed by hand to move all traffic to the recovery Region | Changing weights is a control plane action. If that plane is impaired, nothing changes | Failover routing records with health checks for automatic switching, plus a runbook that a person starts for a deliberate switch |
| Sizing the standby Region at production scale with no reason in the stem | Doubles cost for an RTO nobody asked for. Over-building is the most common DR mistake | Choose the lowest strategy whose RPO and RTO fit the stated numbers; check the stem before you add capacity |
| Calling RDS Proxy a replacement for Multi-AZ | The proxy pools connections but stores no data copy, so it has nothing to fail over to | Use Multi-AZ for the copy and RDS Proxy for connection handling during failover |
Compare
| Aspect | Backup and restore | Pilot light | Warm standby | Multi-site active/active |
|---|---|---|---|---|
| Typical RPO | Hours (minutes with point-in-time recovery) | Minutes | Seconds | Near zero |
| Typical RTO | Up to 24 hours | Tens of minutes | Minutes | Potentially zero |
| What runs in the recovery Region | Nothing; backups only | Replicated data stores | Full stack at reduced capacity | Full stack serving traffic |
| Application servers at failover | Built and deployed from code | Started and scaled | Scaled out | Already serving |
| Relative cost | Lowest | Low | Moderate to high | Highest |
| Aspect | Multi-AZ DB instance | Aurora global database | Read replica (cross-Region) |
|---|---|---|---|
| Purpose | Failover within one Region | Failover across Regions | Read scaling, and a manual promotion target |
| Replication | Synchronous to a standby in another AZ | Storage-level, typically under a second lag | Asynchronous |
| Serves reads from standby | No | Secondary clusters serve reads | Yes, as a read replica |
| Typical failover time | 60 to 120 seconds (automatic) | Under a minute to promote a secondary (Aurora FAQ); the user guide frames RTO in minutes | Manual promotion; takes several minutes or longer |
| Application change | None; the DNS endpoint stays the same | None with the global writer endpoint (Aurora FAQ) | Endpoint change for reads |
Worked exam scenario
Exam scenario
A claims company runs its web tier on Amazon EC2 instances in an Auto Scaling group behind an Application Load Balancer and stores claims in an Amazon RDS for PostgreSQL database. The business requires that the service recover from an Availability Zone outage within 5 minutes with no data loss, and that it stay available if the entire primary Region becomes unavailable, with an RTO of 10 minutes and an RPO of about 1 minute for the Region-level event. The standby Region must take traffic as soon as DNS fails over, even at reduced capacity. The team wants the lowest cost that meets these targets. Which solution meets these requirements MOST cost-effectively?
- A.Run the web tier and database at full capacity in two Regions with active/active writes and a global load balancer that splits traffic evenly.
- B.Deploy the web tier and the database across two Availability Zones in one Region, and back up the database to Amazon S3 once a day.
- C.Deploy the database as Multi-AZ in the primary Region, replicate it continuously to a cross-Region read replica that is promoted on failover, keep the web tier as a scaled-down Auto Scaling group in the recovery Region, and use a health-checked Route 53 failover record.
- D.Use a Multi-AZ database and Auto Scaling group in one Region, and restore a Region-level backup to a new Region after a disaster.
Show the walkthrough
- Split the requirements. The stem names two failure levels. An AZ outage needs recovery within 5 minutes with no data loss. A whole-Region event needs an RTO of 10 minutes and an RPO of about 1 minute.
- Note what that rules out. Each level needs a different mechanism, so a one-Region design can't cover both.
- Eliminate A. Full active/active in two Regions meets the targets, but at the highest cost. Its near-zero RTO is more than the business asked for.
- Eliminate B. Multi-AZ covers the AZ outage. A daily backup, though, can lose up to a day of data, and it doesn't cover a Region event.
- Eliminate D. Restoring a backup in a new Region is backup and restore. Its RPO and RTO are measured in hours, so it misses both Region targets.
- Choose C. Multi-AZ database failover covers the AZ outage within minutes. A continuously replicated cross-Region copy keeps data loss within about a minute while replication keeps up.
- Confirm the cost. The scaled-down web tier behind a health-checked Route 53 failover record is a warm standby. It is already serving, so it fits a 10-minute RTO at moderate cost.
- Check for anything cheaper. A pilot light would cost less, but its servers start from nothing, so recovery takes tens of minutes. That misses a 10-minute RTO, so C is the cheapest design that meets both targets.
Answer: C
Check your understanding
Build it
The lab for this task, Force a Multi-AZ database failover, builds a Multi-AZ DB instance and forces a failover. You confirm that the standby takes over behind the same endpoint. You also check this lesson's trap: the standby can't serve reads.
Key takeaways
- Start from the stated RPO and RTO. Pick the cheapest DR strategy that meets both, and don't over-build.
- Multi-AZ handles a data center failure in one Region. Add a recovery Region only when the requirement names a Region-level event.
- Backup and restore is cheapest and slowest. Pilot light keeps data running, warm standby keeps the whole stack running, and active/active serves from every Region.
- Automate failover with Route 53 failover records and health checks, and use only data plane operations during recovery.
- Make the recovery Region able to take production load: define it as code, and raise its service quotas before you need them.
- Match the recovery strategy to the stated RPO and RTO, not the worst case: Multi-AZ covers a data center, and only a Region-level loss needs a second Region.
Hands-on lab: Force a Multi-AZ Database Failover and Watch the Standby Take Over
Sources
- https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html
- https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_planning_for_recovery_disaster_recovery.html
- https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_withstand_component_failures_avoid_control_plane.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-what-is-isnot-replicated.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.Failover.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-planning.html
- https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-disaster-recovery.html
- https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-connecting.html
- https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Replication.CrossRegion.Promote.html
- https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-proxy.howitworks.html
- https://docs.aws.amazon.com/rds/latest/auroraextendedcontent/aurora-faq-availability-and-durability.html
- https://docs.aws.amazon.com/rds/latest/auroraextendedcontent/aurora-features-availability.html
- https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-failover.html
- https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_tracking_change_management_immutable_infrastructure.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-how-setup.html
- https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html
- https://docs.aws.amazon.com/ebs/latest/userguide/ebs-copy-snapshot.html
- https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html
- https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover-types.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html
- https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions-availability-zones.html
Last reviewed