Task 2.1 · 19 min read
Scaling, Caching and the Edge: Load Balancers, Stateless Tiers and CloudFront
Services in this lesson
- Elastic Load Balancing (ELB)in scope
- Amazon CloudFrontin scope
- Amazon ElastiCachein scope
- Amazon EC2 Auto Scalingin scope
- Amazon RDSin scope
- Amazon API Gatewayin scope
What the exam tests
- Choosing between horizontal scaling (more copies) and vertical scaling (a bigger copy), and making a tier stateless so it can scale out.
- Picking the load balancer from protocol and feature words. Application Load Balancer means HTTP routing. Network Load Balancer means TCP or UDP, extreme throughput and static IP addresses.
- Choosing a caching layer and strategy: Amazon CloudFront at the edge, API Gateway response caching, Amazon ElastiCache with lazy loading or write-through, and expiry times.
- Using CloudFront to speed up static and dynamic content: origin access control for S3, expiry times, invalidation versus versioned file names, and origin failover.
- Knowing when read replicas solve the problem (read-heavy databases) and when they don't (write bottlenecks, failover).
The core concept
A web application grows by adding identical copies of each layer, and it stays fast by answering repeated requests from a cache. The exam rewards the answer that scales out the right layer and caches as close to the user as the data allows.
An analogy: a sandwich shop at lunchtime
Picture a sandwich shop when the lunch rush hits:
- The shop opens more identical counters instead of hiring one super-fast cook. That's horizontal scaling. A faster cook is vertical scaling, and there's a limit to how fast one person gets.
- A host at the door sends each customer to a free counter and skips any counter whose cook is out sick. That's a load balancer.
- Orders are pinned to a shared board, not remembered by one cashier. Any counter can finish any order, and closing a counter loses nothing. That's a stateless tier.
- Popular sandwiches sit pre-made in a case by the door, so most customers never reach a counter. That's a cache.
Where the analogy breaks: a pre-made sandwich is gone once sold, but one cached response can be served to any number of users until it expires.
Walk two readers through it
The walkthrough follows two readers of Harbor Daily, a news site whose app runs in one AWS Region.
Wen reads Harbor Daily from Sydney. Ines, an editor, works for the site and is signed in to leave comments.
- Wen's request reaches a CloudFront edge location. Amazon CloudFront is a content delivery network (CDN): a set of edge locations around the world that keep cached copies of responses. Sydney has one close to Wen.
- The photo on the page is already cached. It stays cached until its TTL (time to live, how long a cached copy is kept) runs out. If it weren't cached, CloudFront would fetch it from the origin, the place the real file lives. Here that's an Amazon S3 bucket, which accepts only CloudFront through origin access control (OAC).
- Wen's
/api/headlinescall misses the cache. CloudFront forwards it to the Application Load Balancer (ALB), the single entry point for application traffic. The ALB runs a health check (a regular test request) on every app copy and sends requests only to healthy ones. - A listener rule routes the call. A listener is the port and protocol the ALB accepts traffic on. A listener rule is its instruction, such as "requests for
/api/*go to the API copies." The ALB picks one of several identical Amazon EC2 instances (virtual servers) in that group. - The app copy needs the top stories. It asks Amazon ElastiCache, an in-memory cache, first. On a miss it queries a read replica, a read-only copy of the database, and stores the answer in ElastiCache. Filling the cache only on a miss is called lazy loading.
- Ines posts a comment. The write goes to the primary database, the only copy that accepts writes. The primary copies the change to the read replica with asynchronous replication: in the background, after the write is confirmed. So the replica lags slightly behind.
- Ines's next click lands on a different app copy. She stays signed in, because her session lives in ElastiCache, not in the memory of the copy that served her before.
- Traffic drops in the evening. Amazon EC2 Auto Scaling removes two app copies. Nobody is signed out, because no copy held anything a later request needed.
Keep Ines's session in one copy's memory instead, and step 7 signs her out, because her next click lands on another copy. Step 8 would then also sign out anyone whose copy was removed.
The rules the exam tests
- Scale out first. Horizontal scaling adds copies and spreads load across them. It has a much higher ceiling (account quotas still apply) and removes single points of failure.
- Scaling out needs stateless copies. Stateless means nothing a later request needs is stored on the copy itself. The exam prefers horizontal scaling whenever a component can run as identical copies.
- Scaling up has a ceiling. Vertical scaling moves to a bigger instance. It stops at the largest size, usually needs a restart, and still leaves one copy.
- A load balancer is the front door. It gives clients one endpoint, health-checks every copy and sends traffic only to healthy ones. EC2 Auto Scaling adds and removes copies behind it (policies are in 3.2).
- Pick the load balancer by layer. Layer 7 means the balancer reads the HTTP request, so it can route by path or host. Layer 4 means it sees only connections and ports.
- Layer 4 carries more than web traffic. TCP and UDP are the two basic ways to send data over a network. TLS is the encryption behind HTTPS.
- Every cache hit is a request the next tier never sees. Put the cache as close to the user as the data allows.
- Match the cache to the data. Shared content goes to CloudFront. Repeated API responses go to the API Gateway stage cache, a cache Amazon API Gateway keeps for one deployed stage of a REST API. Hot database reads and sessions go to ElastiCache.
- Read replicas scale reads only. A replica is a full read-only copy, not a cache, and it can lag. It takes read load off the primary like a cache does. Writes and failover need other answers.
Deep diveScaling up, and why sticky sessions aren't the fix
Vertical scaling is still the right answer when software can't run as more than one copy, such as a legacy app. You accept the size ceiling and the single copy.
Sticky sessions tell the load balancer to keep sending a user to the same copy. They only pin the user to that copy. When that copy goes away, the session goes with it, so treat sticky sessions as a stopgap, not the design.
The real fix is to move sessions out to ElastiCache or Amazon DynamoDB (a key-value database). Then Auto Scaling can remove any instance safely.
Deep diveLoad balancer details the exam checks
- An ALB works at layer 7, the application layer. It reads HTTP, HTTPS and gRPC (a remote procedure call protocol), so it can route on path, host, headers, query strings and HTTP method.
- ALB targets can be instances, IP addresses or AWS Lambda functions.
- An NLB (Network Load Balancer) works at layer 4, the transport layer. It handles TCP, UDP and TLS at very high throughput and routes on protocol and port only.
- NLB targets can be instances, IP addresses or an ALB.
- Fixed IP addresses: an NLB has one static IP address in each Availability Zone (AZ) it uses. An AZ is an isolated group of data centers in a Region.
- Elastic IPs on an NLB: an internet-facing NLB can take an Elastic IP address in each subnet mapping. An Elastic IP is a static public IPv4 address allocated to your account.
- An ALB has no fixed IP address. Clients reach it by its DNS name.
- Cross-zone load balancing spreads traffic evenly across targets in every AZ. For an ALB it is always on at the load balancer level, and you can turn it off per target group. It is off by default for an NLB.
Deep diveCloudFront freshness and failover
- TTL (time to live) is how long CloudFront keeps a cached copy before checking the origin again. The default TTL is 24 hours unless your cache policy sets another value.
- The origin can set freshness with a
Cache-Controlheader. A minimum TTL above 0 beats the origin. CloudFront still caches for that long, even if the origin sendsno-cache. - A cache key is what CloudFront uses to tell cached objects apart. A versioned file name such as
app.v42.jsis a new cache key, so there's nothing stale to remove. - An invalidation removes cached copies before their TTL ends.
- An origin group pairs a primary and a secondary origin. CloudFront fails over to the secondary when the primary returns a status code you chose, such as 500, 502, 503 or 504.
- Connection failures and timeouts count only through those codes. A failed connection triggers failover only if 503 is selected. A timeout triggers it only if 504 is selected.
- Failover applies only to GET, HEAD and OPTIONS requests.
Decision guide
| If the question says… | Reach for… | Because… |
|---|---|---|
| "route /api and /images to different services", "host-based routing", HTTP/HTTPS, containers or Lambda targets | Application Load Balancer | Layer 7 rules route on path, host and headers. Targets can be instances, IPs or Lambda |
| "TCP or UDP", "millions of requests per second", "ultra-low latency", "preserve the client IP" | Network Load Balancer | Layer 4 balancing built for very high throughput on TCP, UDP and TLS |
| "partners must allowlist fixed IP addresses" | Network Load Balancer with an Elastic IP per Availability Zone | An internet-facing NLB takes an Elastic IP per subnet. An ALB is reached by DNS name |
| "users are logged out when instances scale in" | Move sessions to ElastiCache or DynamoDB | Stateless copies can be removed safely. Sticky sessions still lose the session with the target |
| "legacy app can't run as multiple copies", "needs more memory now" | Vertical scaling: a larger instance type | The only lever when software can't be distributed. Accept the ceiling and the single copy |
| "global users", "static assets load slowly", "reduce load on the origin" | CloudFront in front of S3 (with origin access control) or the ALB | Edge locations serve cached copies until the TTL expires. Only misses reach the origin |
| "origin must keep serving when the primary origin returns errors", "failover without changing the URL" | CloudFront origin group: a primary and a secondary origin, with failover status codes | Fails over on status codes you choose. Applies to GET, HEAD and OPTIONS requests only |
| "same database queries repeated thousands of times", "read-heavy", "microsecond latency" | ElastiCache with lazy loading (cache-aside) and a TTL | Caches only requested data, survives a cache node failure, and the TTL limits staleness |
| "cached data must never be stale after an update" | Write-through caching, plus a TTL | Every write updates the cache, so reads are fresh. The TTL clears unread entries |
| "identical API responses for many callers", REST API | API Gateway stage cache | Caches GET responses for 300 seconds by default, up to 3,600, cutting backend calls |
| "reporting queries slow down the transactional database" | A read replica that serves the reporting queries | Asynchronous copies take read load off the primary. More in 3.3 |
| "updated CSS and JavaScript must reach users immediately after each release" | Versioned file names (or an invalidation) | A new name is a new cache key. An invalidation removes copies before they expire |
| "non-HTTP traffic", "static anycast IPs", "fast regional failover" for global users | AWS Global Accelerator (3.4) | Fixed static IPs, routed over the AWS global network to healthy endpoints. No caching |
Correct pattern
REFERENCE PATTERN: one ALB, two services, stateless copies. Path rules send /api/* to an API target group (the set of copies serving one service) and everything else to the web target group.
Each group health-checks its own path. An Auto Scaling group or Amazon ECS (Elastic Container Service) service registers copies into it.
ApiRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
ListenerArn: !Ref HttpsListener
Priority: 10
Conditions:
- Field: path-pattern
PathPatternConfig:
Values: ["/api/*"]
Actions:
- Type: forward
TargetGroupArn: !Ref ApiTargetGroup
ApiTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
VpcId: !Ref Vpc
Protocol: HTTP
Port: 8080
TargetType: ip # ECS tasks on Fargate register by IP
HealthCheckPath: /api/health
# The listener's default action forwards everything else to WebTargetGroup.
REFERENCE PATTERN: lazy loading with a TTL. Read from the cache first. On a miss, read the database and write the result back to the cache with an expiry.
get_product(id):
item = cache.get("product:" + id)
if item exists:
return item # cache hit: database untouched
item = db.query("SELECT ... WHERE id = ?", id)
cache.set("product:" + id, item, ttl = 300 seconds)
return item
Lazy loading caches only what someone asked for, and it survives an empty or failed cache node. The cost is a slower first read and possibly stale data until the TTL ends.
Write-through updates the cache whenever the database changes, so data is fresh. The cost is that every write happens twice, and the cache fills with items that may never be read. Combining write-through with a TTL gets most of both.
Traps
| The trap | Why it fails | Correct pattern |
|---|---|---|
| Adding instances behind a load balancer while sessions live in each instance's memory | Users bounce between copies and lose sessions, and scale-in deletes them | Move sessions to ElastiCache or DynamoDB. Sticky sessions are a stopgap, not the design |
Network Load Balancer to route /orders and /users to different services | NLB feels like the faster upgrade. It works at layer 4 and can't see HTTP paths | Application Load Balancer with path-pattern listener rules |
| Application Load Balancer when partners need fixed IPs in their firewall | ALB is the default web answer, but it is reached only by DNS name | Network Load Balancer with Elastic IPs (or Global Accelerator static anycast IPs, taught in 3.4) |
| "Make it highly available" answered with a bigger instance | It adds capacity, but it's still one instance in one AZ: a single point of failure | Copies across AZs behind a load balancer (2.2) |
| Read replica to fix a write bottleneck or to provide automatic failover | Sounds like more capacity. Replicas serve only reads, writes still hit the primary, and lag is normal | Scale the primary or redesign writes (3.3). Use Multi-AZ for failover (2.2) |
| Caching per-user, rapidly changing responses at CloudFront to cut origin load | Any cache sounds like a win. Responses unique per user or per second rarely get a cache hit | Cache what repeats across users. Send personalized paths straight to the origin with caching disabled |
Minimum TTL above 0 on a behavior whose origin sends no-cache | Seems a safe floor. CloudFront then caches for the minimum TTL even though the origin said not to | Leave minimum TTL at 0 where the origin controls freshness with Cache-Control |
Compare
| Aspect | Application Load Balancer | Network Load Balancer |
|---|---|---|
| OSI layer | 7 (HTTP, HTTPS, gRPC) | 4 (TCP, UDP, TLS) |
| Routing decisions | Path, host, headers, query strings, methods | Protocol and port only |
| Targets | Instances, IP addresses, Lambda functions | Instances, IP addresses, an ALB |
| Fixed IP addresses | No; reached by DNS name | One static IP per Availability Zone; Elastic IPs if internet-facing |
| Cross-zone load balancing | Always on; can be turned off per target group | Off by default |
| Stem keywords | "path-based", "microservices", "HTTP headers", "Lambda targets" | "TCP/UDP", "static IP", "millions of requests", "low latency" |
| Aspect | CloudFront | API Gateway cache | ElastiCache | Read replica |
|---|---|---|---|---|
| Where it sits | Edge locations near users | In front of the API's backend integration | Beside the application, inside the VPC | Beside the primary database |
| What it caches | Whole HTTP responses: files, pages, API GETs | REST API method responses (GET by default) | Anything the code puts there: query results, sessions | A full, read-only copy of the database |
| Who decides freshness | TTLs and origin Cache-Control (default TTL 24 hours unless the cache policy sets another) | Stage TTL (default 300 s, max 3,600 s) | Your code: TTL per key, lazy loading or write-through | Asynchronous replication lag |
| Code changes needed | None for cacheable content | None | Yes: the application reads and writes the cache | Yes: send reads to the replica endpoint |
Worked exam scenario
Exam scenario
A news website runs on Amazon EC2 instances in an Auto Scaling group behind an Application Load Balancer. Article pages and images are the same for every reader. Readers are spread across several continents. When a major story breaks, traffic rises sharply, the instances hit 100% CPU before new ones finish launching, and readers far from the AWS Region report slow page loads. Which solution will MOST improve performance for readers while reducing load on the instances?
- A.Change the instance type in the launch template to a larger size.
- B.Create an Amazon RDS read replica and point the article queries at it.
- C.Put an Amazon CloudFront distribution in front of the Application Load Balancer with a cache policy that caches article pages and images.
- D.Put AWS Global Accelerator in front of the Application Load Balancer.
Show the walkthrough
- Find what repeats. Pages and images are identical for every reader, so they can be cached. Readers are global, so the cache belongs at the edge.
- Eliminate A. Larger instances raise each copy's ceiling, but every request still reaches the origin. Distant readers still wait on a round trip across the world.
- Eliminate B. Nothing in the stem points to the database. The instances' CPU is the bottleneck, and a replica doesn't help distant readers.
- Eliminate D. Global Accelerator speeds the network path to the ALB, but it doesn't cache. Every request still lands on the instances.
- Choose C. CloudFront answers repeated requests from edge locations near readers. That cuts latency and absorbs the spike before it reaches the Auto Scaling group.
Answer: C
Check your understanding
Build it
The lab for task 2.1 is the Loose coupling lab. It builds the queue-and-topic layer from 2.1a.
Its queues let a worker tier like the one in this lesson scale out on backlog instead of on request rate. Treat it as the back half of the stateless-copies pattern.
Key takeaways
- Scale out before you scale up. Identical stateless copies behind a load balancer remove the ceiling and the single point of failure. Scale up only software that can't be distributed.
- Use an ALB for HTTP routing by path, host or header. Use an NLB for TCP or UDP, extreme throughput and fixed IP addresses.
- Cache at the layer closest to the user that the data allows: CloudFront for shared content, the API Gateway cache for repeated API responses, ElastiCache for hot data and sessions.
- Lazy loading tolerates cache failures and some staleness. Write-through keeps data fresh at the cost of extra writes. A TTL balances both.
- Read replicas scale reads, not writes or failover.
- Scale out stateless copies behind the right load balancer, then cache whatever repeats: CloudFront for shared content, an in-memory cache for hot data, read replicas for read-heavy databases.
Hands-on lab: Fan Out Orders with SNS, SQS and a Dead-Letter Queue
Sources
- https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html
- https://docs.aws.amazon.com/elasticloadbalancing/latest/application/rule-condition-types.html
- https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html
- https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-target-group-attributes.html
- https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html
- https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html
- https://docs.aws.amazon.com/elasticloadbalancing/latest/network/edit-target-group-attributes.html
- https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-elasticloadbalancingv2-loadbalancer-subnetmapping.html
- https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html
- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Strategies.html
- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html
- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DownloadDistValuesCacheBehavior.html
- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html
- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html
- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html
- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions-choosing.html
- https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-caching.html
- https://docs.aws.amazon.com/global-accelerator/latest/dg/introduction-how-it-works.html
- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/understanding-the-cache-key.html
Last reviewed