Skip to content
Certification Guide

Task 2.1 · 20 min read

Messaging and Events: Decoupling with Queues, Topics, Buses and Workflows

Services in this lesson

  • Amazon SQSin scope
  • Amazon SNSin scope
  • Amazon EventBridgein scope
  • AWS Step Functionsin scope
  • Amazon API Gatewayin scope
  • Amazon MQin scope
  • AWS Transfer Familyin scope

What the exam tests

  • Whether you can spot a coupling problem and remove it. Typical stems describe a synchronous call chain, a backend that falls over during spikes, or one event that several teams need. The fix is a queue, a topic, an event bus or a workflow.
  • Whether you get delivery semantics right. That covers Amazon SQS standard versus FIFO (first-in, first-out) queues. It also covers at-least-once delivery, idempotent consumers, the visibility timeout and dead-letter queues.
  • Whether you pick the right front door for an API with Amazon API Gateway: REST API versus HTTP API, throttling (capping how many requests a client can send) and caching. You also decide when the API should hand work to a queue instead of waiting for it.
  • Whether you use AWS Step Functions to orchestrate multi-step processes, and pick Standard or Express workflows.
  • Whether you prefer purpose-built managed services over self-run servers. Amazon MQ fits apps that already talk to a message broker (a server that relays messages between apps) over a protocol such as AMQP (Advanced Message Queuing Protocol). AWS Transfer Family fits partners who send files over SFTP (SSH File Transfer Protocol).

The core concept

Two parts of a system are tightly coupled when one must be up, fast and unchanged for the other to work. Loose coupling puts a durable AWS service between them, so each side can fail, scale and change on its own.

An analogy: a busy restaurant

Think of a busy restaurant at dinner time:

  • The ticket rail is a queue. A waiter clips an order ticket to the rail. Whichever cook is free takes it, cooks it and throws the ticket away, so each dish is cooked once.
  • The kitchen speaker is a topic. The manager announces "table 9 has a nut allergy", and every station hears its own copy.
  • The host at the door is an event bus. The host listens to what each guest asks for and sends them to the bar, the patio or the dining room.
  • The recipe card is a workflow. A wedding cake card lists the steps in order, what to do if the oven fails, and when to wait for cooling.

Where the analogy breaks: a cook who drops a ticket loses it, but SQS puts an undeleted message back on the rail. One EventBridge rule can send an event to up to 5 targets. And Step Functions runs the steps rather than just listing them.

Software without the rail has the kitchen's problem in reverse. A web tier that calls an inventory service and waits slows down when inventory slows. If inventory is down, orders fail.

AWS offers four shapes for the durable middle piece: a queue, a topic, an event bus and a workflow. The exam rewards you for naming the shape before you name the service.

Follow one order through it

The diagram shows all four shapes. The walkthrough below follows one order through them.

Enlarge
Queue, topic, event bus and workflow: four shapes of loose coupling.

Kenji runs Paperleaf, an online bookshop, in account 444455556666. A customer, Dana, buys three books during a holiday sale.

  1. The API takes the order. Dana's browser calls Amazon API Gateway, the managed front door for Paperleaf's API (application programming interface). API Gateway puts the order on an SQS queue instead of waiting for the backend.
  2. The queue absorbs the spike. The order is now a message. API Gateway is its producer (the sender), and Kenji's worker fleet is the consumer (the receiver). The workers poll the queue (ask it for messages) at their own pace, and scale on queue depth (the number of messages waiting).
  3. One worker takes it. Worker A receives Dana's order, and the message becomes invisible to other workers for the visibility timeout. When Worker A finishes, it deletes the message.
  4. A crash puts it back. If Worker A crashes before deleting, the message reappears after the timeout, and Worker B processes it again. Consumers must therefore be idempotent: handling the same message twice has the same effect as handling it once.
  5. A poison message moves aside. Suppose another order has a broken address and fails every time: a poison message. After a set number of receives (maxReceiveCount), SQS moves it to a dead-letter queue (DLQ), a side queue where Kenji can inspect it. After the fix, he can redrive it (move it back to the source queue).
  6. The topic fans out. Worker A publishes Dana's order to an Amazon SNS topic. SNS pushes a copy to the billing queue and another to the shipping queue. Each team drains its own queue: this is the fan-out pattern.
  7. The event bus routes facts. Paperleaf's payment provider is a SaaS (software as a service) partner. It sends a checkout.approved event (a record that something happened) to an Amazon EventBridge bus, as AWS services also do. A rule matches on the event's content and starts a Step Functions workflow.
  8. The workflow runs the steps. The workflow is a state machine (a defined set of steps and the moves between them). It reserves stock, charges the card, then notifies the customer. It passes data from each step to the next.

Step Functions has two workflow types. Standard runs up to a year and records every step, exactly once. Express runs up to 5 minutes at high volume, at least once when started asynchronously.

API Gateway also has two API types. A REST API is full-featured and pricier, and an HTTP API is slimmer and cheaper. Both can use a JWT (JSON Web Token) authorizer, which validates a signed token from your identity provider.

The rules the exam tests

  • Queue (Amazon SQS): work done once. Consumers compete, so each message goes to one consumer. The queue is the buffer that absorbs a spike.
  • Topic (Amazon SNS): tell everyone who cares. You publish once, and SNS pushes a copy to every subscriber. An SQS queue behind each subscriber gives it its own buffer.
  • Event bus (Amazon EventBridge): route by content. Rules match on what the event says and send it to targets. The bus also receives events from AWS services and SaaS partners.
  • Workflow (AWS Step Functions): a business process with steps. It knows the order, passes state, retries, branches and waits.
  • Standard queues deliver at least once, in best-effort order. So your consumers must be idempotent.
  • FIFO queues keep order within a message group and remove duplicates. A message group is a tag you set, such as a customer ID, that marks which messages must stay in order. FIFO pays for this with a lower throughput ceiling.

The walkthrough skipped the numbers. These ones anchor most SQS questions.

The SQS numbers to know

  • Message size: up to 1 MiB (mebibyte).
  • Retention: the queue keeps a message for 4 days by default, and you can set 1 minute to 14 days.
  • Visibility timeout: 30 seconds by default, 12 hours at most. A message not deleted in time reappears, and someone processes it again.
Deep diveFIFO throughput and ordering details

Without high-throughput mode, a FIFO queue handles 300 transactions per second (TPS) per partition per API action. A partition is a slice of the queue's storage. The actions are SendMessage, ReceiveMessage and DeleteMessage.

Batches of 10 messages raise that to 3,000 messages per second per action. High-throughput mode raises the ceiling further.

FIFO drops a duplicate sent within the 5-minute deduplication interval.

Different message groups process in parallel. A stuck message group holds back later messages in that group. Other groups keep flowing unless one group's backlog grows very large.

To find groups with messages available, SQS scans only the first 120,000 messages in the queue.

Decision guide

Decision guide
If the question says…Reach for…Because…
"backend can't keep up during spikes", "requests are lost when the worker fails"SQS standard queue between the tiers, workers scaled on queue depthMessages wait up to 14 days. A crashed worker's message reappears after the visibility timeout
"process in the exact order received", "no duplicates", per customer or per accountSQS FIFO queue with a message group ID per customerOrder holds within each group, duplicates within 5 minutes are dropped, and groups run in parallel
"the same event must reach several independent systems"SNS topic fanned out to one SQS queue per consumerPublish once. Each subscriber gets a copy, with its own buffer and retry behavior
"only some subscribers want some messages"SNS subscription filter policySNS drops non-matching messages per subscription, so consumers never receive and discard them
"react to AWS service events", "events from a SaaS partner", "route by event content"EventBridge rule on an event busRules match event patterns, with up to 5 targets per rule. Partner sources feed a bus directly
"replay past events after fixing a bug"EventBridge archive and replayAn archive stores matching events so you can send them back to the bus later
"run this job every night at 02:00" or "once, at a set time"EventBridge SchedulerManaged cron, rate and one-time schedules that invoke targets directly. No cron server
"multi-step process", "retry a step", "wait for approval", runs for hours or daysStep Functions Standard workflowRuns up to one year, exactly once, with full history. Suits non-idempotent steps like payments
"high-volume, short event processing", steps are idempotentStep Functions Express workflowRuns up to five minutes. Asynchronous runs are at-least-once, built for high event rates
"API keys", "per-client throttling", "AWS WAF", "private API", "response caching"API Gateway REST APIThese are REST API features only. HTTP APIs trade them away for a lower price
"simple low-cost API in front of Lambda"API Gateway HTTP APIFewer features at a lower price, enough for a simple Lambda backend
"existing application uses AMQP, MQTT or OpenWire", "migrate without rewriting messaging code"Amazon MQManaged ActiveMQ or RabbitMQ brokers speak the standard protocols the app already uses
"partners upload files over SFTP" and processing should start on arrivalAWS Transfer Family into Amazon S3, S3 events to EventBridgeManaged SFTP endpoint with no servers. The object-created event starts the processing
"a message keeps failing and blocks the queue"Dead-letter queue with a maxReceiveCount redrive policyAfter N receives without a delete, SQS moves it aside for inspection and later redrive

A few terms in the table: AWS WAF is a web application firewall. AWS Lambda runs your code without servers. MQTT and OpenWire are messaging protocols.

Correct pattern

REFERENCE PATTERN: SNS fan-out to SQS with a dead-letter queue and a filter. One OrderPlaced message reaches billing and shipping. Shipping only gets physical orders, and billing failures land in a DLQ after five attempts.

The queue policy is the part people forget. Without it, SNS has no permission to send, and nothing arrives.

Resources:
  OrdersTopic:
    Type: AWS::SNS::Topic

  BillingDLQ:
    Type: AWS::SQS::Queue
    Properties:
      MessageRetentionPeriod: 1209600 # 14 days, longer than the source queue

  BillingQueue:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 180 # at least 6x a 30-second Lambda consumer timeout
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt BillingDLQ.Arn
        maxReceiveCount: 5

  ShippingQueue:
    Type: AWS::SQS::Queue

  AllowTopicToSend:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues: [!Ref BillingQueue, !Ref ShippingQueue]
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: { Service: sns.amazonaws.com }
            Action: sqs:SendMessage
            Resource: [!GetAtt BillingQueue.Arn, !GetAtt ShippingQueue.Arn]
            Condition:
              ArnEquals: { aws:SourceArn: !Ref OrdersTopic }

  BillingSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref OrdersTopic
      Protocol: sqs
      Endpoint: !GetAtt BillingQueue.Arn

  ShippingSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref OrdersTopic
      Protocol: sqs
      Endpoint: !GetAtt ShippingQueue.Arn
      FilterPolicy:
        fulfillment: ["physical"]

Why each piece is there:

  • One queue per consumer. Billing can be down for an hour without shipping noticing. Each one scales on its own queue depth.
  • Visibility timeout sized to the consumer. For a Lambda consumer, AWS recommends at least six times the function timeout. Then retries after throttling don't make messages reappear early.
  • maxReceiveCount of 5. That allows enough retries for short-lived errors before a poison message moves aside.
  • DLQ retention longer than the source queue's. In a standard queue, a message keeps its original enqueue timestamp (the time it was first added to the source queue) when it moves to the DLQ.
  • aws:SourceArn condition. Only this topic can write to the queues. This is confused-deputy protection: it stops another topic from using SNS to write to your queue. IAM itself is taught in 1.1.

Traps

TRAPExam traps
The trapWhy it failsCorrect pattern
One SQS queue shared by billing and shipping so "both can read the order"It looks like a shared inbox. But consumers compete, so each order reaches only one systemSNS topic (or EventBridge rule) with one SQS queue per consuming system
SNS alone in front of a consumer that is sometimes offline or slowSNS feels like "the messaging service". It pushes and keeps no backlog for a consumer to pullSNS to SQS, so the queue holds messages until the consumer is ready
Visibility timeout shorter than processing timeThe 30-second default looks harmless. Messages reappear mid-processing and a second worker repeats themSet the timeout above the worst-case processing time (6x the function timeout for Lambda) and make consumers idempotent
FIFO "to be safe" when order doesn't matterFeels stricter. It caps throughput, and a stuck message group holds back later messages in itStandard queue plus idempotent consumers; FIFO only when the stem demands order or no duplicates
Lambda functions calling each other synchronously to run a five-step processWorks in a demo. You pay for waiting, each function needs retry code, and 15 minutes is the limitStep Functions Standard workflow with Retry and Catch on each state
Express workflow for a payment or other non-idempotent stepCheaper at volume. But asynchronous Express runs at-least-once, so a step can run twiceStandard workflow (exactly-once) for non-idempotent actions
Subscribing a queue to a topic with the CLI or a template and no queue policyThe subscription succeeds, but SNS can't call sqs:SendMessage. Messages never arriveQueue policy allowing sns.amazonaws.com to send, scoped with aws:SourceArn
Deep diveExact numbers behind the traps

The FIFO trap's throughput cap is spelled out in the "FIFO throughput and ordering details" deep dive under The core concept.

Synchronous Express workflows run at-most-once. Asynchronous Express workflows run at-least-once.

The SQS console adds the queue policy for you when you subscribe a queue to a topic. The CLI and templates don't, which is why the missing-policy trap hides in practice.

Compare

Compare
AspectAmazon SQSAmazon SNSAmazon EventBridgeAWS Step Functions
ShapeQueue: point-to-point work bufferTopic: publish/subscribe fan-outEvent bus: content-based routingWorkflow: ordered steps with state
Delivery modelConsumers poll (pull)SNS pushes to subscribersRules push matching events to targetsThe state machine calls each service in turn
Who gets a messageOne consumer from a competing poolEvery subscriber (after its filter)Every rule whose pattern matches (up to 5 targets per rule)The next state in the definition
Holds messages?Yes, 1 minute to 14 days (default 4 days)No backlog to pull. Retries, then DLQ if setOptional archive you can replayStandard keeps execution history for 90 days
OrderingFIFO queues per message groupFIFO topics (to SQS queues)Not guaranteed for unordered subscribers. FIFO subscribers keep order per event groupDefined by the state machine
Typical stem keywords"buffer", "decouple", "spikes", "retry later""notify", "fan out", "multiple subscribers""AWS service events", "SaaS", "route by content", "schedule""orchestrate", "multi-step", "approval", "retry step"

Streaming services keep an ordered, replayable log for many readers. Amazon Kinesis and Amazon MSK are a different tool, taught in 3.5 Data ingestion.

Worked exam scenario

Exam scenario

A concert ticketing company runs a web tier on Amazon EC2 that calls an inventory service synchronously for every purchase. During on-sale events the inventory service is overwhelmed, requests time out, and purchases are lost. The inventory service already ignores repeated purchase IDs. The company wants purchases to survive traffic spikes and inventory outages with the LEAST operational overhead. Which solution meets these requirements?

  1. A.Move the inventory service to a larger EC2 instance type with more CPU and memory.
  2. B.Have the web tier send each purchase to an Amazon SQS standard queue, and have inventory workers poll the queue and scale on the number of visible messages.
  3. C.Have the web tier publish each purchase to an Amazon SNS topic with the inventory service subscribed through an HTTPS endpoint.
  4. D.Run a RabbitMQ cluster on EC2 instances across two Availability Zones between the web tier and the inventory service.
Show the walkthrough
  1. Read the coupling problem. "Calls synchronously", "overwhelmed during spikes" and "purchases are lost" add up to one thing. The web tier depends on the inventory service being up and fast, so you need a durable buffer between them.
  2. Note the semantics clue. The inventory service already ignores repeated purchase IDs. At-least-once delivery is fine, and nothing forces FIFO.
  3. Eliminate A. A bigger instance raises the ceiling but keeps the synchronous dependency. An outage still loses purchases, and the next spike can still exceed it.
  4. Eliminate C. SNS pushes to the HTTPS endpoint. If the inventory service is down or slow, SNS retries for a while, but there is no queue for workers to drain at their own pace.
  5. Eliminate D. A broker would buffer the purchases. But running and patching RabbitMQ on EC2 is the opposite of least operational overhead.
  6. Choose B. The SQS queue holds purchases through outages, for up to 14 days. Workers pull at their own pace, and scaling on queue depth matches capacity to the backlog. That's a fully managed buffer.

Answer: B

Check your understanding

Question 1: An order service must notify a fraud team, a loyalty team and a warehouse team about every order. Each team deploys on its own schedule and may be offline for maintenance. What should the order service publish to?
Question 2: A standard queue keeps messages for 4 days, and so does its dead-letter queue. Failed messages that reach the DLQ after 3 days disappear about a day later, before anyone can inspect them. What should you change?
Question 3: Which TWO requirements point to an API Gateway REST API rather than an HTTP API? (Choose TWO.)
Question 4: Your support team uses a SaaS helpdesk that is an EventBridge partner. Every new high-priority ticket must start an AWS Step Functions workflow, with no polling code. What should you set up?

Build it

Build the fan-out pattern from this lesson in the Loose coupling lab. You'll create an SNS topic and two SQS queues with a filter policy and a dead-letter queue. You'll watch messages vanish when the queue policy is missing, and watch a poison message move to the DLQ.

Key takeaways

  • Loose coupling puts a durable component between producer and consumer, so each can fail, scale and change on its own.
  • SQS standard queues are at-least-once with best-effort order. Make consumers idempotent, size the visibility timeout to the processing time, and use FIFO only when the stem demands order or no duplicates.
  • Fan-out is SNS to one SQS queue per consumer. EventBridge is the answer for routing by event content, AWS service events, SaaS sources or schedules.
  • Step Functions runs multi-step processes: Standard for long-running, exactly-once work, and Express for short, high-volume, idempotent work.
  • Name the shape first: one consumer pulling work means a queue, a copy per subscriber means a topic, content routing means an event bus, ordered stateful steps mean a workflow.

Hands-on lab: Fan Out Orders with SNS, SQS and a Dead-Letter Queue

Sources

Last reviewed