Skip to content
Certification Guide

Lab · Task 2.1 · about 40 minutes

Fan Out Orders with SNS, SQS and a Dead-Letter Queue

Services in this lab

  • Amazon SNSin scope
  • Amazon SQSin scope

Lesson for this lab: Messaging and Events: Decoupling with Queues, Topics, Buses and Workflows

Goal

Build the fan-out pattern from 2.1a Messaging and events: one SNS topic delivers each order to a billing queue and, for physical orders only, a shipping queue, and billing failures move to a dead-letter queue.

Before you start

Why this matters on the exam: questions about "several systems need the same event", "a consumer may be offline" and "a message keeps failing" all land on this pattern. In this lab you also see two traps happen for real: a subscription that looks fine but delivers nothing, and a poison message that keeps coming back until the redrive policy moves it aside.

You create four resources: one SNS topic (lab-orders), two SQS standard queues (lab-billing, lab-shipping) and one SQS dead-letter queue (lab-billing-dlq), plus two subscriptions.

  • Use a sandbox AWS account, not production, with an AWS Budgets alert set. Sign in through IAM Identity Center or another role-based identity, never the root user.
  • Install the AWS CLI v2 and sign in (aws sso login or your usual role-based profile). Your identity needs permissions for sns:* and sqs:* on resources named lab-*.
  • Work in one Region throughout. The commands use a Bash shell; set these variables first and keep the terminal open:
export AWS_REGION=<your-region>      # for example us-east-1
export AWS_DEFAULT_REGION=$AWS_REGION

Steps

0 of 8 steps done

  1. Create the DLQ first, because the billing queue's redrive policy points at it. Give it the maximum retention (14 days) so it outlives the source queue's messages.

    Console: Amazon SQS > Create queue, type Standard, name lab-billing-dlq, Message retention period 14 days.

    DLQ_URL=$(aws sqs create-queue --queue-name lab-billing-dlq \
      --attributes MessageRetentionPeriod=1209600 --query QueueUrl --output text)
    DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
      --attribute-names QueueArn --query Attributes.QueueArn --output text)
    echo "$DLQ_ARN"
    
  2. The billing queue gets a short visibility timeout (10 seconds, so the lab moves quickly) and a redrive policy: once a message has been received 2 times without being deleted, its next receive moves it to the DLQ (maxReceiveCount 2). The shipping queue keeps the defaults.

    Console: Create queue lab-billing, Visibility timeout 10 seconds, Dead-letter queue enabled, choose lab-billing-dlq, Maximum receives 2. Then create lab-shipping with defaults.

    REDRIVE=$(printf '{"deadLetterTargetArn":"%s","maxReceiveCount":"2"}' "$DLQ_ARN")
    BILLING_ATTRS=$(printf '{"VisibilityTimeout":"10","RedrivePolicy":"%s"}' \
      "$(printf '%s' "$REDRIVE" | sed 's/"/\\"/g')")
    BILLING_URL=$(aws sqs create-queue --queue-name lab-billing \
      --attributes "$BILLING_ATTRS" --query QueueUrl --output text)
    SHIPPING_URL=$(aws sqs create-queue --queue-name lab-shipping \
      --query QueueUrl --output text)
    
    BILLING_ARN=$(aws sqs get-queue-attributes --queue-url "$BILLING_URL" \
      --attribute-names QueueArn --query Attributes.QueueArn --output text)
    SHIPPING_ARN=$(aws sqs get-queue-attributes --queue-url "$SHIPPING_URL" \
      --attribute-names QueueArn --query Attributes.QueueArn --output text)
    
  3. Use the CLI (or the SNS console's Create subscription page) for this step, not the SQS console's Subscribe to Amazon SNS topic action. The SQS console adds the queue policy for you, which would hide the trap in the next step.

    TOPIC_ARN=$(aws sns create-topic --name lab-orders --query TopicArn --output text)
    
    BILLING_SUB=$(aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs \
      --notification-endpoint "$BILLING_ARN" --return-subscription-arn \
      --query SubscriptionArn --output text)
    SHIPPING_SUB=$(aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs \
      --notification-endpoint "$SHIPPING_ARN" --return-subscription-arn \
      --query SubscriptionArn --output text)
    
    aws sns set-subscription-attributes --subscription-arn "$SHIPPING_SUB" \
      --attribute-name FilterPolicy --attribute-value '{"fulfillment":["physical"]}'
    

    The filter policy means shipping only receives messages whose fulfillment attribute is physical. Filter policy changes can take up to 15 minutes to take full effect, which is why you set it now.

    Run aws sns list-subscriptions-by-topic --topic-arn "$TOPIC_ARN": both subscriptions show real ARNs, not PendingConfirmation, because you own the queues. Everything looks ready.

  4. EXPECTED FAILURE. Publish a physical order and try to receive it from both queues.

    aws sns publish --topic-arn "$TOPIC_ARN" \
      --message '{"orderId":"1001","item":"desk lamp"}' \
      --message-attributes '{"fulfillment":{"DataType":"String","StringValue":"physical"}}'
    
    aws sqs receive-message --queue-url "$BILLING_URL" --wait-time-seconds 10
    aws sqs receive-message --queue-url "$SHIPPING_URL" --wait-time-seconds 10
    

    What you should see: publish returns a MessageId, so the publisher thinks it succeeded, but both receive-message calls return nothing at all after waiting 10 seconds. The subscriptions are confirmed, yet SNS has no permission to call sqs:SendMessage on your queues, so the messages never arrive. This is the trap from the lesson: decoupled systems fail quietly, and the producer can't tell.

  5. Add a queue policy that lets only this topic send messages, using the aws:SourceArn condition.

    Console: open each queue, Access policy > Edit, and add a statement with principal sns.amazonaws.com, action sqs:SendMessage, the queue's ARN as the resource, and an ArnEquals condition on aws:SourceArn set to the topic ARN.

    allow_topic() {
      local queue_url=$1 queue_arn=$2 policy attrs
      policy=$(printf '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sns.amazonaws.com"},"Action":"sqs:SendMessage","Resource":"%s","Condition":{"ArnEquals":{"aws:SourceArn":"%s"}}}]}' \
        "$queue_arn" "$TOPIC_ARN")
      attrs=$(printf '{"Policy":"%s"}' "$(printf '%s' "$policy" | sed 's/"/\\"/g')")
      aws sqs set-queue-attributes --queue-url "$queue_url" --attributes "$attrs"
    }
    allow_topic "$BILLING_URL" "$BILLING_ARN"
    allow_topic "$SHIPPING_URL" "$SHIPPING_ARN"
    
  6. aws sns publish --topic-arn "$TOPIC_ARN" \
      --message '{"orderId":"1002","item":"bookshelf"}' \
      --message-attributes '{"fulfillment":{"DataType":"String","StringValue":"physical"}}'
    
    aws sns publish --topic-arn "$TOPIC_ARN" \
      --message '{"orderId":"1003","item":"e-book"}' \
      --message-attributes '{"fulfillment":{"DataType":"String","StringValue":"digital"}}'
    

    Order 1001 from the failed step is gone for good. SNS classifies a delivery that an endpoint policy blocks as a client-side error: it doesn't retry, and with no dead-letter queue on the subscription it discards the message. Only orders published after the fix are delivered.

  7. Receive from shipping, then delete what you received. A consumer must delete a message after processing it; otherwise it reappears when the visibility timeout ends.

    aws sqs receive-message --queue-url "$SHIPPING_URL" \
      --max-number-of-messages 10 --wait-time-seconds 10
    

    You should see one message, and its Body (an SNS JSON envelope) contains order 1002. Order 1003 is missing on purpose: the filter policy dropped the digital order for this subscriber. If you also see 1003, the filter policy hasn't propagated yet; wait a few minutes and repeat with new order IDs.

    Delete it with the ReceiptHandle from the output:

    aws sqs delete-message --queue-url "$SHIPPING_URL" --receipt-handle '<receipt-handle>'
    
  8. Billing received both orders. Pretend the billing consumer crashes on every message: receive, never delete, and receive again after the 10-second visibility timeout.

    for attempt in 1 2 3; do
      echo "Attempt $attempt"
      aws sqs receive-message --queue-url "$BILLING_URL" \
        --max-number-of-messages 10 --wait-time-seconds 5 \
        --attribute-names ApproximateReceiveCount \
        --query 'Messages[].Attributes.ApproximateReceiveCount' --output text
      sleep 12
    done
    

    On attempts 1 and 2 you see the messages come back with a receive count of 1, then 2: each time the visibility timeout expires, the same messages become visible again. On attempt 3 the receive count would exceed maxReceiveCount (2), so SQS moves the messages to the DLQ instead of returning them, and the output is empty (None). Without a DLQ, a message like this would keep coming back until its retention period ends.

    If an attempt returned only one of the two messages, the other is one receive behind. Run the loop once more until the output stays empty.

Verify

Run each check. Counts in SQS are approximate and can take up to a minute to settle.

  1. The DLQ holds the two billing messages:
aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages

Expected: "ApproximateNumberOfMessages": "2".

  1. The billing and shipping queues are empty:
aws sqs get-queue-attributes --queue-url "$BILLING_URL" \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible
aws sqs get-queue-attributes --queue-url "$SHIPPING_URL" \
  --attribute-names ApproximateNumberOfMessages

Expected: every value is "0".

  1. The shipping subscription carries the filter:
aws sns get-subscription-attributes --subscription-arn "$SHIPPING_SUB" \
  --query Attributes.FilterPolicy

Expected: "{\"fulfillment\":[\"physical\"]}".

  1. In the console, open lab-billing-dlq and choose Send and receive messages > Poll for messages. The two messages are order 1002 and order 1003, ready to inspect or redrive once the "bug" is fixed.

Clean up

Delete everything in dependency order: subscriptions, then the topic, then the source queues, then the DLQ they point at.

aws sns unsubscribe --subscription-arn "$BILLING_SUB"
aws sns unsubscribe --subscription-arn "$SHIPPING_SUB"
aws sns delete-topic --topic-arn "$TOPIC_ARN"

aws sqs delete-queue --queue-url "$BILLING_URL"
aws sqs delete-queue --queue-url "$SHIPPING_URL"
aws sqs delete-queue --queue-url "$DLQ_URL"

Console equivalent: Amazon SNS > Subscriptions, delete both lab-orders subscriptions; Topics, delete lab-orders; Amazon SQS > Queues, delete lab-billing, lab-shipping and lab-billing-dlq.

Confirm nothing is left (deleted queues can take up to 60 seconds to disappear from listings):

aws sns list-topics --query "Topics[?contains(TopicArn, 'lab-orders')]"
aws sns list-subscriptions --query "Subscriptions[?contains(TopicArn, 'lab-orders')]"
aws sqs list-queues --queue-name-prefix lab-

Expected: [], [], and no QueueUrls in the last output. Also check the SNS and SQS consoles in this Region. This lab created no other billable resources.

Sources

Last reviewed