Lab · Task 2.1 · about 40 minutes
Fan Out Orders with SNS, SQS and a Dead-Letter Queue
Services in this lab
- Amazon SNSin scope
- Amazon SQSin scope
Lesson for this lab: Messaging and Events: Decoupling with Queues, Topics, Buses and Workflows
Goal
Build the fan-out pattern from 2.1a Messaging and events: one SNS topic delivers each order to a billing queue and, for physical orders only, a shipping queue, and billing failures move to a dead-letter queue.
Before you start
Why this matters on the exam: questions about "several systems need the same event", "a consumer may be offline" and "a message keeps failing" all land on this pattern. In this lab you also see two traps happen for real: a subscription that looks fine but delivers nothing, and a poison message that keeps coming back until the redrive policy moves it aside.
You create four resources: one SNS topic (lab-orders), two SQS standard queues (lab-billing, lab-shipping) and one SQS dead-letter queue (lab-billing-dlq), plus two subscriptions.
- Use a sandbox AWS account, not production, with an AWS Budgets alert set. Sign in through IAM Identity Center or another role-based identity, never the root user.
- Install the AWS CLI v2 and sign in (
aws sso loginor your usual role-based profile). Your identity needs permissions forsns:*andsqs:*on resources namedlab-*. - Work in one Region throughout. The commands use a Bash shell; set these variables first and keep the terminal open:
export AWS_REGION=<your-region> # for example us-east-1
export AWS_DEFAULT_REGION=$AWS_REGION
Steps
0 of 8 steps done
Create the DLQ first, because the billing queue's redrive policy points at it. Give it the maximum retention (14 days) so it outlives the source queue's messages.
Console: Amazon SQS > Create queue, type Standard, name
lab-billing-dlq, Message retention period 14 days.DLQ_URL=$(aws sqs create-queue --queue-name lab-billing-dlq \ --attributes MessageRetentionPeriod=1209600 --query QueueUrl --output text) DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \ --attribute-names QueueArn --query Attributes.QueueArn --output text) echo "$DLQ_ARN"The billing queue gets a short visibility timeout (10 seconds, so the lab moves quickly) and a redrive policy: once a message has been received 2 times without being deleted, its next receive moves it to the DLQ (maxReceiveCount 2). The shipping queue keeps the defaults.
Console: Create queue
lab-billing, Visibility timeout 10 seconds, Dead-letter queue enabled, chooselab-billing-dlq, Maximum receives 2. Then createlab-shippingwith defaults.REDRIVE=$(printf '{"deadLetterTargetArn":"%s","maxReceiveCount":"2"}' "$DLQ_ARN") BILLING_ATTRS=$(printf '{"VisibilityTimeout":"10","RedrivePolicy":"%s"}' \ "$(printf '%s' "$REDRIVE" | sed 's/"/\\"/g')") BILLING_URL=$(aws sqs create-queue --queue-name lab-billing \ --attributes "$BILLING_ATTRS" --query QueueUrl --output text) SHIPPING_URL=$(aws sqs create-queue --queue-name lab-shipping \ --query QueueUrl --output text) BILLING_ARN=$(aws sqs get-queue-attributes --queue-url "$BILLING_URL" \ --attribute-names QueueArn --query Attributes.QueueArn --output text) SHIPPING_ARN=$(aws sqs get-queue-attributes --queue-url "$SHIPPING_URL" \ --attribute-names QueueArn --query Attributes.QueueArn --output text)Use the CLI (or the SNS console's Create subscription page) for this step, not the SQS console's Subscribe to Amazon SNS topic action. The SQS console adds the queue policy for you, which would hide the trap in the next step.
TOPIC_ARN=$(aws sns create-topic --name lab-orders --query TopicArn --output text) BILLING_SUB=$(aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs \ --notification-endpoint "$BILLING_ARN" --return-subscription-arn \ --query SubscriptionArn --output text) SHIPPING_SUB=$(aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs \ --notification-endpoint "$SHIPPING_ARN" --return-subscription-arn \ --query SubscriptionArn --output text) aws sns set-subscription-attributes --subscription-arn "$SHIPPING_SUB" \ --attribute-name FilterPolicy --attribute-value '{"fulfillment":["physical"]}'The filter policy means shipping only receives messages whose
fulfillmentattribute isphysical. Filter policy changes can take up to 15 minutes to take full effect, which is why you set it now.Run
aws sns list-subscriptions-by-topic --topic-arn "$TOPIC_ARN": both subscriptions show real ARNs, notPendingConfirmation, because you own the queues. Everything looks ready.EXPECTED FAILURE. Publish a physical order and try to receive it from both queues.
aws sns publish --topic-arn "$TOPIC_ARN" \ --message '{"orderId":"1001","item":"desk lamp"}' \ --message-attributes '{"fulfillment":{"DataType":"String","StringValue":"physical"}}' aws sqs receive-message --queue-url "$BILLING_URL" --wait-time-seconds 10 aws sqs receive-message --queue-url "$SHIPPING_URL" --wait-time-seconds 10What you should see:
publishreturns aMessageId, so the publisher thinks it succeeded, but bothreceive-messagecalls return nothing at all after waiting 10 seconds. The subscriptions are confirmed, yet SNS has no permission to callsqs:SendMessageon your queues, so the messages never arrive. This is the trap from the lesson: decoupled systems fail quietly, and the producer can't tell.Add a queue policy that lets only this topic send messages, using the
aws:SourceArncondition.Console: open each queue, Access policy > Edit, and add a statement with principal
sns.amazonaws.com, actionsqs:SendMessage, the queue's ARN as the resource, and anArnEqualscondition onaws:SourceArnset to the topic ARN.allow_topic() { local queue_url=$1 queue_arn=$2 policy attrs policy=$(printf '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sns.amazonaws.com"},"Action":"sqs:SendMessage","Resource":"%s","Condition":{"ArnEquals":{"aws:SourceArn":"%s"}}}]}' \ "$queue_arn" "$TOPIC_ARN") attrs=$(printf '{"Policy":"%s"}' "$(printf '%s' "$policy" | sed 's/"/\\"/g')") aws sqs set-queue-attributes --queue-url "$queue_url" --attributes "$attrs" } allow_topic "$BILLING_URL" "$BILLING_ARN" allow_topic "$SHIPPING_URL" "$SHIPPING_ARN"aws sns publish --topic-arn "$TOPIC_ARN" \ --message '{"orderId":"1002","item":"bookshelf"}' \ --message-attributes '{"fulfillment":{"DataType":"String","StringValue":"physical"}}' aws sns publish --topic-arn "$TOPIC_ARN" \ --message '{"orderId":"1003","item":"e-book"}' \ --message-attributes '{"fulfillment":{"DataType":"String","StringValue":"digital"}}'Order 1001 from the failed step is gone for good. SNS classifies a delivery that an endpoint policy blocks as a client-side error: it doesn't retry, and with no dead-letter queue on the subscription it discards the message. Only orders published after the fix are delivered.
Receive from shipping, then delete what you received. A consumer must delete a message after processing it; otherwise it reappears when the visibility timeout ends.
aws sqs receive-message --queue-url "$SHIPPING_URL" \ --max-number-of-messages 10 --wait-time-seconds 10You should see one message, and its
Body(an SNS JSON envelope) contains order1002. Order1003is missing on purpose: the filter policy dropped the digital order for this subscriber. If you also see 1003, the filter policy hasn't propagated yet; wait a few minutes and repeat with new order IDs.Delete it with the
ReceiptHandlefrom the output:aws sqs delete-message --queue-url "$SHIPPING_URL" --receipt-handle '<receipt-handle>'Billing received both orders. Pretend the billing consumer crashes on every message: receive, never delete, and receive again after the 10-second visibility timeout.
for attempt in 1 2 3; do echo "Attempt $attempt" aws sqs receive-message --queue-url "$BILLING_URL" \ --max-number-of-messages 10 --wait-time-seconds 5 \ --attribute-names ApproximateReceiveCount \ --query 'Messages[].Attributes.ApproximateReceiveCount' --output text sleep 12 doneOn attempts 1 and 2 you see the messages come back with a receive count of
1, then2: each time the visibility timeout expires, the same messages become visible again. On attempt 3 the receive count would exceedmaxReceiveCount(2), so SQS moves the messages to the DLQ instead of returning them, and the output is empty (None). Without a DLQ, a message like this would keep coming back until its retention period ends.If an attempt returned only one of the two messages, the other is one receive behind. Run the loop once more until the output stays empty.
Verify
Run each check. Counts in SQS are approximate and can take up to a minute to settle.
- The DLQ holds the two billing messages:
aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
--attribute-names ApproximateNumberOfMessages
Expected: "ApproximateNumberOfMessages": "2".
- The billing and shipping queues are empty:
aws sqs get-queue-attributes --queue-url "$BILLING_URL" \
--attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible
aws sqs get-queue-attributes --queue-url "$SHIPPING_URL" \
--attribute-names ApproximateNumberOfMessages
Expected: every value is "0".
- The shipping subscription carries the filter:
aws sns get-subscription-attributes --subscription-arn "$SHIPPING_SUB" \
--query Attributes.FilterPolicy
Expected: "{\"fulfillment\":[\"physical\"]}".
- In the console, open
lab-billing-dlqand choose Send and receive messages > Poll for messages. The two messages are order1002and order1003, ready to inspect or redrive once the "bug" is fixed.
Clean up
Delete everything in dependency order: subscriptions, then the topic, then the source queues, then the DLQ they point at.
aws sns unsubscribe --subscription-arn "$BILLING_SUB"
aws sns unsubscribe --subscription-arn "$SHIPPING_SUB"
aws sns delete-topic --topic-arn "$TOPIC_ARN"
aws sqs delete-queue --queue-url "$BILLING_URL"
aws sqs delete-queue --queue-url "$SHIPPING_URL"
aws sqs delete-queue --queue-url "$DLQ_URL"
Console equivalent: Amazon SNS > Subscriptions, delete both lab-orders subscriptions; Topics, delete lab-orders; Amazon SQS > Queues, delete lab-billing, lab-shipping and lab-billing-dlq.
Confirm nothing is left (deleted queues can take up to 60 seconds to disappear from listings):
aws sns list-topics --query "Topics[?contains(TopicArn, 'lab-orders')]"
aws sns list-subscriptions --query "Subscriptions[?contains(TopicArn, 'lab-orders')]"
aws sqs list-queues --queue-name-prefix lab-
Expected: [], [], and no QueueUrls in the last output. Also check the SNS and SQS consoles in this Region. This lab created no other billable resources.
Sources
- https://docs.aws.amazon.com/sns/latest/dg/subscribe-sqs-queue-to-sns-topic.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-subscribe-queue-sns-topic.html
- https://docs.aws.amazon.com/sns/latest/dg/sns-subscription-filter-policies.html
- https://docs.aws.amazon.com/sns/latest/dg/sns-dead-letter-queues.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-messages.html
- https://docs.aws.amazon.com/cli/latest/userguide/cli_sqs_code_examples.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteQueue.html
Last reviewed