Skip to content
Certification Guide

Task 3.5 · 22 min read

High-Performing Data Ingestion and Transformation

Services in this lesson

  • Amazon Kinesisin scope
  • Amazon MSKscope not listed
  • AWS Gluein scope
  • Amazon Athenain scope
  • Amazon EMRin scope
  • AWS Lake Formationin scope
  • AWS DataSyncin scope
  • AWS Snow Familyin scope
  • AWS Storage Gatewayin scope
  • Amazon Quickin scope
  • Amazon S3in scope

What the exam tests

  • Whether you pick the streaming service from how the data is read. Kinesis Data Streams fits several consumers reading records in order within each shard, during a time window. Firehose fits when the goal is files in S3.
  • Whether you know when Amazon MSK (Managed Streaming for Apache Kafka) fits. Apache Kafka is an open-source streaming platform, and MSK fits producers or consumers that already use its APIs.
  • Whether you can size a Kinesis stream from the write and read rates in the stem. That means shards in provisioned mode, or on-demand capacity mode when the load is unpredictable.
  • Whether you can read the Firehose buffering choices in a stem: buffer size, buffer interval, and whichever fills first. You should know what each does to latency and file size.
  • Whether you choose the transformation engine by workload shape. AWS Glue runs managed Spark ETL (extract, transform, load) jobs, Amazon EMR runs custom big data frameworks, and Amazon Athena runs SQL over data already in S3.
  • Whether you recognize the data format and layout that make later queries fast. That means Parquet (a columnar file format), compression, splittable files, and partitions that match the query filters.

This lesson owns Kinesis Data Streams and Firehose throughput and buffering, and the choice between Kinesis and MSK. It also covers AWS Glue for ETL throughput, and Athena and EMR for transformation speed.

It also covers Lake Formation for organizing the lake, and the file choices that drive query speed. DataSync and Snow Family appear only for how fast they move bulk data.

Related topics live elsewhere. Messaging and event routing with Amazon SQS, Amazon SNS and Amazon EventBridge is in 2.1a Messaging and events. S3 request rates and prefix design are in 3.1 Storage performance.

Data transfer cost is in 4.4 Network cost. DataSync and Snow Family cost, and storage classes, are in 4.1 Storage cost.

The core concept

Ingestion performance has two halves: how fast data gets into a data lake without loss or throttling, and how fast it can be read later. A data lake is one central store, here Amazon S3, that holds raw and processed data as files.

Throttling means the service rejects requests that go past a capacity limit. Most wrong answers fix one half and ignore the other.

Two failures show up again and again. One is a stream that accepts every record but writes thousands of tiny files to S3. The other is a fast query engine pointed at CSV (comma-separated values) files, so every query scans every column of every row.

An analogy: a parcel depot

Picture a parcel depot. Parcels arrive on several conveyor belts, and each belt can carry only so many parcels a minute. A label on each parcel decides which belt it rides, and inspectors watch the belts as parcels pass.

At the end of the belts, a van waits at the dock. It leaves when it is full or when a set time has passed, whichever comes first. The warehouse it delivers to stores parcels on shelves sorted by date, so a clerk looking for Tuesday's parcels walks straight to one aisle.

Where the analogy breaks: an inspector who takes a parcel off a real belt removes it. A stream keeps each record for a time window, and every reader sees the same records.

Walk one reading through it

The diagram shows the whole pipeline. The walkthrough follows one sensor reading from a buoy to an analyst's query.

Enlarge
One lake, several writers. Each stage has a throughput or layout choice the stem can test.

Tidewater Sensors runs buoys that measure water quality along a coast. Zainab is the data engineer who built the pipeline, and Oskar is the analyst who queries it.

  1. A buoy sends a reading. The buoy is a producer: anything that writes records into a stream. It writes to Amazon Kinesis Data Streams, a service that keeps records in order within each shard.
  2. The partition key picks a shard. The stream is split into shards, fixed units of capacity. Each record carries a partition key, and Zainab uses the buoy ID, so records spread across shards.
  3. The shard accepts the write. In provisioned mode, each shard accepts 1 MB per second or 1,000 records per second of writes. Past that, the producer is throttled.
  4. An alert service reads the record at once. It is a consumer, a program that reads from the stream. Zainab registered it for enhanced fan-out, so it gets its own 2 MB per second per shard.
  5. Amazon Data Firehose picks up the same record. Firehose is a delivery service, not a stream you read from. It holds records in a memory buffer and writes them to S3 in batches.
  6. Firehose converts the batch to Parquet. It uses a schema from the AWS Glue Data Catalog, a central catalog of table definitions (schema and S3 location). The files land in S3 already columnar: each column is stored together.
  7. The file lands in a date folder. The S3 prefix holds year and month, so the lake is partitioned by date.
  8. A nightly Glue job cleans new files. It is an ETL (extract, transform, load) job on AWS Glue. Job bookmarks let it skip files it already processed.
  9. Oskar queries one month. Amazon Athena runs SQL directly over the files in S3. The date partitions limit Athena to one month's folder, and Parquet limits it to the columns Oskar names.
  10. AWS Lake Formation decides who may read. It governs the registered S3 location and the tables, so Oskar's team sees only the tables it is allowed to see.

Steps 3 and 4 are the "get in" half. Steps 6 to 9 are the "read later" half, and they matter as much.

The rules the exam tests

  • Kinesis Data Streams keeps records ordered within each shard, for real-time reads by several consumers. The partition key decides the shard.
  • Shard limits (provisioned mode): writes of 1 MB per second or 1,000 records per second per shard. Reads of 2 MB per second per shard, shared by all standard consumers.
  • Enhanced fan-out gives each registered consumer its own 2 MB per second per shard. A stream can have up to 20 registered consumers.
  • On-demand mode sets no shard counts. It scales to up to double the peak write rate of the previous 30 days.
  • On-demand can still throttle. Traffic that passes double the previous peak within about 15 minutes gets throttled.
  • Firehose delivers when the buffer size or the buffer interval is reached, whichever comes first. A bigger buffer means fewer, larger files.
  • MSK is the answer when the stem names Kafka. Without Kafka in the stem, Kinesis is usually the simpler managed answer.
  • Engines: Glue is serverless Spark ETL, EMR is a cluster you size, and Athena is SQL over S3 data.
  • Columnar files plus partitions on the filter columns decide how much data each query reads.
Deep diveFirehose buffer numbers and format conversion

The buffer size and interval are hints, so Firehose may use different values when that is optimal. For an S3 destination, the buffer size runs from 1 to 128 MB.

The buffer interval runs from 0 to 900 seconds. The defaults are 5 MB and 300 seconds, and the first condition met triggers delivery.

A small buffer gives lower latency and many small files. A large buffer gives fewer, larger files, which query engines read faster, at the cost of waiting.

Firehose can convert JSON records to Parquet or ORC (Optimized Row Columnar). It reads the schema from a table in the Glue Data Catalog. With conversion turned on, the buffer size can't be set below 64 MB, and the default becomes 128 MB.

Deep diveWhy columnar files and partitions are fast

A columnar format such as Parquet or ORC stores each column together. A query that selects three columns out of 80 reads only those three.

Parquet and ORC also compress each column and are splittable, so several readers can work on one file in parallel.

Partitioning by a column that appears in the filters, such as date, lets the engine skip whole folders. That is partition pruning.

Athena can compute partition values from table properties with partition projection. That avoids a Glue Data Catalog lookup for every query on a heavily partitioned table.

The walkthrough picked Kinesis and Glue, but a stem may point to a different streaming service or transform engine.

Kinesis or MSK

Amazon MSK runs Apache Kafka with Kafka's own APIs (application programming interfaces). It is right when the team already runs Kafka, or depends on Kafka client libraries or topics.

It also fits when the team needs Kafka features, such as consumer groups (sets of consumers that split a topic's records among them). Moving Kafka producers and consumers to Kinesis would mean rewriting them.

Choosing the transformation engine

The three engines differ in how much infrastructure you manage.

  • AWS Glue runs serverless jobs on Apache Spark, an open-source engine for large-scale data processing. Auto Scaling adds workers while a job runs, and job bookmarks let a rerun read only new files.
  • Amazon EMR runs Spark, Apache Hive (SQL-style queries on a cluster) and other open-source frameworks on a cluster you size. EMR Serverless is the other option: AWS manages the workers, and you submit jobs to an application.
  • Amazon Athena can transform data with CTAS (CREATE TABLE AS SELECT) and INSERT INTO, but it is not a Spark engine. Its speed depends on how the data is stored in S3.
Deep diveGlue workers and bookmarks

Glue workers are sized in DPUs (data processing units). One G.1X worker is 1 DPU, with 4 vCPUs (virtual CPUs) and 16 GB of memory.

Job bookmarks record which S3 data a job already processed. A transformation context is a name that identifies each source, transform or sink for bookmark tracking.

Without bookmarks, each run reads every input file again. A rerun then repeats work and can duplicate output.

Other pieces the stem may add

Three more pieces complete the ingestion picture.

  • AWS Storage Gateway (S3 File Gateway) presents an NFS (Network File System) or SMB (Server Message Block) file share on premises. It copies the files into an S3 bucket, so applications that write files keep their file interface.
  • Amazon Quick builds dashboards on data that Athena queries. It visualizes the lake without changing how the data is ingested.
  • Producers in a VPC (virtual private cloud) reach Kinesis through an interface VPC endpoint, a private entry point to the service inside the VPC. The IAM (Identity and Access Management) role on each producer decides which streams it may write to.
Deep diveBulk moves: DataSync and offline transfer

AWS DataSync moves files over the network using an agent, a virtual machine appliance you deploy next to your storage. For millions of small files, run several tasks in parallel, each with its own agent.

Several agents on one location don't add high availability. Every agent assigned to a location must be online before a task can start.

When the data is too large for the network window, a physical device moves it faster. AWS no longer offers Snow Family devices to new customers, so treat Snow Family as legacy. Partner-provided offline transfer is the alternative.

Decision guide

Decision guide
If the question says…Reach for…Because…
Several consumers must read the same real-time records, and one of them needs its own read throughputKinesis Data Streams with enhanced fan-out consumersEach registered consumer gets 2 MB per second per shard, so a busy one doesn't slow the others.
Unpredictable traffic, and the team does not want to manage shard countsKinesis Data Streams in on-demand modeIt scales to double the last 30 days' peak. Passing that within about 15 minutes can throttle.
Steady, known write rate, and the stem gives a number of records or megabytes per secondProvisioned mode, with shards = max of write MB/s ÷ 1 and write records/s ÷ 1,000Each shard is a fixed unit. Sizing for the peak keeps producers from being throttled.
Records must reach S3 as files in near real time, with no consumer codeFirehose delivery stream to S3, with the buffer tuned for file sizeFirehose handles batching, retries and delivery. The interval caps how long a record waits.
Analysts query the S3 data with SQL and the stream writes JSON, and query cost and speed matterFirehose record format conversion to Parquet, with a Glue table as the schemaFiles land columnar and compressed, so Athena reads only the columns each query selects.
The team already runs Apache Kafka, uses Kafka client libraries or topics, or must keep Kafka compatibilityAmazon MSKMSK keeps the Kafka APIs. Moving to Kinesis would mean rewriting producers and consumers.
Spark ETL on a schedule, with no cluster to manage, and the job reprocesses only new filesAWS Glue ETL job with job bookmarks and Auto ScalingGlue manages and scales Spark workers, and skips data it already processed.
A custom big data framework such as Hive, Presto or a specific Spark version, or long-running cluster controlAmazon EMR on EC2 or on EKSYou choose the cluster shape and frameworks. Pick EMR when Glue's runtime can't run the job.
Spark or Hive jobs run in bursts, and the stem wants no cluster lifecycle at allEMR ServerlessAn application runs jobs on demand, and AWS provisions and releases the workers.
SQL over S3 data that is already in place, with no ETL neededAmazon Athena with Parquet files and partition projectionAthena queries in place. Columnar files and filter-column partitions cut the data scanned.
A lake must restrict which roles see which tables, columns or S3 paths, from one placeAWS Lake Formation with a registered S3 location and table permissionsPermissions apply to catalog tables and the registered path, not one policy per prefix.
Millions of small files must move from on premises, and one agent is the bottleneckSeveral DataSync tasks, each with its own agent, in parallelParallel tasks spread the work. Extra agents on one location add no availability.
Bulk data is too large for the network to finish in the window the stem givesPartner-provided offline transfer, or Snow Family for existing customersA device beats the network. Snow Family is closed to new customers. Cost is in 4.1.
Applications on premises keep writing files over NFS or SMB, and those files must reach S3 for analyticsAWS Storage Gateway (S3 File Gateway)It keeps the file share the applications use and copies each file into S3.
Analysts need dashboards and charts over the S3 lake, with no code to writeAmazon Quick, with Athena as the data sourceQuick visualizes what Athena queries. It doesn't ingest or transform the data.
Producers in a VPC must write to Kinesis without traffic leaving the Amazon networkInterface VPC endpoint for Kinesis Data Streams, with an IAM role per producerThe endpoint uses AWS PrivateLink. Each IAM role limits which streams a producer can write to.

Correct pattern

REFERENCE PATTERN: a clickstream producer writes to a Kinesis stream. A Firehose delivery stream converts each batch to Parquet in a partitioned S3 prefix, and Athena queries the result through partition projection.

# 1. Stream: on-demand mode, so shard counts are not set by hand.
aws kinesis create-stream --stream-name clickstream \
  --stream-mode-details StreamMode=ON_DEMAND

# 2. Firehose: read from the stream, convert JSON to Parquet with a Glue table schema,
#    buffer 128 MB or 300 seconds, whichever comes first.
aws firehose create-delivery-stream --delivery-stream-name clickstream-to-lake \
  --delivery-stream-type KinesisStreamAsSource \
  --kinesis-stream-source-configuration \
    "KinesisStreamARN=<stream-arn>,RoleARN=<firehose-role-arn>" \
  --extended-s3-destination-configuration file://s3-destination.json

The s3-destination.json file sets these fields:

  • BucketARN, and a Prefix that includes year=!{timestamp:yyyy}/month=!{timestamp:MM}/.
  • An ErrorOutputPrefix. Firehose requires one when Prefix contains expressions.
  • BufferingHints, with SizeInMBs of 128 and IntervalInSeconds of 300.
  • A DataFormatConversionConfiguration that points at the Glue table.

With conversion turned on, SizeInMBs can't be set below 64. Check the Firehose API reference for the full field list before you write the file.

-- 3. Athena: partition projection avoids a Glue catalog lookup for every partition.
CREATE EXTERNAL TABLE clicks (
  user_id string,
  page string,
  event_ts timestamp
)
PARTITIONED BY (year string, month string)
STORED AS PARQUET
LOCATION 's3://<your-lake-bucket>/clickstream/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.year.type' = 'integer',
  'projection.year.range' = '2024,2030',
  'projection.month.type' = 'integer',
  'projection.month.range' = '1,12',
  'projection.month.digits' = '2',
  'storage.location.template' = 's3://<your-lake-bucket>/clickstream/year=${year}/month=${month}/'
);

Why it works: the stream absorbs bursts without shard planning. Firehose turns many small records into a few large columnar files, and the partition folders match the analysts' filters.

Athena then reads only the month and the columns a query names. Lake Formation can sit on top of the same registered location to decide who may read the table.

Traps

TRAPExam traps
The trapWhy it failsCorrect pattern
Setting Firehose to a 60-second buffer to get Parquet files of 128 MB from a stream that writes 1 MB per minuteThe interval fires before the size, so each file holds about 1 MB. Queries open thousands of them.Lower the interval only if latency matters, and accept smaller files. For columnar output, size the stream to fill the buffer in the chosen interval.
Adding standard consumers and expecting each to get its own read throughputStandard consumers share 2 MB per second per shard. Five busy consumers slow each other down.Register enhanced fan-out consumers for the consumers that need dedicated throughput, or add shards.
Using on-demand mode to absorb a spike ten times the last month's peakOn-demand scales to double the 30-day peak. Passing that within about 15 minutes throttles.Use provisioned mode with shards sized for the known spike.
Choosing a Kinesis Data Streams partition key with few distinct values, such as a region codeMost records land on one or two shards, which hit 1 MB per second while the rest sit idle.Use a key with many distinct values, such as a device or user ID, so writes spread across shards.
Writing CSV from Glue or Firehose to S3 and then querying it with Athena for speedCSV has no column pruning and no default compression, so every query scans whole rows.Convert to Parquet or ORC during ingestion with Firehose, or in the Glue job, and compress it.
Relying on multiple DataSync agents for high availabilityEvery agent on a location must be online before a task starts. Extra agents add throughput only.Run parallel tasks, one agent each, for speed. For availability, plan a second path or a partner-provided offline transfer.
Choosing MSK because the stem says "streaming" with no mention of KafkaMSK brings a Kafka cluster to run and Kafka client code to keep, for no stated reason.Use Kinesis Data Streams for AWS-native streaming. Choose MSK when Kafka APIs, topics or client libraries are named in the stem.
Running an Athena query on a table with thousands of partitions and assuming the Glue catalog is freeAthena asks the Glue Data Catalog for every partition before pruning, which slows each query.Use partition projection on the table so Athena computes the partition values from its properties.
Expecting Glue to restart a failed job from where it stopped without bookmarksWithout job bookmarks, each run rereads every input file and can duplicate output.Enable job bookmarks and give each read a transformation context, so only new data is processed on the next run.
Giving a producer a long-lived IAM user access key in its code, so that it can write to KinesisA long-lived key leaks easily and works wherever it is copied. A producer on AWS doesn't need one.Give the producer an IAM role that allows kinesis:PutRecords on the one stream ARN, and reach the stream through an interface VPC endpoint.
Expecting Amazon Quick to make a slow Athena query fasterQuick only draws charts. The query scans the same bytes, so a dashboard over JSON stays slow.Fix the scan first with Parquet and partition projection in Athena, then point Quick at the table.

Compare

Compare
AspectKinesis Data StreamsAmazon Data FirehoseAmazon MSK
What it isOrdered, replayable stream with shardsManaged delivery of buffered data to a destinationManaged Apache Kafka clusters
Who reads the dataYour consumers, shared or enhanced fan-outThe destination, such as S3, with no consumer codeYour Kafka consumers, through the Kafka APIs
Capacity you setShards (provisioned) or none (on-demand)None; buffer size and interval per streamBroker instance types and storage
Typical latency knobConsumer polling and enhanced fan-outBuffer interval, from 0 to 900 seconds for S3Producer and consumer settings in the Kafka client
Reach for it whenReal-time consumers, AWS-native streamingFiles in S3, optionally converted to ParquetExisting Kafka code, topics or tooling
Compare
AspectAWS GlueAmazon EMR (on EC2 or EKS)Amazon Athena
RoleServerless ETL, Spark jobs and crawlersBig data clusters for Spark, Hive and othersInteractive SQL over data in S3
Infrastructure you manageNone; you choose the worker type and countCluster shape and frameworks (EC2 or EKS)None
Incremental processingJob bookmarksYour own state handlingNot applicable; it reads what is stored
Performance leversWorker type, Auto Scaling, partitioned outputInstance types, cluster size, Spark tuningParquet or ORC, partitions, partition projection
Reach for it whenManaged Spark ETL on a schedule or on arrivalFramework control, or a job Glue can't runAd hoc SQL over S3 with no pipeline to run

Worked exam scenario

Exam scenario

A retailer streams point-of-sale events from 4,000 stores into an analytics lake. The events arrive as JSON about 60 MB per minute. Analysts run SQL in Athena, mostly for one day at a time, and report that queries scan far more data than they need and take minutes. The team wants the LEAST operational overhead. What should a solutions architect do?

  1. A.Keep the JSON files, and add more Athena workgroups so queries run in parallel.
  2. B.Write the events to an Amazon EMR cluster that runs Hive queries against local disks.
  3. C.Deliver the stream through Amazon Data Firehose to S3 with Parquet conversion, partition the prefix by date, and use partition projection in Athena.
  4. D.Move the events into Amazon MSK and have analysts query the Kafka topics with SQL.
Show the walkthrough
  1. Highlight the keywords. JSON input, Athena, queries for one day at a time, and the LEAST operational overhead. The queries already use Athena, so the fix must change how the data is stored.
  2. Eliminate A. More workgroups change who runs the queries, not how much data they scan. The JSON files still force every query to read whole rows.
  3. Eliminate B. An EMR cluster adds infrastructure to run and doesn't change the JSON layout for Athena. It also goes against the lowest-overhead requirement.
  4. Eliminate D. MSK is a streaming platform, not an SQL store for analysts. The stem doesn't mention Kafka, so the cluster is extra work for the same data problem.
  5. Land on C. Firehose converts the JSON to Parquet as it lands and buffers it into larger files. Date partitions limit Athena to one day's folder, and Parquet limits it to the named columns. Partition projection skips the catalog lookup.

Answer: C

Check your understanding

Question 1: A telemetry system sends 3,500 records per second, each 0.5 KB, to one provisioned Kinesis Data Streams stream that has one shard. Producers are throttled on writes. How many shards does the stream need?
Question 2: A producer calls PutRecords with 500 records. The response has HTTP status 200 and a FailedRecordCount of 12. Which TWO actions keep every record from being lost? (Choose TWO.) (Choose TWO.)
Question 3: A company runs a nightly Glue ETL job that reads new JSON files from an S3 prefix. Last week a rerun after a failure duplicated yesterday's output. What should the team change?
Question 4: An IoT platform writes 50,000 records per second to an application that already runs on Apache Kafka with Kafka client libraries. Several consumers rely on consumer groups and Kafka topics. Which service fits best with the least rewrite of existing code?
Question 5: A data lake holds finance tables in S3. Analysts in one team may read only the finance tables and must not see the payroll columns. The team wants one place to control this across Athena and Glue jobs. Which solution fits?

Build it

The Build it lab for 3.5 builds a small S3 data lake and compares a JSON table with a partitioned Parquet table in Athena. It then shows a Kinesis shard limit being hit and cleans up everything it creates.

Key takeaways

  • Kinesis Data Streams sets its throughput in shards: 1 MB per second and 1,000 records per second of writes, and 2 MB per second of reads per shard. Enhanced fan-out gives each registered consumer its own 2 MB per second per shard.
  • Kinesis keeps records ordered within each shard. On-demand mode scales to up to double the 30-day peak, and passing that within about 15 minutes can throttle.
  • Firehose delivers when the buffer size or interval is reached, whichever comes first. A larger buffer makes larger files, and with Parquet conversion on, the buffer must be at least 64 MB.
  • Use MSK when the stem names Kafka or Kafka clients. Pick Glue, EMR or Athena for transforms by how much infrastructure you want to manage.
  • Choose the file layout by how the data will be queried. Columnar Parquet or ORC with partitions on the filter columns cuts each scan, and partition projection removes the per-query catalog lookup.
  • Match the ingestion path to the arrival pattern: Kinesis Data Streams for ordered real-time reads, Firehose for buffered delivery to S3, and MSK when the team already runs Kafka.

Hands-on lab: Shards Throttle, Parquet Scans Less

Sources

Last reviewed