Lab · Task 3.5 · about 60 minutes
Shards Throttle, Parquet Scans Less
Services in this lab
- Amazon Kinesisin scope
- Amazon Athenain scope
- AWS Gluein scope
- Amazon S3in scope
Lesson for this lab: High-Performing Data Ingestion and Transformation
Goal
Overload a one-shard stream on purpose, read the partial failures, add a shard to clear them, and then show that a Parquet copy of a JSON table with a date partition scans far less data for the same query.
Before you start
- An AWS account you can experiment in, ideally a sandbox account with a budget alert. Don't use the root user. Sign in with an IAM Identity Center user or a role that can manage Kinesis streams, S3 buckets, Glue databases and Athena queries.
- AWS CLI v2 configured for that identity, and one Region for the whole lab (for example
us-east-1). Run every command in one terminal session so the shell variables carry over. - Python 3 on your machine for generating the test records. The lab creates no EC2 instances.
Resources this lab creates (the clean-up list): one Kinesis data stream named lab35-stream with one shard that is later resharded to two, one S3 bucket holding the raw JSON, the Athena results and the Parquet table, one Glue database named lab35 with two tables, and the Athena queries you run in the primary workgroup. Athena keeps query history for 45 days, so the clean-up doesn't try to delete it.
Steps
0 of 5 steps done
Use one bucket for the raw data, the Athena results and the Parquet table. Replace
<your-lab-bucket>with a globally unique name.export LAB_BUCKET=<your-lab-bucket> export REGION=us-east-1 aws s3 mb s3://$LAB_BUCKET --region $REGION aws kinesis create-stream --stream-name lab35-stream --shard-count 1 \ --stream-mode-details StreamMode=PROVISIONED --region $REGION aws glue create-database --database-input '{"Name":"lab35"}' --region $REGIONWait until the stream is active before you continue.
aws kinesis wait stream-exists --stream-name lab35-stream --region $REGION aws kinesis describe-stream-summary --stream-name lab35-stream --region $REGION \ --query 'StreamDescriptionSummary.OpenShardCount'Expected output:
1.EXPECTED FAILURE. One shard accepts 1 MB per second or 1,000 records per second of writes. This step runs several producer processes at the same time to push the stream past that limit. Some records in each
PutRecordsresponse come back withProvisionedThroughputExceededException. The call itself still succeeds, which is the trap: aPutRecordscall can return a success status while part of the batch fails. Your code must checkFailedRecordCountand retry the failed entries.Save this as
produce.pyin your working directory. The AWS CLI v2 expects blob values such asDataas base64 strings by default, so the script encodes each record before it callsput-records. Each process writes its own batch file, so parallel runs don't overwrite each other.import base64, json, os, random, string, subprocess, sys stream = "lab35-stream" region = os.environ.get("REGION", "us-east-1") batches = int(sys.argv[1]) if len(sys.argv) > 1 else 6 batch_file = f"batch-{os.getpid()}.json" def record(i): body = {"id": i, "page": "".join(random.choices(string.ascii_lowercase, k=1500))} return {"Data": json.dumps(body).encode(), "PartitionKey": str(i % 997)} failed_total = 0 for b in range(batches): records = [record(b * 500 + i) for i in range(500)] payload = [{"Data": base64.b64encode(r["Data"]).decode(), "PartitionKey": r["PartitionKey"]} for r in records] with open(batch_file, "w") as f: json.dump(payload, f) out = subprocess.run( ["aws", "kinesis", "put-records", "--stream-name", stream, "--records", f"file://{batch_file}", "--region", region, "--output", "json"], capture_output=True, text=True, check=True).stdout failed = json.loads(out)["FailedRecordCount"] failed_total += failed print(f"batch {b}: FailedRecordCount={failed}") print(f"total failed records: {failed_total}")Run six producer processes in parallel, each sending six batches of 500 records. Each batch is about 0.75 MB of data, and a single process sends its batches one after another, so one process alone may not reach the shard limit.
for p in 1 2 3 4 5 6; do REGION=$REGION python3 produce.py 6 & done; waitCheck
FailedRecordCountin each batch's output. It can be above zero in some batches. The total depends on your network and on how fast the CLI calls return. If every batch reports zero, add more producers or runproduce.pywith 20 batches. The call still returns HTTP 200 when some records fail, so a script that checks only for a transport or service error would miss the loss.Before the full run, send one batch and confirm the CLI accepts the base64
Datavalues.Split the stream to two shards with uniform scaling, which divides the hash key range evenly.
aws kinesis update-shard-count --stream-name lab35-stream \ --target-shard-count 2 --scaling-type UNIFORM_SCALING --region $REGION aws kinesis wait stream-exists --stream-name lab35-stream --region $REGION aws kinesis describe-stream-summary --stream-name lab35-stream --region $REGION \ --query 'StreamDescriptionSummary.OpenShardCount'Expected output:
2. Run the same parallel load again.for p in 1 2 3 4 5 6; do REGION=$REGION python3 produce.py 6 & done; waitThe failed count should fall. Two shards accept about 2 MB per second in total, so if your producers send more than that, some failures remain. Sizing a stream means matching the rate in the stem, not only making one run pass.
Write a small JSON dataset with two days of data into the raw prefix, then create a JSON table over it in the
lab35database. Athena needs a results location, so use the same bucket.Athena runs queries asynchronously, so a query can still be
QUEUEDorRUNNINGright after you start it. First define a small helper that polls until the query finishes and prints its final state:wait_query() { while true; do STATE=$(aws athena get-query-execution --query-execution-id "$1" --region $REGION \ --query 'QueryExecution.Status.State' --output text) case "$STATE" in QUEUED|RUNNING) sleep 2 ;; *) echo "$STATE"; return ;; esac done }Then write the data and create the JSON table:
python3 - <<'PY' > events.json import json, random for day in ["2026-11-14", "2026-11-15"]: for i in range(5000): print(json.dumps({"user_id": f"u{i % 500}", "page": random.choice(["home", "cart", "checkout", "search"]), "event_day": day, "note": "x" * 200})) PY aws s3 cp events.json s3://$LAB_BUCKET/raw/events/events.json --region $REGION cat > json-table.sql <<'SQL' CREATE EXTERNAL TABLE lab35.events_json ( user_id string, page string, event_day string, note string ) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' LOCATION 's3://<your-lab-bucket>/raw/events/'; SQL sed -i "s/<your-lab-bucket>/$LAB_BUCKET/" json-table.sql QID=$(aws athena start-query-execution --query-string file://json-table.sql \ --result-configuration OutputLocation=s3://$LAB_BUCKET/athena-results/ \ --region $REGION --query QueryExecutionId --output text) wait_query $QIDExpected output:
SUCCEEDED. Next, write a CTAS query that stores the same rows as Parquet, partitioned byevent_day. CTAS is one of the Athena transform patterns: it reads the JSON table and writes a new table in another format.cat > parquet-ctas.sql <<'SQL' CREATE TABLE lab35.events_parquet WITH ( format = 'PARQUET', parquet_compression = 'SNAPPY', partitioned_by = ARRAY['event_day'], external_location = 's3://<your-lab-bucket>/curated/events/' ) AS SELECT user_id, page, note, event_day FROM lab35.events_json; SQL sed -i "s/<your-lab-bucket>/$LAB_BUCKET/" parquet-ctas.sql QID2=$(aws athena start-query-execution --query-string file://parquet-ctas.sql \ --result-configuration OutputLocation=s3://$LAB_BUCKET/athena-results/ \ --region $REGION --query QueryExecutionId --output text) wait_query $QID2Expected output:
SUCCEEDED. Thepartitioned_bycolumn must be the last column in the SELECT list, which is whyevent_daycomes last. The CTASexternal_locationmust be empty and must not be set when the workgroup enforces its own result location, because Athena fails a CTAS that specifies one; if the CTAS fails with a location error, check the workgroup's result settings first.Run one filtered query against each table. The query reads one day and two columns. Athena reports how many bytes it scanned once the query finishes, so the loop waits with
wait_querybefore it reads the statistics. Run it in the same terminal, where the helper is defined.for T in events_json events_parquet; do Q=$(aws athena start-query-execution --region $REGION \ --query-string "SELECT page, COUNT(*) FROM lab35.$T WHERE event_day = '2026-11-15' GROUP BY page" \ --query-execution-context Database=lab35 \ --result-configuration OutputLocation=s3://$LAB_BUCKET/athena-results/ \ --query QueryExecutionId --output text) wait_query $Q > /dev/null aws athena get-query-execution --query-execution-id $Q --region $REGION \ --query '[QueryExecution.Statistics.DataScannedInBytes, QueryExecution.Status.State]' --output text | sed "s/^/$T: /" doneExpected output: two lines, one per table, both
SUCCEEDED. The Parquet line reports far fewer bytes scanned than the JSON line, because the JSON table reads every row of both days and every column, while the Parquet table reads one partition folder and only thepagecolumn. Small files can hide the gap, so compare the two numbers rather than their size on disk.
Verify
aws kinesis describe-stream-summary --stream-name lab35-stream --region $REGION --query 'StreamDescriptionSummary.OpenShardCount'returns2.- The second parallel run of
produce.pyreports fewer failed records than the first run, which reported failures in at least one batch. aws glue get-table --database-name lab35 --name events_parquet --region $REGION --query 'Table.PartitionKeys'listsevent_day.aws s3 ls s3://$LAB_BUCKET/curated/events/ --recursive --region $REGIONshows one folder perevent_day=value, each containing Parquet files.- The Parquet query line in step 5 shows fewer bytes scanned than the JSON query line.
Clean up
Delete every resource in dependency order, then confirm nothing is left. Also remove the local files this lab wrote: produce.py, batch-*.json, events.json, json-table.sql and parquet-ctas.sql. They hold test records and SQL only.
aws glue delete-table --database-name lab35 --name events_parquet --region $REGION
aws glue delete-table --database-name lab35 --name events_json --region $REGION
aws glue delete-database --name lab35 --region $REGION
aws kinesis delete-stream --stream-name lab35-stream --region $REGION
aws kinesis wait stream-not-exists --stream-name lab35-stream --region $REGION
aws s3 rb s3://$LAB_BUCKET --force --region $REGION
Final check: nothing from this lab remains.
aws kinesis list-streams --region $REGION --query 'StreamNames'
aws glue get-databases --region $REGION --query 'DatabaseList[?Name==`lab35`]'
aws s3 ls | grep "$LAB_BUCKET" || echo "bucket removed"
Expected output: the stream list has no lab35-stream, the database query returns an empty list, and the bucket check prints bucket removed.
Sources
- https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html
- https://docs.aws.amazon.com/streams/latest/dev/how-do-i-size-a-stream.html
- https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-resharding-scaling.html
- https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-sdk.html
- https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html
- https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration-changes.html
- https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html
- https://docs.aws.amazon.com/athena/latest/ug/partition-projection.html
- https://docs.aws.amazon.com/athena/latest/ug/ctas.html
- https://docs.aws.amazon.com/athena/latest/ug/ctas-considerations-limitations.html
- https://docs.aws.amazon.com/athena/latest/ug/workgroups-troubleshooting.html
Last reviewed