Skip to content
Certification Guide

Lab · Task 3.5 · about 60 minutes

Shards Throttle, Parquet Scans Less

Services in this lab

  • Amazon Kinesisin scope
  • Amazon Athenain scope
  • AWS Gluein scope
  • Amazon S3in scope

Lesson for this lab: High-Performing Data Ingestion and Transformation

Goal

Overload a one-shard stream on purpose, read the partial failures, add a shard to clear them, and then show that a Parquet copy of a JSON table with a date partition scans far less data for the same query.

Before you start

  • An AWS account you can experiment in, ideally a sandbox account with a budget alert. Don't use the root user. Sign in with an IAM Identity Center user or a role that can manage Kinesis streams, S3 buckets, Glue databases and Athena queries.
  • AWS CLI v2 configured for that identity, and one Region for the whole lab (for example us-east-1). Run every command in one terminal session so the shell variables carry over.
  • Python 3 on your machine for generating the test records. The lab creates no EC2 instances.

Resources this lab creates (the clean-up list): one Kinesis data stream named lab35-stream with one shard that is later resharded to two, one S3 bucket holding the raw JSON, the Athena results and the Parquet table, one Glue database named lab35 with two tables, and the Athena queries you run in the primary workgroup. Athena keeps query history for 45 days, so the clean-up doesn't try to delete it.

Steps

0 of 5 steps done

  1. Use one bucket for the raw data, the Athena results and the Parquet table. Replace <your-lab-bucket> with a globally unique name.

    export LAB_BUCKET=<your-lab-bucket>
    export REGION=us-east-1
    
    aws s3 mb s3://$LAB_BUCKET --region $REGION
    aws kinesis create-stream --stream-name lab35-stream --shard-count 1 \
      --stream-mode-details StreamMode=PROVISIONED --region $REGION
    aws glue create-database --database-input '{"Name":"lab35"}' --region $REGION
    

    Wait until the stream is active before you continue.

    aws kinesis wait stream-exists --stream-name lab35-stream --region $REGION
    aws kinesis describe-stream-summary --stream-name lab35-stream --region $REGION \
      --query 'StreamDescriptionSummary.OpenShardCount'
    

    Expected output: 1.

  2. EXPECTED FAILURE. One shard accepts 1 MB per second or 1,000 records per second of writes. This step runs several producer processes at the same time to push the stream past that limit. Some records in each PutRecords response come back with ProvisionedThroughputExceededException. The call itself still succeeds, which is the trap: a PutRecords call can return a success status while part of the batch fails. Your code must check FailedRecordCount and retry the failed entries.

    Save this as produce.py in your working directory. The AWS CLI v2 expects blob values such as Data as base64 strings by default, so the script encodes each record before it calls put-records. Each process writes its own batch file, so parallel runs don't overwrite each other.

    import base64, json, os, random, string, subprocess, sys
    
    stream = "lab35-stream"
    region = os.environ.get("REGION", "us-east-1")
    batches = int(sys.argv[1]) if len(sys.argv) > 1 else 6
    batch_file = f"batch-{os.getpid()}.json"
    
    def record(i):
        body = {"id": i, "page": "".join(random.choices(string.ascii_lowercase, k=1500))}
        return {"Data": json.dumps(body).encode(), "PartitionKey": str(i % 997)}
    
    failed_total = 0
    for b in range(batches):
        records = [record(b * 500 + i) for i in range(500)]
        payload = [{"Data": base64.b64encode(r["Data"]).decode(), "PartitionKey": r["PartitionKey"]}
                   for r in records]
        with open(batch_file, "w") as f:
            json.dump(payload, f)
        out = subprocess.run(
            ["aws", "kinesis", "put-records", "--stream-name", stream,
             "--records", f"file://{batch_file}", "--region", region, "--output", "json"],
            capture_output=True, text=True, check=True).stdout
        failed = json.loads(out)["FailedRecordCount"]
        failed_total += failed
        print(f"batch {b}: FailedRecordCount={failed}")
    print(f"total failed records: {failed_total}")
    

    Run six producer processes in parallel, each sending six batches of 500 records. Each batch is about 0.75 MB of data, and a single process sends its batches one after another, so one process alone may not reach the shard limit.

    for p in 1 2 3 4 5 6; do REGION=$REGION python3 produce.py 6 & done; wait
    

    Check FailedRecordCount in each batch's output. It can be above zero in some batches. The total depends on your network and on how fast the CLI calls return. If every batch reports zero, add more producers or run produce.py with 20 batches. The call still returns HTTP 200 when some records fail, so a script that checks only for a transport or service error would miss the loss.

    Before the full run, send one batch and confirm the CLI accepts the base64 Data values.

  3. Split the stream to two shards with uniform scaling, which divides the hash key range evenly.

    aws kinesis update-shard-count --stream-name lab35-stream \
      --target-shard-count 2 --scaling-type UNIFORM_SCALING --region $REGION
    
    aws kinesis wait stream-exists --stream-name lab35-stream --region $REGION
    aws kinesis describe-stream-summary --stream-name lab35-stream --region $REGION \
      --query 'StreamDescriptionSummary.OpenShardCount'
    

    Expected output: 2. Run the same parallel load again.

    for p in 1 2 3 4 5 6; do REGION=$REGION python3 produce.py 6 & done; wait
    

    The failed count should fall. Two shards accept about 2 MB per second in total, so if your producers send more than that, some failures remain. Sizing a stream means matching the rate in the stem, not only making one run pass.

  4. Write a small JSON dataset with two days of data into the raw prefix, then create a JSON table over it in the lab35 database. Athena needs a results location, so use the same bucket.

    Athena runs queries asynchronously, so a query can still be QUEUED or RUNNING right after you start it. First define a small helper that polls until the query finishes and prints its final state:

    wait_query() {
      while true; do
        STATE=$(aws athena get-query-execution --query-execution-id "$1" --region $REGION \
          --query 'QueryExecution.Status.State' --output text)
        case "$STATE" in
          QUEUED|RUNNING) sleep 2 ;;
          *) echo "$STATE"; return ;;
        esac
      done
    }
    

    Then write the data and create the JSON table:

    python3 - <<'PY' > events.json
    import json, random
    for day in ["2026-11-14", "2026-11-15"]:
        for i in range(5000):
            print(json.dumps({"user_id": f"u{i % 500}", "page": random.choice(["home", "cart", "checkout", "search"]),
                              "event_day": day, "note": "x" * 200}))
    PY
    aws s3 cp events.json s3://$LAB_BUCKET/raw/events/events.json --region $REGION
    
    cat > json-table.sql <<'SQL'
    CREATE EXTERNAL TABLE lab35.events_json (
      user_id string,
      page string,
      event_day string,
      note string
    )
    ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
    LOCATION 's3://<your-lab-bucket>/raw/events/';
    SQL
    sed -i "s/<your-lab-bucket>/$LAB_BUCKET/" json-table.sql
    
    QID=$(aws athena start-query-execution --query-string file://json-table.sql \
      --result-configuration OutputLocation=s3://$LAB_BUCKET/athena-results/ \
      --region $REGION --query QueryExecutionId --output text)
    wait_query $QID
    

    Expected output: SUCCEEDED. Next, write a CTAS query that stores the same rows as Parquet, partitioned by event_day. CTAS is one of the Athena transform patterns: it reads the JSON table and writes a new table in another format.

    cat > parquet-ctas.sql <<'SQL'
    CREATE TABLE lab35.events_parquet
    WITH (
      format = 'PARQUET',
      parquet_compression = 'SNAPPY',
      partitioned_by = ARRAY['event_day'],
      external_location = 's3://<your-lab-bucket>/curated/events/'
    ) AS
    SELECT user_id, page, note, event_day FROM lab35.events_json;
    SQL
    sed -i "s/<your-lab-bucket>/$LAB_BUCKET/" parquet-ctas.sql
    
    QID2=$(aws athena start-query-execution --query-string file://parquet-ctas.sql \
      --result-configuration OutputLocation=s3://$LAB_BUCKET/athena-results/ \
      --region $REGION --query QueryExecutionId --output text)
    wait_query $QID2
    

    Expected output: SUCCEEDED. The partitioned_by column must be the last column in the SELECT list, which is why event_day comes last. The CTAS external_location must be empty and must not be set when the workgroup enforces its own result location, because Athena fails a CTAS that specifies one; if the CTAS fails with a location error, check the workgroup's result settings first.

  5. Run one filtered query against each table. The query reads one day and two columns. Athena reports how many bytes it scanned once the query finishes, so the loop waits with wait_query before it reads the statistics. Run it in the same terminal, where the helper is defined.

    for T in events_json events_parquet; do
      Q=$(aws athena start-query-execution --region $REGION \
        --query-string "SELECT page, COUNT(*) FROM lab35.$T WHERE event_day = '2026-11-15' GROUP BY page" \
        --query-execution-context Database=lab35 \
        --result-configuration OutputLocation=s3://$LAB_BUCKET/athena-results/ \
        --query QueryExecutionId --output text)
      wait_query $Q > /dev/null
      aws athena get-query-execution --query-execution-id $Q --region $REGION \
        --query '[QueryExecution.Statistics.DataScannedInBytes, QueryExecution.Status.State]' --output text | sed "s/^/$T: /"
    done
    

    Expected output: two lines, one per table, both SUCCEEDED. The Parquet line reports far fewer bytes scanned than the JSON line, because the JSON table reads every row of both days and every column, while the Parquet table reads one partition folder and only the page column. Small files can hide the gap, so compare the two numbers rather than their size on disk.

Verify

  • aws kinesis describe-stream-summary --stream-name lab35-stream --region $REGION --query 'StreamDescriptionSummary.OpenShardCount' returns 2.
  • The second parallel run of produce.py reports fewer failed records than the first run, which reported failures in at least one batch.
  • aws glue get-table --database-name lab35 --name events_parquet --region $REGION --query 'Table.PartitionKeys' lists event_day.
  • aws s3 ls s3://$LAB_BUCKET/curated/events/ --recursive --region $REGION shows one folder per event_day= value, each containing Parquet files.
  • The Parquet query line in step 5 shows fewer bytes scanned than the JSON query line.

Clean up

Delete every resource in dependency order, then confirm nothing is left. Also remove the local files this lab wrote: produce.py, batch-*.json, events.json, json-table.sql and parquet-ctas.sql. They hold test records and SQL only.

aws glue delete-table --database-name lab35 --name events_parquet --region $REGION
aws glue delete-table --database-name lab35 --name events_json --region $REGION
aws glue delete-database --name lab35 --region $REGION

aws kinesis delete-stream --stream-name lab35-stream --region $REGION
aws kinesis wait stream-not-exists --stream-name lab35-stream --region $REGION

aws s3 rb s3://$LAB_BUCKET --force --region $REGION

Final check: nothing from this lab remains.

aws kinesis list-streams --region $REGION --query 'StreamNames'
aws glue get-databases --region $REGION --query 'DatabaseList[?Name==`lab35`]'
aws s3 ls | grep "$LAB_BUCKET" || echo "bucket removed"

Expected output: the stream list has no lab35-stream, the database query returns an empty list, and the bucket check prints bucket removed.

Sources

Last reviewed