Lab · Task 4.1 · about 40 minutes
Lifecycle a Bucket and Hit the Archive Wall
Services in this lab
- Amazon S3in scope
- Amazon S3 Glacierin scope
- AWS CLIin scope
Lesson for this lab: Cost-Optimized Storage
Goal
Apply an S3 Lifecycle waterfall to a bucket, see why an archived object can't be read until it is restored, and see S3 reject a rule that breaks a minimum storage duration.
Before you start
- An AWS account you can experiment in. Use a sandbox account with a budget alert, never a production account.
- Sign in as an IAM Identity Center user or an IAM role with permission to manage S3 buckets. Don't use the root user.
- AWS CLI v2 configured for that identity (
aws sts get-caller-identityreturns your role, not root). - Pick one Region and use it throughout. The commands use
<region>and<your-bucket-name>; bucket names must be globally unique, so add something personal such as your initials and today's date.
The exam point of view: every cheaper S3 class charges you somewhere else. In this lab you see two of those charges as errors instead of on a bill: a read that fails because the object is archived, and a lifecycle rule S3 refuses because it would move data out of a class before its minimum duration.
Resources you will create (all deleted in Clean up):
- One S3 general purpose bucket.
- Two small objects in it:
logs/app-hot.log(S3 Standard) andlogs/app-archived.log(S3 Glacier Flexible Retrieval). - One lifecycle configuration on the bucket.
Steps
0 of 6 steps done
Console: open the Amazon S3 console, choose Create bucket, keep the type as General purpose, enter
<your-bucket-name>, leave Block all public access turned on, and choose Create bucket.CLI:
aws s3api create-bucket \ --bucket <your-bucket-name> \ --region <region> \ --create-bucket-configuration LocationConstraint=<region>In
us-east-1, leave out--create-bucket-configuration.Create two small text files and upload them with different storage classes. In the console, upload each file into a
logs/folder and, under Properties, set Storage class to Standard for the first and Glacier Flexible Retrieval for the second.CLI:
echo "hot log line" > app-hot.log echo "archived log line" > app-archived.log aws s3api put-object --bucket <your-bucket-name> \ --key logs/app-hot.log --body app-hot.log aws s3api put-object --bucket <your-bucket-name> \ --key logs/app-archived.log --body app-archived.log \ --storage-class GLACIERGLACIERis the API name of S3 Glacier Flexible Retrieval. You are uploading straight into it so you don't have to wait for a lifecycle rule to run.aws s3api get-object --bucket <your-bucket-name> \ --key logs/app-hot.log hot-copy.log aws s3api get-object --bucket <your-bucket-name> \ --key logs/app-archived.log archived-copy.logThe first command succeeds and prints the object metadata.
EXPECTED FAILURE: the second command fails with an error, because the archived object must be restored before it can be read. Objects in S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive are not available for real-time reads; you must restore a temporary copy first. This is the trap from the lesson: the cheapest class is wrong whenever the application needs to read the data right away.
Save this as
bad-lifecycle.json. It moves objects to S3 Glacier Instant Retrieval after 4 days and to S3 Glacier Deep Archive after 20 days, which would leave them in S3 Glacier Instant Retrieval for only 16 of its 90 minimum days.{ "Rules": [ { "ID": "too-fast", "Status": "Enabled", "Filter": { "Prefix": "logs/" }, "Transitions": [ { "Days": 4, "StorageClass": "GLACIER_IR" }, { "Days": 20, "StorageClass": "DEEP_ARCHIVE" } ] } ] }aws s3api put-bucket-lifecycle-configuration \ --bucket <your-bucket-name> \ --lifecycle-configuration file://bad-lifecycle.jsonEXPECTED FAILURE: S3 rejects the request with an error and saves no lifecycle configuration. A single rule can't transition objects out of a class before that class's minimum storage duration has passed; here the Deep Archive step would have to be at day 94 or later.
Save this as
lifecycle.json. It is the lesson's reference pattern without the versioning clause (this bucket is not versioned).{ "Rules": [ { "ID": "logs-cool-down-and-expire", "Status": "Enabled", "Filter": { "Prefix": "logs/" }, "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" }, { "Days": 90, "StorageClass": "GLACIER_IR" }, { "Days": 180, "StorageClass": "DEEP_ARCHIVE" } ], "Expiration": { "Days": 2557 } }, { "ID": "clean-up-incomplete-uploads", "Status": "Enabled", "Filter": {}, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } } ] }aws s3api put-bucket-lifecycle-configuration \ --bucket <your-bucket-name> \ --lifecycle-configuration file://lifecycle.jsonConsole equivalent: on the bucket's Management tab, choose Create lifecycle rule and add the same transitions and expiration, then create a second rule that deletes incomplete multipart uploads after 7 days.
This one is accepted because each step leaves objects in the earlier class for at least its minimum: 60 days in S3 Standard-IA and 90 days in S3 Glacier Instant Retrieval. Note that your two test objects are far smaller than 128 KB, so by default this rule would never transition them. That is deliberate S3 behavior, and the reason the lesson says to bundle small objects.
Console: select
logs/app-archived.log, choose Actions, then Initiate restore. Set the number of days to1and the retrieval tier to Expedited.CLI:
aws s3api restore-object --bucket <your-bucket-name> \ --key logs/app-archived.log \ --restore-request '{"Days":1,"GlacierJobParameters":{"Tier":"Expedited"}}'Expedited is the fastest tier. If S3 can't accept an Expedited request at a time of high demand, run the same command with
"Tier":"Standard"instead, which typically takes 3–5 hours. Check progress:aws s3api head-object --bucket <your-bucket-name> --key logs/app-archived.logWhile the restore runs, the output includes
"Restore": "ongoing-request=\"true\"". When it showsongoing-request="false"with an expiry date, run theget-objectcommand from step 3 again. It now succeeds.
Verify
-
The lifecycle configuration is the working one:
aws s3api get-bucket-lifecycle-configuration --bucket <your-bucket-name>Expected: two rules,
logs-cool-down-and-expirewith three transitions (STANDARD_IAat 30,GLACIER_IRat 90,DEEP_ARCHIVEat 180) and an expiration at 2557 days, andclean-up-incomplete-uploadswithDaysAfterInitiation7. There is no rule namedtoo-fast. -
The restored object is still archived; only a temporary copy is readable:
aws s3api head-object --bucket <your-bucket-name> --key logs/app-archived.logExpected:
"StorageClass": "GLACIER"and aRestorefield withongoing-request="false"and anexpiry-date. After that date, S3 removes the temporary copy and reads fail again until you restore once more. -
archived-copy.logexists locally and containsarchived log line.
Clean up
Delete everything in dependency order: objects first, then the bucket.
-
Delete both objects:
aws s3api delete-object --bucket <your-bucket-name> --key logs/app-hot.log aws s3api delete-object --bucket <your-bucket-name> --key logs/app-archived.log -
Remove the lifecycle configuration (optional, since deleting the bucket removes it, but it keeps the steps explicit):
aws s3api delete-bucket-lifecycle --bucket <your-bucket-name> -
Delete the bucket:
aws s3api delete-bucket --bucket <your-bucket-name> --region <region>In the console: empty the bucket, then choose Delete and type the bucket name to confirm.
-
Delete the local files
app-hot.log,app-archived.log,hot-copy.log,archived-copy.log,bad-lifecycle.jsonandlifecycle.json. -
Check that nothing remains:
aws s3api list-buckets --query "Buckets[?Name=='<your-bucket-name>']"Expected:
[]. No other resources were created in this lab, so nothing billable is left.
Sources
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects-retrieval-options.html
- https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
- https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortIncompleteMultipartUpload.html
Last reviewed