Best way to track s3 bucket size growth over a user defined time period using athena queries or other tools

0

I see from this blog post that we can get S3 bucket size in athena using storage_mb like this:

https://aws.amazon.com/blogs/storage/query-amazon-s3-analytics-data-with-amazon-athena/

However, how might one track the bucket size growth over time, ideally over an interval defined by the customer?

asked 4 years ago871 views
1 Answer
0
Accepted Answer

If you prefer the Athena route then you can use something like this (you can easily subdivide the results however you want, in the following example I subdivide by the storage tier):

SELECT format_datetime(line_item_usage_start_date,
         'M/d/y') AS "Day", line_item_resource_id AS "Bucket",
    CASE
    WHEN line_item_usage_type LIKE '%TimedStorage-Byte%' THEN
    'Standard'
    WHEN line_item_usage_type LIKE '%TimedStorage-SIA%' THEN
    'Infrequent Access'
    WHEN line_item_usage_type LIKE '%TimedStorage-Glacier%' THEN
    'Glacier'
    WHEN line_item_usage_type LIKE '%TimedStorage-RRS%' THEN
    'Reduced Redundancy'
    ELSE 'Other'
    END AS "Tier", sum(line_item_usage_amount) *30 AS "Storage IN GB", sum(line_item_unblended_cost) AS "Unblended Cost"
FROM customer_cur_data.customer_all
WHERE line_item_product_code = 'AmazonS3'
        AND line_item_usage_type LIKE '%TimedStorage%'
        AND line_item_usage_start_date = timestamp '2020-08-31'
GROUP BY  1, 2, 3
ORDER BY  2, 1, 3

In the Where section you would need to modify line_item_usage_start_date = timestamp '2020-08-31' to be whatever range or value you want. Also in the Select modify the sum(line_item_usage_amount) *30 to multiply by the actual number of days in the month (Feb would multiply by 28). The sum of unblended cost is just that day's cost. Multiply by the same number of days in the month to project the monthly cost given that amount of storage.

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions