What is causing the excessive number of requests to s3?

Question

I searched in athena and uploaded the searched content to s3.

I used the select statement about 10 times and uploaded a small query log to s3.

However, looking at the charges charged, there were hundreds of thousands of requests.

It is unknown what caused this number of requests.

Why is there such a large number of requests?

Accepted Answer

Without proper partitioning, Athena will scan large data sets and therefore it could end up costing a lot. It is very important to partition the data correctly - https://docs.aws.amazon.com/athena/latest/ug/partitions.html

Answer

Yep Indranil is right about the partitioning.  For a tutorial and tips about this, also see https://www.linkedin.com/pulse/using-athena-partition-projection-query-cloudtrail-other-kinsman/.  It uses AWS logs as examples but the information is generic.

What is causing the excessive number of requests to s3?

相關內容