query S3 objects' metadata

Question

In terms of comparing and validating my transfer of data to S3 files, I need to be able to query the S3 objects' metadata in a given bucket.

Is there a way to query the S3 metadata, maybe a catalogue or something? 
At the moment I created a script which goes through the all objects and extracts the metadata which has been saved in a DB so I can use it. 
In my case, we have a few million files and it takes 5 hours for the script to finish.

I am looking for out of a box solution/AWS product or something that keeps this "S3 metadata catalogue" up to date.

Could you recommend me something?

Accepted Answer

Depending on what object metadata you are interested in, you could consider enabling Amazon S3 Inventory. Amazon S3 inventory provides comma-separated values (CSV) or Apache optimized row columnar (ORC) or Apache Parquet (Parquet) output files that list your objects and their corresponding metadata (such as object size, last modified date, encryption status and other fields) on a daily or weekly basis for an S3 bucket or objects that share a prefix (objects that have names that begin with the same string). Once the S3 inventory data has been generated in an S3 bucket, you can query these files (containing the metadata such as last_modified_date, e_tag etc) in Athena. For more details, please review the documentation and the blog:

[+] https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html [+] https://aws.amazon.com/blogs/storage/manage-and-analyze-your-data-at-scale-using-amazon-s3-inventory-and-amazon-athena/

[+] https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html
[+] https://aws.amazon.com/blogs/storage/manage-and-analyze-your-data-at-scale-using-amazon-s3-inventory-and-amazon-athena/

query S3 objects' metadata

Relevant content