Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

query S3 objects' metadata

0

In terms of comparing and validating my transfer of data to S3 files, I need to be able to query the S3 objects' metadata in a given bucket.

Is there a way to query the S3 metadata, maybe a catalogue or something? At the moment I created a script which goes through the all objects and extracts the metadata which has been saved in a DB so I can use it. In my case, we have a few million files and it takes 5 hours for the script to finish.

I am looking for out of a box solution/AWS product or something that keeps this "S3 metadata catalogue" up to date.

Could you recommend me something?

  • Chenjie Ni
    2 years ago

    No out-of-box solution unfortunately. You need to build it yourself.

Topics
Storage
Tags
Amazon S3 Glacier
Language
English
asked 2 years ago1.3K views
2 Answers
2
Accepted Answer

Depending on what object metadata you are interested in, you could consider enabling Amazon S3 Inventory. Amazon S3 inventory provides comma-separated values (CSV) or Apache optimized row columnar (ORC) or Apache Parquet (Parquet) output files that list your objects and their corresponding metadata (such as object size, last modified date, encryption status and other fields) on a daily or weekly basis for an S3 bucket or objects that share a prefix (objects that have names that begin with the same string). Once the S3 inventory data has been generated in an S3 bucket, you can query these files (containing the metadata such as last_modified_date, e_tag etc) in Athena. For more details, please review the documentation and the blog:

[+] https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html [+] https://aws.amazon.com/blogs/storage/manage-and-analyze-your-data-at-scale-using-amazon-s3-inventory-and-amazon-athena/

AWS
SUPPORT ENGINEER
answered 2 years ago
EXPERT
reviewed 2 years ago
  • Dimo Boyadzhiev
    2 years ago

    That seems to be what I am looking for. The best will be if it was real-time but once a day also could work in my case.

    Thanks!

0

We covered your question during our re:Post Live segment on AWS On Air S06E06 on Friday January 31st. You can see our segment covering this question at 00:58:17. The recent launch of S3 Metadata tables could address your need in an easier fashion.

AWS
EXPERT
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content