Compacting S3 data
We have a bucket with many small JSON files.
When we query it with Athena, it often errors out with "too many requests" (from S3).
We can compact the data, e.g. by day.
The question is what is the best way to do this? Are there any existing AWS services for this? Any existing tools that can be used?
The number of files isn't that large, less than a million, but it is evergrowing.
- Topics
- Storage
- Language
- English
asked 3 years ago1.2K views
- Newest
- Most votes
- Most comments
Hi, I would suggest you to explore AWS Glue which an ETL tool for which you can easily build data transformation pipelines.
See https://aws.amazon.com/glue/
In you case, you would schedule a daily run of to merge the json of the day before.
This blog post can give you a good starting point to process your json: https://aws.amazon.com/blogs/big-data/simplify-querying-nested-json-with-the-aws-glue-relationalize-transform/
Best,
Didier
Relevant content
asked 2 years ago
asked 4 years ago
asked 2 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 4 years ago
Hi @Didier_Durand, thank you for your answer. I am a big fan of the Glue data catalogue, but not a big fan of the ETL, as it's difficult to test and only supports two languages (Java, Python) that we do not have expertise in. Otherwise, I do agree that this could be a viable solution.