By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Glue Table and Athena: What if some files are compressed and others are not in S3, will it break anything?

0

I have a glue table specified that refers to a location in S3. I noticed that when Firehose puts data there in parquet format it might be not always compressed.

Will it cause issues for Glue table, crawler or Athena to access the data (both compressed and not compressed)?

Topics
Analytics
Tags
Amazon AthenaAmazon Data FirehoseAWS Glue
Language
English
hRed
asked 6 months ago283 views
1 Answer
0
Accepted Answer

parquet files are self descriptive, it's even possible to use different compression inside a file for different columns/pages. Therefore it should work, but since it's not the normal case, it's possible that some tool has a bug and assume a single code is used.

profile pictureAWS
EXPERT
Gonzalo Herreros
answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content