Parquet vs ORC on EMR

0

What are the general pros and cons for Parquet vs. ORC specifically as it relates to EMR (EMRFS)

If the customer is planning to also leverage Redshift and Athena on the same data lake does this change the equation?

AWS
질문됨 7년 전568회 조회
1개 답변
0
수락된 답변

For EMR:

Parquet and ORC overlap quite a bit in terms of use cases as both are columnar formats. The last time (a few years ago) I was involved in a design evaluation to choose between the two, ORC's native indexing ended up being a measurable advantage in terms of performance in our use case, namely Hive queries that filtered results based on a handful of columns with (relatively) low cardinality (at least when compared with the number of rows in the data set). If that fits the customer's use case, that may be a good reason to go the ORC route. The caveat here is that there are third party solutions available in the ecosystem that can help close that index feature gap if the customer is willing to install and manage them.

For Athena/Redshift:

  • As a straightforward consideration of compatibility, Athena supports both formats. Assuming the same compression library is used with both formats (the two have different defaults), I am not yet aware of a significant performance delta between the two if all other things are equal.
  • Assuming your question regarding Redshift is in the context of leaving the data in S3 and leveraging Spectrum, based on the docs Parquet is currently supported but not ORC.

http://docs.aws.amazon.com/redshift/latest/dg/c-spectrum-data-files.html

중재자
답변함 7년 전
AWS
지원 엔지니어
검토됨 한 달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠