Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

how to config athena spark by use dynamic partition mode when overwrite

0

I want to save parquet file to s3 by partition mode default is not dynamic and the old data will be erased . How can I config athena spark session setting to dynamic mode?

Topics
Analytics
Tags
Amazon AthenaSession Management
Language
English
asked 3 years ago506 views
1 Answer
0

Hello,

This could have been addressed by setting "spark.sql.sources.partitionOverwriteMode" to "dynamic". But as of today, Athena Spark does not allow to make any changes to the spark config.

AWS
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content