how to config athena spark by use dynamic partition mode when overwrite

I want to save parquet file to s3 by partition mode default is not dynamic and the old data will be erased . How can I config athena spark session setting to dynamic mode?

Topics

Analytics

Tags

Amazon Athena Session Management

Language

English

rePost-User-8379266

asked a year ago290 views

1 Answer

Newest
Most votes
Most comments

Are these answers helpful? Upvote the correct answer to help the community benefit from your knowledge.

Hello,

This could have been addressed by setting "spark.sql.sources.partitionOverwriteMode" to "dynamic". But as of today, Athena Spark does not allow to make any changes to the spark config.

Sachin_KV

answered a year ago

Relevant content

Athena Dynamic Partition Projection Issue
ebaranauskas
asked a year ago
external table in athena does not get data from partitioned parquet files
Accepted Answer
rePost-User-8741020
asked a year ago
pyspark dataframe in glue notebook mode=overwrite creates empty files at each path level
rePost-User-6463058
asked a year ago
AWS Glue Interactive Notebooks: default df.saveAsTable() behavior overwrites entire table instead of dynamically based on dataframe values
Jaime
asked a year ago
How can I understand AWS Config billing by retrieving the number of configuration items recorded per month?
AWS OFFICIALUpdated 2 years ago
How can I troubleshoot the error "FAILED: SemanticException table is not partitioned but partition spec exists" in Athena?
AWS OFFICIALUpdated a year ago
How can I create and use partitioned tables in Amazon Athena?
AWS OFFICIALUpdated 3 years ago
How can I use partition projection to create CloudTrail tables for Athena queries?
AWS OFFICIALUpdated 3 months ago
Efficiently way to use a dynamic BGP to create a VPN tunnel between AWS and Oracle Cloud Infrastructure
EXPERT
Vinay Gugueoth
published 6 months ago
AWS Glue Dynamic Frame – JDBC Performance Tuning Configuration
EXPERT
AWSRB
published a year ago