Overwrite in glue script

0

Hi all,

I can able to append my data to curated. But I can't able to overwrite. During overwrite curated existing file was deleted but not write a new file. My script: df. write.mode('overwrite'). Parquet (s3_file_path)

asked 10 months ago702 views
1 Answer
0

You should not use the same path as source and destination, even when appending, it can result in racing issues and jobs that are not idempotent. When you select overwrite, first deletes the files on the destination.

profile pictureAWS
EXPERT
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions