By using AWS re:Post, you agree to the Terms of Use

AWS Glue gives error: "An error occurred while calling File already exists:"


I'm testing a relatively simple pyspark script that I first wrote (and tested) EMR. On the EMR script works as intended, but in Glue, the script starts writing output to desired S3 location and stops midway with this error:

An error occurred while calling File already exists:s3://bucket/prefix/part-xxxx.json

Syntax I'm using to write DF:

df \
.write.format('json') \
.option('header', 'false') \

The prefix didn't exist on S3 before running the script. I'd appreciate any and all help on how to get this fixed.

asked 9 months ago54 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions