AWS Glue gives error: "An error occurred while calling o285.save. File already exists:"

0

I'm testing a relatively simple pyspark script that I first wrote (and tested) EMR. On the EMR script works as intended, but in Glue, the script starts writing output to desired S3 location and stops midway with this error:

An error occurred while calling o285.save. File already exists:s3://bucket/prefix/part-xxxx.json

Syntax I'm using to write DF:

df \
.write.format('json') \
.option('header', 'false') \
.save('s3://...')

The prefix didn't exist on S3 before running the script. I'd appreciate any and all help on how to get this fixed.

已提問 2 年前檢視次數 165 次
沒有答案

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南