Overwrite in glue script

0

Hi all,

I can able to append my data to curated. But I can't able to overwrite. During overwrite curated existing file was deleted but not write a new file. My script: df. write.mode('overwrite'). Parquet (s3_file_path)

主题

分析存储数据库

标签

AWS Glue 存储提取变换和加载数据

语言

English

rePost-User-guna

已提问 1 年前792 查看次数

1 回答

最新
投票最多
评论最多

这些答案有用吗？为正确答案投票，以帮助社区从您的知识中受益。

0

You should not use the same path as source and destination, even when appending, it can result in racing issues and jobs that are not idempotent. When you select overwrite, first deletes the files on the destination.

专家

Gonzalo Herreros

已回答 1 年前

相关内容

How can I control the columns of AWS Billing CSV file?我应该如何控制AWS账单CSV文件中的字段呢？
rePost-User-1766999
已提问 1 年前
报错 : If Bucket contains '.' like part.txt in any url in rest api
专家
rePost Polyglot
已提问 5 个月前
AWS CodeDeploy：STRING_VALUE can not be converted to an Integer
专家
rePost Polyglot
已提问 5 个月前
How to use android sdk to create mqtt5 client
rePost-User-8512942
已提问 1 年前
如何解决 AWS Glue 中的“Unable to infer schema”异常？
AWS 官方已更新 3 年前
如何解决“CloudFront wasn't able to connect to the origin（CloudFront 无法连接到源）”错误？
AWS 官方已更新 2 年前
如何解决 Amazon ECS 中的“[AWS service] was unable to place a task because no container instance met all of its requirements”错误？
AWS 官方已更新 1 年前
我从 Amazon S3 桶下载或复制对象时，为什么会出现错误“The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access”（加密文字指的是不存在、此区域不存在或者您无权访问的客户主密钥）？
AWS 官方已更新 1 年前