By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Glue Job store in "Temporary path" files, is it safe to delete this files periodically in scheduled job?

0

In Glue Jobs "Temporary path" specified s3 buckets folder we have bunch of files, is it safe to delete this files periodically? Does this affect on "Job bookmark" or some functionality in Glue Job?

Topics
Analytics
Tags
AWS Glue
Language
English
Data engineer
asked 5 months ago395 views
1 Answer
0

The temporary directory is used mainly for Redshift and relationalize temporary files, once the job has completed it is safe to delete. Won't delete bookmarks.
If you know the maximum time a job can run, you can provide a safe expiration policy so files never expire mid run.

profile pictureAWS
EXPERT
Gonzalo Herreros
answered 5 months ago
profile picture
EXPERT
Giovanni Lauria
reviewed a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content