Cleaning a bucket with purge_s3_path with Glue console.

Question

With the Glue Console (Glue 3.0 - python and spark), I'm need to overwrite the data of an S3 bucket in a automated daily process. I tried with the `glueContext.purge_s3_path(  "s3://bucket-to-clean-path/", { "retentionPeriod": 1, "manifestFilePath": "s3://bucket-for-manifest-path/" } )
` function, but it is not cleaning the bucket before sending the data.

The idea is to create a process to transform some data, send it to the bucket, sent it to QS throught a manifest and repeat the process daily. All the script works as it should, it is just that the bucket keeps the data from the previous runs.

Does someone knows what could be causing this problem?

**Edit**: I tried changing the retention period to one hour (several hours after the data was upload), but it still doesn't remove the files. If I remove the script non relate with emptying the bucket, the job allways takes one minute.

In the manifest appears a partition in success and another in failure, but none of them are from the files that should be removed.
Example of the partition: run-1639727067782-part-r-00000

Accepted Answer

Under options there are two parms.  One is retention period and the other is manifest file.  The retention period is 7 days by default and manifest file path might show what files are deleted successfully and which ones failed.  Hope that helps.  https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-glue-context.html

Answer

Hopefully you worked this out at the time, but I just also hit this issue. It appears you have to over-right the retention period set to 0 to delete immediately.

```
glueContext.purge_s3_path("s3://my_bucket/path_to_purge/", options={"retentionPeriod": 0}, transformation_ctx="")
```

Cleaning a bucket with purge_s3_path with Glue console.

相關內容