1 Answer
- Newest
- Most votes
- Most comments
1
Hi,
Since the logic work as expected but you cannot see any temporary files in your S3 bucket. I would recommend you to check that the Glue Job configuration correctly specifies the path to the temporary files for the S3 storage bucket. Ensure that the --TempDir
parameter[1] is specified at runtime or that the correct S3 bucket path is used to store the temporary files. If not specified, AWS Glue generates these temporary files in the S3 temporary directory, which is different from the output_path
specified in your script.
Note, if no customized temp path, Glue will use the default temp path (usually something like s3://aws-glue-assets-<region>-<account_id>/temporary/
)
References: [1] Job Parameter reference
Thanks Yuki
Relevant content
- asked 2 years ago
- asked 17 days ago
- asked 2 months ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 11 days ago
I have specified the s3 bucket path in Temporary path option. But still its not creating any bookmark files in this s3 bucket.
I think the below code is causing the issue with job bookmark. But this is important for me as i am defining how the folder name should be created:
outputbucketname = args['target_BucketName']
timestamp = datetime.datetime.now().strftime("%Y%m%d") filename = f"aks{timestamp}" output_path = f"{outputbucketname}/{filename}"