AWS Glue using s3 for data shuffling

Question

Hi,

I am using s3 for data shuffling in my glue job.

When I ran the notebook, it failed with FileNotFoundException of some objects. However, I can see the objects on the shuffling s3 bucket. Any thoughts on what can be missing?

I used the following magic: 
%%configure
{
    "--write-shuffle-files-to-s3": "true", 
    "--conf": "spark.shuffle.glue.s3ShuffleBucket=s3://x/tmp/"
}

Thanks!
MC

Gnanesh_N · Answer

Hello,

"FileNotFoundError: No such file or directory" is a generic error which can be occurred due to multiple reasons and few of them are as below:[1]

- Specified S3 bucket/path does not exist
-It is possible the underlying files have been updated at the same time while reading from the job
- IAM role associated with the AWS Glue Job does not have the required permissions for the respective S3 bucket
- Any specified bucket policy not allowing to access the S3 bucket.
-  If the path into script/code is not defined properly

I would request you to check the above and see if there is any discrepancy in accessing those s3 objects.

Thank you!

References:
[1]https://stackoverflow.com/questions/22282760/filenotfounderror-errno-2-no-such-file-or-directory - [3rd party]

mykc · Answer

Thanks for the hints. Yes, it turns out to be missing some IAM permission. Out of curiosity, why AWS treats such a permission issue as FileNotFoundError? IMHO, it makes the troubleshooting easier if it is an Access Denied Error.

AWS Glue using s3 for data shuffling

Add your answer

Relevant content