Error Running Glue Crawler
I am trying to catalog some data from S3 as part of Lake Formation setup. I am using a crawler to read a CSV file from an S3 bucket. The crawler is set up to use the default AWSGlueServiceRoleDefault IAM role, which has full access to S3. However, the job keeps failing with the following error:
ERROR : Insufficient Lake Formation permission(s) on s3://<<bucket-name>>/<<file-name>>
(Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 6ca688e0-1b6d-11ea-9499-cb57d11186a3). For more information, see Setting up IAM Permissions in the Developer Guide (http://docs.aws.amazon.com/glue/latest/dg/getting-started-access.html)
As per the documentation, using AWSGlueServiceRoleDefault should be sufficient to execute the crawler. Any ideas why this is failing?
If you have registered the S3 path in Lake Formation, then Lake Formation will be trying to vend temporary credentials to Glue to use to crawl that location, based on the permissions defined in Lake Formation (so not using S3 IAM permissions at all). So you need to GRANT permissions on the database and table (at least select permissions) for AWSGlueServiceRoleDefault ... you do this from the Lake Formation console.
Check this guide to Migrate Permissions to Lake Formation.
403 Access denied error from S3 in GlueAccepted Answerasked 5 years ago
AWS Glue crawler detecting a .(dot) before header of a csv fileasked 3 years ago
AWS Glue crawlerasked a month ago
Glue Crawler getting 403 from S3 because "ciphertext refers to a CMK that doesn't exist." (using SSE-S3, not KMS)Accepted Answerasked 3 months ago
How could we have Glue to get data from csv as String?Accepted Answerasked 2 months ago
Error Running Glue CrawlerAccepted Answerasked 3 years ago
Can an Glue Crawler use a S3 Lambda Access Point as a data store?asked 8 days ago
AWS Crawler to directly read Delta lake files from S3asked 5 days ago
I need to read S3 data, transform and put into Data Catalog. Should I be using a Crawler?Accepted Answerasked 4 months ago
Delete partitions in Glue Data Catalog using crawler not working.asked 2 months ago