Glue Crawler not working properly

Hey, this is a common one when setting up a Glue crawler for Redshift. Even if you’ve attached all the right policies, that "Crawler cannot be started" error usually means either a missing secretsmanager:ListSecrets permission or a misconfiguration in the iam:PassRole setup.

Let’s walk through what to check and fix:

1. Make sure your role can actually access the secret Your inline policy covers specific secret actions, but for Glue crawlers, you also need this permission to list secrets globally: { "Effect": "Allow", "Action": "secretsmanager:ListSecrets", "Resource": "*" } Without ListSecrets, the crawler fails during setup because it can't look up the secret ID from Glue’s metadata reference.

2. Double-check the iam:PassRole permission Make sure: The IAM role running the crawler has iam:PassRole for the Glue role (not just itself) The Resource should match the full ARN of the role passed to the crawler:

"iam:PassRole", "Resource": "arn:aws:iam::<account>:role/AWSGlueServiceRole-RedshiftCrawlerAccess" And confirm that:

This exact role ARN is used in the crawler setup The role trusts the Glue service (glue.amazonaws.com)

3. Check the trust relationship In the Glue role (AWSGlueServiceRole-RedshiftCrawlerAccess), verify the trust policy includes:

{ "Effect": "Allow", "Principal": { "Service": "glue.amazonaws.com" }, "Action": "sts:AssumeRole" } If this is missing, the crawler won’t be able to assume the role even if it’s attached.

Try running the crawler again after updating the above. If it still fails, check CloudTrail for failed PassRole or GetSecretValue actions, it’ll tell you which exact permission is missing.

It appears that your Glue Crawler is encountering permission issues despite the policies you've attached to the IAM role. Here are a few things to check and consider:

1. Ensure that the IAM role you've created is actually associated with the Glue Crawler. Sometimes, the error can occur if the correct role isn't selected for the crawler.

2. The inline policy you've provided looks good for Secrets Manager access, but make sure the ARNs are correct for your specific resources.

3. For Redshift access, instead of using the broad AmazonRedshiftFullAccess policy, consider creating a more specific policy that grants only the necessary permissions for your use case. This should include permissions to describe clusters and get cluster credentials.

4. Ensure that your Redshift cluster's security group allows incoming traffic from AWS Glue.

5. Check if your Redshift cluster requires SSL connections. If so, make sure your Glue connection is configured to use SSL.

6. Verify that the secret in Secrets Manager contains the correct credentials for your Redshift cluster.

7. Make sure the AWSGlueServiceRole policy is the AWS managed policy and not a custom one.

8. Consider adding explicit permissions for Glue operations in your role, such as "glue:GetConnection" and "glue:GetSecurityConfiguration".

9. Check CloudTrail logs for more detailed information about which specific permissions might be missing.

If the issue persists after verifying these points, you may need to temporarily enable more verbose logging for the Glue service to pinpoint the exact permission or configuration causing the problem.
Sources
AWS managed policy: AmazonDataZoneRedshiftGlueProvisioningPolicy - Amazon DataZone
Unload data from an Amazon Redshift cluster across accounts to Amazon S3 - AWS Prescriptive Guidance