Unable to Run DBT Glue Job in eu-west-2 in DEV Account


Dear all,

We are looking to implement Glue ETL jobs into our Data Pipeline processing.

To achieve this we are using the DBT Glue Adaptor https://github.com/aws-samples/dbt-glue

and following this AWS tutorial demo to get started - https://aws.amazon.com/blogs/big-data/build-your-data-pipeline-in-your-aws-modern-data-platform-using-aws-lake-formation-aws-glue-and-dbt-core/

We're able to successfully deploy and run this tutorial in regions - us-west-1, us-east-1, eu-west-1, eu-central-1. We are also able to deploy this in eu-west-2 in our STAGING account.

However, we are NOT able to run it in eu-west-2 (London) in our DEV Account.

Please note we use eu-west-2 as our primary region and have previously used Lake Formation for proof of concept projects. So I'm not sure if this has something to do with it.

We've tried everything including deleting all stacks and IAM Permissions/Roles/Policies related to Glue, Lake Formation in eu-west-2 and it doesn't seem to work.

We don't think its a problem with the dbt glue connector as this works correctly across other regions including eu-west-2 in our STAGING account.

Can anyone kindly advise how to debug or resolve this?

Thanks in advance,

Eric Sales De Andrade

asked a year ago146 views
1 Answer

Resolved - Turns out the glue service role didn’t have access to the default glue db in Lake Formation

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions