AWS Glue Studio Connection w/ Secrets Manager

Question

Hi. Trying to configure an ETL job from S3 to RDS Aurora. I want to make use of secrets manager to create the connection instead of hard-coding userid/pw. The connection itself, when tested via Connections -> Test Connection is successful.

But when I try to use that same connection and execute my ETL job I get an error on the console:

```
An error occurred while calling o73.getCatalogSink. None.get

```
Digging deeper into the CloudWatch logs I see another error which points to secrets manager:

```
22/08/16 20:49:14 INFO GlueContext: Glue secret manager integration: secretId is not provided.

```

Is this a bug, or am I doing something wrong?
Thank you,

Answer

From the error message, it seems like the GLue job is unable to connect to the Secrets Manager. This could be due to one the below reasons:
* The permissions for Glue job to read SecretsManager
* If your Glue connection uses a VPC, ensure those Glue connections are listed on your Glue job details > Advanced Properties > Connections
* Ensure you are able to print the correct credentials using print() statements to debug. These statements need to be removed before implementation.

In Glue, if you could refer to the below code as a reference to use in your Pyspark or Python codebase or [this link](https://aws.amazon.com/blogs/big-data/use-aws-glue-to-run-etl-jobs-against-non-native-jdbc-data-sources/):
```

import boto3
import json

#loading secrets for retrieving db credentials
secret_arn = "arn:aws:secretsmanager:::secret:/dev/gluesecret"
region_name = ""
session = boto3.session.Session()
sm_client = session.client(
    service_name='secretsmanager',
    region_name=region_name
)
db_secret_response = sm_client.get_secret_value(
    SecretId=secret_arn
)
if 'SecretString' in db_secret_response:
    secret = db_secret_response['SecretString']
secret = json.loads(secret)
user = secret["username"]
password = secret["password"]
```

AWS Glue Studio Connection w/ Secrets Manager

Relevant content