Secret credentials not passed when using Glue Custom Connector

Question

I created a custom Glue Connector for a JDBC resource I want to connect to in an ETL job.  I created a connection for this connector referencing the credential secret.  When I attempt to connect to to the data source using this connection using this code:

```
dyf = glue_context.create_dynamic_frame.from_options(
    connection_type="custom.jdbc",
    connection_options={
        "dbTable": "MY_TABLE",
        "connectionName": "connector-connection",
        "useConnectionProperties": "true",
        # "user": conf["user"],
        # "password": conf["password"],
    },
    transformation_ctx="dyf_1709299053",
)
```
I get the following error from the driver: 
`Insufficient information to log on the database. Missing parameters are: USER PASSWORD.`
Passing the credentials in the `connection_options` (uncommenting the above lines) does not work either.

How can I pass in these credentials from the connection?  Do I need to change some setting in my connector config?

Creating a connection in the following manner works, but I would prefer the first approach as it seems a bit simpler and I like not having to pull the credential parameters into the job.
```
conf = glue_context.extract_jdbc_conf('jdbc-connection')

connection_options = {
    "url": conf["fullUrl"],
    "className": "driver name",
    "user": conf["user"],
    "password": conf["password"],
    "dbTable": "MY_TABLE",
}

connection_dyf = glue_context.create_dynamic_frame.from_options(
    connection_type="custom.jdbc", connection_options=connection_options
)
```

Answer

I don't think in the first case "useConnectionProperties" is doing anything since that is for JDBC connections (not custom), the connection name should be enough and I think that is that is what is overriding the user/password when you pass it. I suspect there is something wrong with the secret you have configured (make sure the key in the secret is "username" and not "user".

Answer

I found [this post](https://dev.to/aws-builders/secrets-and-aws-glue-custom-connectors-4je3) which mentions setting `user=${username},password=${password}` placeholders as part of the URL base in the connector.  Adding this configuration fixed the issue I was facing.

This is mentioned in [this AWS doc](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html), but it was unclear to me how to use these.  For instance, I initially tried to use `${secretKey}` but I am apparently not using this as intended.

Final tip, after updating the connector with the credential placeholders, you need to update the connection by selecting `default` from the `Connection credential type` dropdown.  This refreshes the connection with the connector updates.

Secret credentials not passed when using Glue Custom Connector

相關內容