Glue visual UI config for job fails to enumerate Redshift database when using Redshift managed VPC endpoints

Question

Hi
I have setup Redshift in its own VPC and also setup Redshift managed VPC endpoints in another VPC in order to allow Glue connectors to access it. I have confirmed that Redshift can be accessed fine from the Glue connector (JDBC connector since the Redshift connector cannot be used like this) and it passes the connector test. However when trying to use the visual ETL job editor to config a Redshift target, the visual editor fails to enumerate the Redshift database using that connector - it always remains blank without any error.
I believe this to be a bug in the visual editor, because if I change the job to a script job, the job runs perfectly fine. I was stumped on this for days before suspecting this bug and using the script editor instead.
It seems the visual editor is only happy with Redshift being a simpler setup where it resides in the same VPC as the Glue job running, and using the Redshift connector, not the JDBC one.

Has anyone else experienced this ?

re:Post Agent · Answer

The issue you're experiencing with the AWS Glue visual ETL job editor failing to enumerate the Redshift database when using Redshift managed VPC endpoints is not uncommon. While it's possible this could be a bug in the visual editor as you suspect, there are a few potential reasons and workarounds to consider:

1. VPC Configuration: Ensure that your VPC configuration, including route tables and security groups, allows proper communication between AWS Glue and Redshift. Even with managed VPC endpoints, there might be network-related issues preventing the visual editor from accessing the Redshift database.

2. IAM Permissions: Verify that the IAM role used by Glue has the necessary permissions to describe Redshift clusters and list workgroups and namespaces, especially if you're using Redshift Serverless.

3. Connection Type: The visual editor might have limitations when working with JDBC connections to Redshift, as opposed to the native Redshift connector. This could explain why the script editor works while the visual editor doesn't.

4. VPC Endpoints: Double-check that you have all the required VPC endpoints set up, including those for AWS Glue, Amazon S3, AWS Secrets Manager, and AWS Security Token Service (STS).

5. Redshift Accessibility: If using a provisioned Redshift cluster, ensure it's configured to be publicly accessible or that your VPC configuration allows AWS Glue to reach the cluster.

As a workaround, since you've confirmed that the script editor works correctly, you might need to continue using that approach for now. You could also try creating the connection using AWS Glue Studio, which sometimes provides a more guided approach and might help identify any missing configurations.

If the issue persists, it would be worthwhile to open a support case with AWS to report this potential bug in the visual editor. In the meantime, using the script editor or AWS Glue Studio might be your best options for working with Redshift in this configuration.  
**Sources**  
[The new version 2 redshift connection for Glue fails | AWS re:Post](https://repost.aws/questions/QUfAEVnkwyRo2QVJi86FLU-Q/the-new-version-2-redshift-connection-for-glue-fails)  
[Glue ETL job to load data from S3 to Redshift failed | AWS re:Post](https://repost.aws/questions/QUUoGg8UCER2aBs9bi6t2z8g/glue-etl-job-to-load-data-from-s3-to-redshift-failed)

Glue visual UI config for job fails to enumerate Redshift database when using Redshift managed VPC endpoints

Add your answer

Relevant content