AWS Glue Visual Studio - Redshift Target Node

0

Hey all, I have been trying to perform a simple S3 to redshift data push using an S3 source node and a Amazon Redshift target node. I have been getting errors such as 'Failed to connect to IP Address', 'py4j.protocol.Py4JJavaError: An error occurred while calling o101.getSink. : com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

Have checked permissions, policies, etc with AWS Admin. Any tips to fix this? Note- I am attempting this in a glue job which also has several other nodes for transforms, etc.

4 Respostas
0
Resposta aceita

In my case, I had 2 connections on my glue job and I only required the JDBC connection to redshift. I had an additional network connection which wasn't required since everything was on the same network

sg03
respondido há 5 meses
0

Notice that Glue is trying to use the MySQL driver driver, which will use a different port and connectivity.
Doublecheck you are using the right connection and that is well defined (or point the target directly to Redshift without a connection/table)

profile pictureAWS
ESPECIALISTA
respondido há 5 meses
  • Hey Gonzalo thanks very much for your response. Currently, I have picked the appropriate connection to my database in redshift, and also picked the schema and the table to ingest into in redshift. What do you mean by point to redshift without the connection/table? Where would the data end up going then?

0

Double check your Glue connection was set up for Redshift and not generic JDBC (as was pointed out, your connection thinks it is MySQL). Also as a reminder, when using Glue with Redshift we also strongly recommend using Glue 4.0 as the newest connectors are exponentially better than the old.

For running queries before or after a data load I recommend using the redshift_connector python library on pypi via --additional-python-modules.

AWS
Zach
respondido há 5 meses
0

'Communications link failure' typically indicates that the Glue job is unable to reach the source and/or destination target.

Check the vpc setup, like the subnets and the security groups, refer the below documentation for reference

[+] Redshift connections - Set up Amazon VPC - https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-redshift-home.html#aws-glue-programming-etl-redshift-config-vpc

AWS
ENGENHEIRO DE SUPORTE
respondido há 5 meses

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas