How to write data from Glue ETL Streaming Job to Kinesis Data Stream?

0

Hello! According to the documentation, it should be possible to write data to Kinesis from Glue Streaming ETL Job using glueContext.write_dynamic_frame_from_options. On the other hand, this feature is very poorly documented. Here you won't find any mention about Kinesis. Moreover, I can't find any example of writing to Kinesis using Glue.

After multiple tries, I managed to send some data to Kinesis using this piece of code:

        kinesissink = glueContext.write_dynamic_frame.from_options(frame=apply_mapping, connection_type="kinesis",
                                                              connection_options={"streamARN": "arn:aws:kinesis:eu-central-1:ACCOUNT_ID:stream/outputstream"},
                                                              transformation_ctx="kinesissink")

The problem is that it works if and only if this stream is in eu-east-1! I tried to pass various parameters to connection_options, including: streamName, endpointUrl, region - nothing helped. In the logs of the job I can see:

2024-03-06T23:35:08.603+01:00	24/03/06 22:35:08 INFO LogInputStreamReader: [2024-03-06 22:35:08.597475] [0x000000ba][0x00007f21499727c0] [info] [kinesis_producer.cc:87] Using Region: us-east-1
2024-03-06T23:35:08.604+01:00	24/03/06 22:35:08 INFO LogInputStreamReader: [2024-03-06 22:35:08.597485] [0x000000ba][0x00007f21499727c0] [info] [kinesis_producer.cc:48] Using default Kinesis endpoint

It's Using Region: us-east-1 no matter what. I defined my job in eu-central-1.

I'd appreciate any help.

2 Respuestas
0

I think is a defect and the code is getting the property from the original config and not the case insensitive one.
Please try with "endpointurl" and confirm if it changes the region and I will open a ticket to correct the defect.

profile pictureAWS
EXPERTO
respondido hace 2 meses
0

Hello! I tried this and with no effect - it didn't work. Still region is us-east-1...

respondido hace 2 meses

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas