How to write data from Glue ETL Streaming Job to Kinesis Data Stream?

0

Hello! According to the documentation, it should be possible to write data to Kinesis from Glue Streaming ETL Job using glueContext.write_dynamic_frame_from_options. On the other hand, this feature is very poorly documented. Here you won't find any mention about Kinesis. Moreover, I can't find any example of writing to Kinesis using Glue.

After multiple tries, I managed to send some data to Kinesis using this piece of code:

        kinesissink = glueContext.write_dynamic_frame.from_options(frame=apply_mapping, connection_type="kinesis",
                                                              connection_options={"streamARN": "arn:aws:kinesis:eu-central-1:ACCOUNT_ID:stream/outputstream"},
                                                              transformation_ctx="kinesissink")

The problem is that it works if and only if this stream is in eu-east-1! I tried to pass various parameters to connection_options, including: streamName, endpointUrl, region - nothing helped. In the logs of the job I can see:

2024-03-06T23:35:08.603+01:00	24/03/06 22:35:08 INFO LogInputStreamReader: [2024-03-06 22:35:08.597475] [0x000000ba][0x00007f21499727c0] [info] [kinesis_producer.cc:87] Using Region: us-east-1
2024-03-06T23:35:08.604+01:00	24/03/06 22:35:08 INFO LogInputStreamReader: [2024-03-06 22:35:08.597485] [0x000000ba][0x00007f21499727c0] [info] [kinesis_producer.cc:48] Using default Kinesis endpoint

It's Using Region: us-east-1 no matter what. I defined my job in eu-central-1.

I'd appreciate any help.

2 Antworten
0

I think is a defect and the code is getting the property from the original config and not the case insensitive one.
Please try with "endpointurl" and confirm if it changes the region and I will open a ticket to correct the defect.

profile pictureAWS
EXPERTE
beantwortet vor 2 Monaten
0

Hello! I tried this and with no effect - it didn't work. Still region is us-east-1...

beantwortet vor 2 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen