Hello! According to the documentation, it should be possible to write data to Kinesis from Glue Streaming ETL Job using glueContext.write_dynamic_frame_from_options
. On the other hand, this feature is very poorly documented. Here you won't find any mention about Kinesis. Moreover, I can't find any example of writing to Kinesis using Glue.
After multiple tries, I managed to send some data to Kinesis using this piece of code:
kinesissink = glueContext.write_dynamic_frame.from_options(frame=apply_mapping, connection_type="kinesis",
connection_options={"streamARN": "arn:aws:kinesis:eu-central-1:ACCOUNT_ID:stream/outputstream"},
transformation_ctx="kinesissink")
The problem is that it works if and only if this stream is in eu-east-1! I tried to pass various parameters to connection_options
, including: streamName
, endpointUrl
, region
- nothing helped.
In the logs of the job I can see:
2024-03-06T23:35:08.603+01:00 24/03/06 22:35:08 INFO LogInputStreamReader: [2024-03-06 22:35:08.597475] [0x000000ba][0x00007f21499727c0] [info] [kinesis_producer.cc:87] Using Region: us-east-1
2024-03-06T23:35:08.604+01:00 24/03/06 22:35:08 INFO LogInputStreamReader: [2024-03-06 22:35:08.597485] [0x000000ba][0x00007f21499727c0] [info] [kinesis_producer.cc:48] Using default Kinesis endpoint
It's Using Region: us-east-1
no matter what. I defined my job in eu-central-1.
I'd appreciate any help.