Can a single Glue streaming job handle multiple kinesis sources

Question

In a Glue streaming job, is it possible read from Multiple Kinesis sources in my spark script?

something like:

```
streams = ["streamA", "streamB"]

for stream in streams:
    process_stream(stream)

def process_stream(stream_name):
    glueContext.forEachBatch(
            frame=dataframe,
            batch_function=process_batch_with_stream_name,
            options={
                "windowSize": "60 seconds",
                "checkpointLocation": args["TempDir"] + f"/job_{JOB_NAME}/" 
            }
```

Accepted Answer

Hello,

To read multiple kinesis sources you can create a DataFrame for each stream and use a union function before passing it to forEachBatch. If you want to process the data separately on the same job, separate threads should be coordinated which is complex to implement and hence it is not recommended.

You can also refer to the following documentation for more guidance on Streaming ETL jobs in AWS Glue: https://docs.aws.amazon.com/glue/latest/dg/add-job-streaming.html

If you need specific guidance for your use-case, please open a support case with AWS using the following link: https://console.aws.amazon.com/support/home#/case/create

Answer

Yes, you just need to create a DataFrame for each stream and union() them before passing it to forEachBatch.    
Notice that assumes your function can process data coming from either of them.   
If you mean processing them in separately on the same job, that requires calling forEachBatch on separate threads and coordinating them, it's much more complex to operate and not recommended.

Answer

Thanks!
I ended up using separate thread for each stream.
Why is it not recommended?

Can a single Glue streaming job handle multiple kinesis sources

相关内容