Can a single Glue streaming job handle multiple kinesis sources

0

In a Glue streaming job, is it possible read from Multiple Kinesis sources in my spark script?

something like:

streams = ["streamA", "streamB"]

for stream in streams:
    process_stream(stream)

def process_stream(stream_name):
    glueContext.forEachBatch(
            frame=dataframe,
            batch_function=process_batch_with_stream_name,
            options={
                "windowSize": "60 seconds",
                "checkpointLocation": args["TempDir"] + f"/job_{JOB_NAME}/" 
            }
YK
asked 2 months ago508 views
3 Answers
0
Accepted Answer

Hello,

To read multiple kinesis sources you can create a DataFrame for each stream and use a union function before passing it to forEachBatch. If you want to process the data separately on the same job, separate threads should be coordinated which is complex to implement and hence it is not recommended.

You can also refer to the following documentation for more guidance on Streaming ETL jobs in AWS Glue: https://docs.aws.amazon.com/glue/latest/dg/add-job-streaming.html

If you need specific guidance for your use-case, please open a support case with AWS using the following link: https://console.aws.amazon.com/support/home#/case/create

AWS
SUPPORT ENGINEER
answered 2 months ago
0

Thanks! I ended up using separate thread for each stream. Why is it not recommended?

YK
answered 2 months ago
  • They could have interference (e.g. fighting for driver memory) and in general much harder to monitor and operate (e.g., what happens if one of them fails, do you restart the whole job?)

0

Yes, you just need to create a DataFrame for each stream and union() them before passing it to forEachBatch.
Notice that assumes your function can process data coming from either of them.
If you mean processing them in separately on the same job, that requires calling forEachBatch on separate threads and coordinating them, it's much more complex to operate and not recommended.

profile pictureAWS
EXPERT
answered 2 months ago
profile picture
EXPERT
reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions