- 新しい順
- 投票が多い順
- コメントが多い順
Hello,
To read multiple kinesis sources you can create a DataFrame for each stream and use a union function before passing it to forEachBatch. If you want to process the data separately on the same job, separate threads should be coordinated which is complex to implement and hence it is not recommended.
You can also refer to the following documentation for more guidance on Streaming ETL jobs in AWS Glue: https://docs.aws.amazon.com/glue/latest/dg/add-job-streaming.html
If you need specific guidance for your use-case, please open a support case with AWS using the following link: https://console.aws.amazon.com/support/home#/case/create
Yes, you just need to create a DataFrame for each stream and union() them before passing it to forEachBatch.
Notice that assumes your function can process data coming from either of them.
If you mean processing them in separately on the same job, that requires calling forEachBatch on separate threads and coordinating them, it's much more complex to operate and not recommended.
関連するコンテンツ
- 質問済み 9ヶ月前
- 質問済み 6ヶ月前
- AWS公式更新しました 3年前
- AWS公式更新しました 3年前
They could have interference (e.g. fighting for driver memory) and in general much harder to monitor and operate (e.g., what happens if one of them fails, do you restart the whole job?)