Adding our own worker to MWAA

0

Hi,
I was wondering if it was possible (or in the roadmap) to add our own Worker to MWAA ?

We are currently running 95% of our dags on MWAA with small workers.
but the 5% left are more compute heavy and crash MWAA worker even if we scale up.

We have some machines at the office that we could use has worker to compute those dags.
Is that something we can (/will be able to) do?

We could tell the heavy compute dags to go in a specific queue and our worker at home would listen only to those queue.

Thanks for your feedbacks in advance

LouisAW
質問済み 3年前823ビュー
2回答
0

Hi!

Worker pool support is not on the immediate roadmap for MWAA, so the recommendation is to run those few high-compute workloads on AWS Batch, ECS, or EKS--an example of the latter at https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html

Thanks!

AWS
John_J
回答済み 3年前
0

Thanks !
I'll take a look.

LouisAW
回答済み 3年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ