Adding our own worker to MWAA

0

Hi,
I was wondering if it was possible (or in the roadmap) to add our own Worker to MWAA ?

We are currently running 95% of our dags on MWAA with small workers.
but the 5% left are more compute heavy and crash MWAA worker even if we scale up.

We have some machines at the office that we could use has worker to compute those dags.
Is that something we can (/will be able to) do?

We could tell the heavy compute dags to go in a specific queue and our worker at home would listen only to those queue.

Thanks for your feedbacks in advance

LouisAW
asked 2 years ago125 views
2 Answers
0

Hi!

Worker pool support is not on the immediate roadmap for MWAA, so the recommendation is to run those few high-compute workloads on AWS Batch, ECS, or EKS--an example of the latter at https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html

Thanks!

John_J
answered 2 years ago
0

Thanks !
I'll take a look.

LouisAW
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions