Adding our own worker to MWAA

Question

Hi,  
I was wondering if it was possible (or in the roadmap) to add our own Worker to MWAA ?  
  
We are currently running 95% of our dags on MWAA with small workers.  
but the 5% left are more compute heavy and crash MWAA worker even if we scale up.  
  
We have some machines at the office that we could use has worker to compute those dags.  
Is that something we can (/will be able to) do?  
  
We could tell the heavy compute dags to go in a specific queue and our worker at home would listen only to those queue.  
  
Thanks for your feedbacks in advance

John_J · Answer

Hi!  
  
Worker pool support is not on the immediate roadmap for MWAA, so the recommendation is to run those few high-compute workloads on AWS Batch, ECS, or EKS--an example of the latter at https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html  
  
Thanks!

LouisAW · Answer

Thanks !  
I'll take a look.

Adding our own worker to MWAA

Relevant content