MWAA celery.worker_autoscale setup

Question

Currently I am running my MWAA on small instance with:

`celery.worker_autoscale = 5,5`

And sometimes my tasks get SIGKILL, I assume it's because tasks are taking too many memory.

In this article : [https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-tuning.html#best-practices-tuning-tasks-params]()

I see recommendation: `mw1.small - 5,0`

How does this setup works in real life scenario? My understanding is that worker is limited to run 5 tasks, once there is queued tasks autoscaling adds additional worker, which also can run 5 tasks.

But then what's the difference between `5,0` and `5,5` ?

Accepted Answer

Hello,
The `celery.worker_autoscale` property takes value in the order `max_concurrency,min_concurrency`, and determines how many concurrent tasks can run within a worker. As such, a value of `5,5` (default for MWAA small class) means a worker will always keep 5 processes at both minimum and maximum. Whereas, a value of `5,0` means a worker will start with 0 process, but can grow up to 5 process if necessary. In other words, for more resource intensive tasks, having `5,0` can potentially allow more resource to a task when there aren't many concurrent tasks. These values need to be tuned based on the chosen worker size (small, medium or large) and nature of the task.

Ref: https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/configurations-ref.html#worker-autoscale

I hope this helps.

MWAA celery.worker_autoscale setup

Relevanter Inhalt