MWAA - Fargate worker spinup time

0

Hi team, I'm curious to know if anyone knows the approximate amount of time it takes for MWAA to request extra workers from Fargate and how long it takes for the fargate worker to be ready to start taking on tasks. Searching around the internet I see anywhere from 10 minutes to 1 minute. From what I've seen I'd guess its like 5 minutes?

Does anyone have documentation on this or can give me an estimation of the process from MWAA request to the handoff of a MWAA task to the worker?

Thanks

2 Answers
0

The time it takes for an Amazon ECS fargate instance to become ready depends on various factors, including the startup time of your Docker container, the size and complexity of your task. Fargate does not cache images, and therefore the whole image is pulled from the registry when a task runs. It is recommended to use larger task sizes with additional vCPUs. This is to reduce the time taken to extract the image. To further answer your question, we require details that are non-public information. Please open a support case with AWS using the following link [1].

[1] https://console.aws.amazon.com/support/home#/case/create

[+] https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/pull-behavior.html#fargate-pull-behavior

[+] What is AWS Fargate? - https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html

[+] Best Practices - Speeding up task launch - https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/task.html

[+] Get started with Amazon Managed Workflows for Apache Airflow - https://docs.aws.amazon.com/mwaa/latest/userguide/get-started.html

AWS
answered 4 months ago
0

Hi there,

I don't think it warrants a support case at the moment. I just noticed it seemed like it was not an insignificant amount of time for the instances to spin up and was curious.

I thought that perhaps the airflow fargate instances would be about the same size, but I guess requirements.txt and any other add-ons could raise the start time.

Thanks for your response.

answered 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions