MWAA environment stuck as “Unavailable” since recent AWS outage — unable to apply configuration changes

Hi,

Yes, during the service disruption event occurred on 10/20, this behavior was commonly observed by multiple customers, in some cases, it came back in 15-20 minutes, while in others it took little longer but I don;t recall issue persisting up until now.

Having said that however, most of these reported issues were resolved by end of the event. If you are still encountering this issue, I'd highly encourage you to reach out to AWS support giving details of your MWAA environment, support will be able to assist and help in resolving the issue.

Let me know if you have any questions further, happy to help.

When an Amazon MWAA environment is stuck in an "Unavailable" state after an outage, it can prevent you from applying configuration changes. Even though your DAGs may be running, the environment management functionality is impaired.

Here are some steps you can take to address this issue:

1. Test your environment configuration locally using the aws-mwaa-local-runner or aws-mwaa-docker-images tools available on GitHub. This can help verify that your configuration is valid before attempting to apply it.

2. Run the Verify Environment script from AWS Support Tools to check your Amazon VPC network setup and configuration for your MWAA environment. This might identify networking issues that could be preventing the environment from becoming available.

3. If you're trying to update parameters like celery.worker_autoscale, consider that when an environment update fails, MWAA rolls back to the previous stable version of the environment. Your environment being stuck in "Unavailable" suggests there might be an underlying issue preventing successful updates.

4. For your specific configuration, consider these optimization suggestions:

• The celery.worker_autoscale setting controls how many tasks can run concurrently on a worker. Setting it to 20,1 might be too high for your environment class (mw1.medium), causing resource constraints.
• Ensure your worker, scheduler, and webserver logs are set to WARNING level instead of INFO to reduce excessive logging that could impact performance.
• Consider staggering task execution to distribute the workload more evenly.

5. Since you mentioned that production is technically running fine, you might want to create a new environment with your desired configuration and migrate your workloads to it, rather than continuing to try to fix the current one.

6. As you suggested, contacting AWS Support would be appropriate in this situation. They can investigate the specific state of your environment and potentially help recover it or provide guidance on creating a new environment with your desired configuration.

This appears to be an unusual situation where the environment is functional enough to run DAGs but is stuck in a state that prevents configuration updates, likely due to the previous outage affecting the control plane functionality.
Sources
Troubleshooting: Creating and updating an Amazon MWAA environment - Amazon Managed Workflows for Apache Airflow
Resolve tasks stuck in running state in Amazon MWAA environment | AWS re:Post
Resolve MWAA SIGTERM SIGKILL errors | AWS re:Post