Amazon MWAA environment stuck at Updating status

Hi,

MWAA updates rebuild the ECS Fargate containers via CloudFormation based upon the current environment settings. If there is any sort of failure--network, execution role permissions, KMS, requirements install, or configuration overrides--the Fargate containers will fail to stabilize and after the Fargate creation timeout (2 hours or so) the stack will roll back and the environment status will be "Update Failed" or "Rollback Failed", the former still being usable.

For configuration overrides or requirements, the recommendation is to test with https://github.com/aws/aws-mwaa-local-runner. For requirements, please use --constraint per https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html

Before performing an update, it is good to ensure nothing has changed with networking, IAM, or KMS to avoid those creating an issue.

Thanks

Hi!

I've asked the support team to investigate. Please open a ticket if this has not yet been resolved.

Thanks!

Did you manage to resolve that? Having the same issue and only basic support, which does not allow me to create a support ticket... Which is a bit annoying, because I do have no way to fix this myself.

Hi!

This may be related to your issue with the Secrets Manager backend. Environments do not show available until the Web Server stabilizes, and if it never does then the MWAA environment rolls back to the last known good configuration. All the while the system will show "updating". It should eventually stabilize to "Available" if it is the above circumstance.

If this is not resolved and you would like the environment removed on your behalf please send me a private message with the account number and environment name and we'll have it deleted for you.

Thanks!

Hi
I'm having the same issue with an MWAA environment remaining in the creating state for many hours.
It appears stuck in this state and unable to resolve itself - therefore I'm unable to delete it.
Also unable to raise a support ticket as only have basic plan and there doesnt seem to be any way to resolve it myself.
Would it be possible to have it removed if i send you the details?

I am having the same issue. Environment stuck at Creating status for two hours, doesn't let me delete it. Can you please delete it for me I really don't want to get charged for this. Like the poster above me I don't have tech support since I was experimenting with this on my personal and was not on the corporate account

Edited by: awsmlguy on Mar 13, 2021 3:34 AM

For anyone else who might come across this, it seems to have been stuck at 'Creating' since the private subnets in my VPC did not have routes out to the Internet. As soon as I created the route the status changed to 'Create Failed', which then allowed me to delete the Environment

It would be good if there is a check before the Create processed starts where availability of connectivity out to the Internet from the subnet is verified.

Edited by: awsmlguy on Mar 13, 2021 4:08 AM

Edited by: awsmlguy on Mar 13, 2021 4:26 AM

Hi! If you encounter any networking issues like this again, you may want to check out our new troubleshooting script that automatically checks the Amazon VPC network setup and configuration for an Amazon MWAA environment. You can access the script via AWS Support Tools on GitHub: https://github.com/awslabs/aws-support-tools/tree/master/MWAA.

Thanks!

Hi!
I am running into the same issue as above. I have an environment stuck on "Updating" for a couple of hours now. I did add the AWS secrets manager backend before I started having this issue, so perhaps it is related?
However, I ran the diagnostic tool and didn't encounter any networking issues or failed checks.

Is it possible to have this environment removed on my behalf?

Hi all,

I have the same problem. I just enabled extra logging and my instance stuck updating. I can do nothing. Also, I see "The scheduler does not appear to be running." and have no idea what is going on. Guess why? Because my instance is still "Updating". That's super annoying! How can I use it in production if it is not working?

Any advice on how to fix it?

Same here. I used terraform to update some log level. I reckon it's just some simple updates without recreating Airflow instance. However, it was stuck for around 90 minutes and then terraform was timeout. I also checked the status in the AWS console and it still shows updating (3 hours passed). And I am unable to manually delete it neither since it's still in the updating status. This is really frustrating since it blocks my CI right now. I am gonna go to bed and let's see what will happen tomorrow morning. It really wastes my time and I have to work overtime tomorrow again.

So I also have gotten the same error here. I believe I may have tried adding some additional logs and changing the instance size to be larger and its been stuck on updating for several hours now.

I also ran the utility and this is these are the only failures I get. I also have an error on creation into the same vpc and subnets that is also stuck.

Action: logs:GetLogGroupFields is not allowed on resource arn:aws:logs:us-west-2:************:log-group:airflow-airflow--test-
failed with implicitDeny
airflow:PublishMetrics is not allowed on resource
logs:GetLogGroupFields is not allowed on resource