My MWAA Environment stuck in updating

1

Hi, I update my MWAA environment through a requirements.txt file , but it remains in the update state. In the scheduler log, requirements_install and scheduler_console continues to repeat.

Error in scheduler_console is ERROR: You need to upgrade the database. Please run 'airflow db upgrade'. Make sure the command is run using Airflow version 2.5.1.

Is there a timeout for updates? If not, I want to know how to stop. please help me.

  • I just received this error as well starting around 2023-01-26 01:21:02 UT. Did not update anything in MWAA prior to this. The airflow-MyAirflowEnvironment-Worker log shows this may be related to a python package dependency?

    /usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:27 DeprecationWarning: The sql_alchemy_schema option in [core] has been moved to the sql_alchemy_schema option in [database] - the old setting has been used, but please update your config.
    /usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:70 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
    ERROR: You need to upgrade the database. Please run `airflow db upgrade`. Make sure the command is run using Airflow version 2.5.1.
    

    We are running airflow v2.2.2 using MWAA. Apparently we cannot just upgrade to 2.4.3 via the AWS console (UI gives an error "unable to upgrade"). To resolve this issues, we found we had to specify exact python package versions in a requirements.txt file. To determine the acceptable package versions, run a worker task that just does pip freeze and look at the logs. Then update the requirements.txt to reference these versions, before referencing/uploading to MWAA.

    Long term, I expect we will eventually need to start up a fresh MWAA env with the latest airflow version.

sh
asked a year ago2550 views
2 Answers
1
Accepted Answer

Hello,

When an MWAA environment is stuck in a state that is not "available" it should time out after a few hours and result to an "Update Failed" state. You can see a similar re:Post question at [1]. For configuration or requirements updates it is always recommended to test the environment on the local runner [2] before deploying to MWAA. See our best practices guide at [3].

If you are still experiencing this issue I would recommend you open a case with AWS Support at [4] so we can assist you in determining a root cause, as we will require access to information about your environment.

Hope this helps!

[1] https://repost.aws/questions/QUmknyg682SW2kMjFPVQtcJg/amazon-mwaa-environment-stuck-at-updating-status

[2] https://github.com/aws/aws-mwaa-local-runner

[3] https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html

[4] https://support.console.aws.amazon.com/

AWS
Joel_P
answered a year ago
profile picture
EXPERT
reviewed 9 months ago
0

Was there a solution to this issue? We have the same issue. Also, the local runner doesn't run on my machine. I've opened on issue on the repo, but it's hard to test when your local runner doesn't work.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions