My MWAA Environment stuck in updating

1

Hi, I update my MWAA environment through a requirements.txt file , but it remains in the update state. In the scheduler log, requirements_install and scheduler_console continues to repeat.

Error in scheduler_console is ERROR: You need to upgrade the database. Please run 'airflow db upgrade'. Make sure the command is run using Airflow version 2.5.1.

Is there a timeout for updates? If not, I want to know how to stop. please help me.

  • I just received this error as well starting around 2023-01-26 01:21:02 UT. Did not update anything in MWAA prior to this. The airflow-MyAirflowEnvironment-Worker log shows this may be related to a python package dependency?

    /usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:27 DeprecationWarning: The sql_alchemy_schema option in [core] has been moved to the sql_alchemy_schema option in [database] - the old setting has been used, but please update your config.
    /usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:70 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
    ERROR: You need to upgrade the database. Please run `airflow db upgrade`. Make sure the command is run using Airflow version 2.5.1.
    

    We are running airflow v2.2.2 using MWAA. Apparently we cannot just upgrade to 2.4.3 via the AWS console (UI gives an error "unable to upgrade"). To resolve this issues, we found we had to specify exact python package versions in a requirements.txt file. To determine the acceptable package versions, run a worker task that just does pip freeze and look at the logs. Then update the requirements.txt to reference these versions, before referencing/uploading to MWAA.

    Long term, I expect we will eventually need to start up a fresh MWAA env with the latest airflow version.

sh
已提问 2 年前3839 查看次数
2 回答
1
已接受的回答

Hello,

When an MWAA environment is stuck in a state that is not "available" it should time out after a few hours and result to an "Update Failed" state. You can see a similar re:Post question at [1]. For configuration or requirements updates it is always recommended to test the environment on the local runner [2] before deploying to MWAA. See our best practices guide at [3].

If you are still experiencing this issue I would recommend you open a case with AWS Support at [4] so we can assist you in determining a root cause, as we will require access to information about your environment.

Hope this helps!

[1] https://repost.aws/questions/QUmknyg682SW2kMjFPVQtcJg/amazon-mwaa-environment-stuck-at-updating-status

[2] https://github.com/aws/aws-mwaa-local-runner

[3] https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html

[4] https://support.console.aws.amazon.com/

AWS
Joel_P
已回答 2 年前
profile picture
专家
已审核 1 年前
0

Was there a solution to this issue? We have the same issue. Also, the local runner doesn't run on my machine. I've opened on issue on the repo, but it's hard to test when your local runner doesn't work.

已回答 2 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则