My MWAA Environment stuck in updating

1

Hi, I update my MWAA environment through a requirements.txt file , but it remains in the update state. In the scheduler log, requirements_install and scheduler_console continues to repeat.

Error in scheduler_console is ERROR: You need to upgrade the database. Please run 'airflow db upgrade'. Make sure the command is run using Airflow version 2.5.1.

Is there a timeout for updates? If not, I want to know how to stop. please help me.

  • I just received this error as well starting around 2023-01-26 01:21:02 UT. Did not update anything in MWAA prior to this. The airflow-MyAirflowEnvironment-Worker log shows this may be related to a python package dependency?

    /usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:27 DeprecationWarning: The sql_alchemy_schema option in [core] has been moved to the sql_alchemy_schema option in [database] - the old setting has been used, but please update your config.
    /usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:70 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
    ERROR: You need to upgrade the database. Please run `airflow db upgrade`. Make sure the command is run using Airflow version 2.5.1.
    

    We are running airflow v2.2.2 using MWAA. Apparently we cannot just upgrade to 2.4.3 via the AWS console (UI gives an error "unable to upgrade"). To resolve this issues, we found we had to specify exact python package versions in a requirements.txt file. To determine the acceptable package versions, run a worker task that just does pip freeze and look at the logs. Then update the requirements.txt to reference these versions, before referencing/uploading to MWAA.

    Long term, I expect we will eventually need to start up a fresh MWAA env with the latest airflow version.

sh
posta 2 anni fa3791 visualizzazioni
2 Risposte
1
Risposta accettata

Hello,

When an MWAA environment is stuck in a state that is not "available" it should time out after a few hours and result to an "Update Failed" state. You can see a similar re:Post question at [1]. For configuration or requirements updates it is always recommended to test the environment on the local runner [2] before deploying to MWAA. See our best practices guide at [3].

If you are still experiencing this issue I would recommend you open a case with AWS Support at [4] so we can assist you in determining a root cause, as we will require access to information about your environment.

Hope this helps!

[1] https://repost.aws/questions/QUmknyg682SW2kMjFPVQtcJg/amazon-mwaa-environment-stuck-at-updating-status

[2] https://github.com/aws/aws-mwaa-local-runner

[3] https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html

[4] https://support.console.aws.amazon.com/

AWS
Joel_P
con risposta 2 anni fa
profile picture
ESPERTO
verificato un anno fa
0

Was there a solution to this issue? We have the same issue. Also, the local runner doesn't run on my machine. I've opened on issue on the repo, but it's hard to test when your local runner doesn't work.

con risposta 2 anni fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande