MWAA gone missing

Question

All of our MWAA dags have gone missing.  They were there a few hours ago and now there are no dags available.  We are using Dagfactory for a few months now and it has been working fine.  The web dashboard is showing 0 Dags, the logs are still populating.  Anyone see this before?

[update] in the airflow scheduler log:

Process ForkProcess-2: (exception)
Followed by exception and "No such file or directory" error.

I checked the S3 bucket and file is available.  Is it possible k8 crashed?

The scheduler recovered but without the Dagfactory yml or config file.

Any thoughts on how to reset the MWAA to restart?

Answer

Hi,

I Understand all of your DAGs are vanished suddenly in the UI and from the logs you could see "No such file or directory"  exception though the files are exist

In order to validate the missing DAG you can verify from "DagBagSize" metric which shows any DAG is missing also you can check for any dip down in the metric.

=> Also you can check "ImportErrors" metric which shows if dags that are found but not able to be processed properly, for example missing/incompatible dependencies.

Ref : https://docs.aws.amazon.com/mwaa/latest/userguide/monitoring-dashboard.html
Ref : https://docs.aws.amazon.com/mwaa/latest/userguide/access-metrics-cw-202.html

=> Please check if any CPU/Memory Utilization on WebServer, Scheduler, Worker which spikes more utilization.

=> Also try to import a below sample DAG into your S3 and see if this new DAGs are getting reflected and able to run on UI.

```
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.utils.dates import days_ago
with DAG(dag_id="bash_command_dag2345", schedule_interval=None, catchup=False, start_date=days_ago(1)) as dag:
    cli_command = BashOperator(
        task_id="bash_command",
        bash_command="ls /usr/local/airflow/dags"
    )
```

=> From the above exception "No such file or directory" if any files are deleted under S3 , Since you have confirmed files are present please ensure that if any of your DAG files are deleted or File Versioning has deleted due to life cycle policy, you can restore the files using below doc

Ref : https://docs.aws.amazon.com/AmazonS3/latest/userguide/RestoringPreviousVersions.html

=> If any KMS key used in Environment, Please ensure your MWAA execution role has the key id as part of KMS encryption context since KMS key aliases which doesn't support in MWAA.

Thanks!

MWAA gone missing

Relevant content