How to get rid of MWAA lingering errors?

This issue with lingering errors in the Airflow UI after fixing DAG files is likely related to how Apache Airflow processes DAGs in Amazon MWAA.

There are a few potential causes and solutions to consider:

1. DAG Processing Time: In Amazon MWAA, new DAG files typically take about 300 seconds (5 minutes) to be processed, while updates to existing DAGs take around 30 seconds. The lingering errors you're seeing might be due to this processing delay.

2. Resource Contention: High CPU and memory utilization in the scheduler, web server, or worker components can cause issues with DAG processing. Check your CloudWatch metrics for CPUUtilization and MemoryUtilization to see if resource constraints might be causing delayed processing.

3. Complex DAGs: Complex DAGs can contribute to processing issues. Consider optimizing your DAGs by:

• Moving imports and logic inside task functions when possible
• Avoiding heavy data processing in Apache Airflow workers
• Creating an .airflowignore file to exclude unnecessary DAGs from processing
• Avoiding calling Apache Airflow variables in top-level code

4. Python Interpreter Setting: Ensure that core.execute_tasks_new_python_interpreter is set to True in your Airflow configurations. This parameter is activated by default but confirming it might help.

5. Environment Class: If the issue persists, consider upgrading your environment class to provide more resources for DAG processing.

To troubleshoot further, you could monitor the scheduler logs in CloudWatch for any errors related to DAG processing, and consider implementing some of the DAG optimization practices mentioned above.
Sources
Resolve MWAA SIGTERM SIGKILL errors | AWS re:Post
Amazon MWAA frequently asked questions - Amazon Managed Workflows for Apache Airflow
Resolve tasks stuck in running state in Amazon MWAA environment | AWS re:Post
Troubleshoot Apache Airflow UI access issues in MWAA | AWS re:Post

Hi

• The MWAA lingering error issue might occur because Airflow's web server doesn't automatically refresh when DAG errors are resolved, it depends on the scheduler to reserialize DAGs, which may not complete properly after fixes and which is resulting to the web UI to display cached errors.[1]

• The quickest solution is to force an environment restart by making any minor configuration change in the MWAA console, which triggers a full restart and clears cached errors. You can also diagnose the issue by checking CloudWatch DAGProcessing logs to see if the scheduler is still detecting errors.

• The most impactful setting for your issue is likely dag_processor.min_file_process_interval, reducing this from 30 to 10 seconds will make the processor check for file changes more frequently, clearing errors faster. Setting core.store_dag_code to False forces fresh parsing instead of using cached serialized DAGs.

https://airflow.apache.org/docs/apache-airflow/3.0.6/administration-and-deployment/dag-serialization.html

Reference: [1] https://repost.aws/questions/QUb7U-sleJQweFauStZcFV7Q

Hello!

The quickest solution is to force an environment restart by making any minor configuration change in the MWAA console, which triggers a full restart and clears cached errors.

It doesn't help.