MPAA - "Broken plugin" messages on DAGs page

Question

Starting sometime in the last couple of weeks, we are getting multiple "Broken plugin" messages on the DAGs page of our MPAA environment. We use custom plugins to add both a DAG factory function and custom operators. Those plugins have dependencies on other libraries that are listed in our requirements.txt. According to the MPAA docs, neither the plugins nor the requirements.txt should be loaded on the Airflow web server, but it looks like the plugins are, and just the requirements.txt is being skipped.

From web server logs:

2021-12-20T09:17:13.203-08:00	File "/usr/local/airflow/plugins/operators/bi_data_extract_operator.py", line 17, in <module>
2021-12-20T09:17:13.224-08:00	import awswrangler as wr
2021-12-20T09:17:13.243-08:00	ModuleNotFoundError: No module named 'awswrangler'

From web server logs:

```
2021-12-20T09:17:13.203-08:00	File "/usr/local/airflow/plugins/operators/bi_data_extract_operator.py", line 17, in 
2021-12-20T09:17:13.224-08:00	import awswrangler as wr
2021-12-20T09:17:13.243-08:00	ModuleNotFoundError: No module named 'awswrangler'
```

Answer

As of Airflow 2.2.2 on MWAA, both plugins and requirements are installed on the web server. However, if a private web server is used then pypi.org is not accessible by your requirements.txt and as such you'll need to package your libraries as WHL files per https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-python-wheels .

Further, as of Airflow 2.0 you do not need to package custom operators as plugins, and instead they may simply be placed in your S3 dags folder. See https://airflow.apache.org/docs/apache-airflow/stable/plugins.html .

Further, as of Airflow 2.0 you do not need to package custom operators as plugins, and instead they may simply be placed in your S3 dags folder.  See https://airflow.apache.org/docs/apache-airflow/stable/plugins.html .

Answer

This appears to have resolved itself.

MPAA - "Broken plugin" messages on DAGs page

Relevanter Inhalt