Broken Plugin Banner in MWAA With No Broken Plugins

Question

Specifically I'm getting the banner error:

```
Broken plugin: [/usr/local/airflow/plugins/custom_operators/databricks.py] No module named 'airflow.providers.databricks'
```
I am using the operator defined in this file in one of my DAGs and its working just fine. The module it is pointing to is available and working fine.

The version of mwaa I'm using is (installed with terraform) :
```
module "mwaa" {
  source = "cloudposse/mwaa/aws"
  version = "0.6.0"
```
https://github.com/cloudposse/terraform-aws-mwaa
with  Airflow version 2.7.2

requirements.txt:
```
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.11.txt"
flake8==7.0.0
mypy==1.2.0
types-requests==2.31.0.6
pytest==7.4.2
pytest-cov==4.1.0
pytest-custom-exit-code==0.3.0
apache-airflow-providers-databricks==4.5.0
airflow-provider-fivetran-async==2.1.0
```
Thanks for the help

re:Post Agent · Answer

The issue you're experiencing with the broken plugin banner in Amazon Managed Workflows for Apache Airflow (MWAA) is likely due to a discrepancy between the plugin installation on the web server and the worker nodes.

In MWAA environments running Apache Airflow v2.2.2 and above, plugins and Python dependencies are installed directly on the web server. However, there can sometimes be inconsistencies between what's installed on the web server and what's available on the worker nodes.

The error message you're seeing suggests that the 'airflow.providers.databricks' module is not found on the web server, even though it's working fine in your DAGs (which run on the worker nodes).

To resolve this issue, you can try the following steps:

1. Ensure that the 'apache-airflow-providers-databricks' package is listed in your requirements.txt file with the correct version. It appears you've already done this with version 4.5.0.

2. Make sure you're using the correct constraint file for your Airflow version. You're using the correct constraint file for Airflow 2.7.2, which is good.

3. Try updating your environment. This will trigger a reinstallation of packages on both the web server and worker nodes, which might resolve the discrepancy.

4. If the issue persists, you may need to create a new environment. Sometimes, creating a new environment can resolve issues that persist through updates.

5. Check the scheduler logs for any additional error messages that might provide more context about the plugin loading issue.

It's worth noting that even though you're seeing this error banner, if your DAGs are working correctly, it might just be a UI issue rather than a functional problem. However, resolving it will ensure consistency across your environment and prevent potential future issues.  
**Sources**  
[Troubleshooting: Creating and updating an Amazon MWAA environment - Amazon Managed Workflows for Apache Airflow](https://docs.aws.amazon.com/mwaa/latest/userguide/t-create-update-environment.html)  
[Resolve errors when you use the Apache Airflow CLI in MWAA | AWS re:Post](https://repost.aws/knowledge-center/mwaa-apache-airflow-cli)  
[Apache Airflow provider packages installed on Amazon MWAA environments - Amazon Managed Workflows for Apache Airflow](https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html)

Broken Plugin Banner in MWAA With No Broken Plugins

답변 추가

관련 콘텐츠