- Newest
- Most votes
- Most comments
It looks like you're encountering an issue with importing the Snowflake operator in your Amazon MWAA environment. The error indicates that the module airflow.providers.snowflake.transfers.s3_to_snowflake cannot be found.
This issue is likely related to the version of Apache Airflow you're using. If you're using Apache Airflow v1, the import path is different from what you're trying to use.
For Apache Airflow v1, you should modify your imports as follows:
from airflow.contrib.hooks.snowflake_hook import SnowflakeHook from airflow.contrib.operators.snowflake_operator import SnowflakeOperator
Additionally, update your requirements.txt file to include:
asn1crypto == 0.24.0
snowflake-connector-python == 1.7.2
The apache-airflow-providers-snowflake package is designed for Apache Airflow v2, not v1. If you're using Apache Airflow v1, this package won't work correctly.
If you're using Apache Airflow v2, then your import statement should be correct, but you might need to ensure that the requirements are properly installed and that there are no version conflicts.
I recommend testing your DAGs, custom plugins, and Python dependencies locally using the aws-mwaa-local-runner on GitHub before deploying to your MWAA environment. This can help identify and resolve dependency issues before they affect your production environment.
Sources
Troubleshooting: DAGs, Operators, Connections, and other issues in Apache Airflow v1 - Amazon Managed Workflows for Apache Airflow
Troubleshooting Amazon Managed Workflows for Apache Airflow - Amazon Managed Workflows for Apache Airflow
I have tried with different version, but all are having same issue. These are the airflow versions I have been using in AWS MWAA 2.5.1, 2.7.2 and 2.10.3
The most common common cause for the ModuleNotFoundError error message is the dependent Python package not being installed on the environment.
You can verify whether the apache-airflow-providers-snowflake package is installed by checking the requirement install logs in the scheduler, worker, and web server log groups.
If there are errors that occur during the requirement install process, it can prevent all packages from being installed. More details on installing additional packages on the environment can be found here.
Another common cause of this error is modules or operators being renamed in different versions of the provider package.
For Snowflake provider package, the version used with Airflow 2.5.1, apache-airflow-providers-snowflake==4.0.2 has the operator S3ToSnowflakeOperator. However, in the version used with Airflow 2.7.2 and the version used with Airflow 2.10.3 this operator is no longer available and was replaced with CopyFromExternalStageToSnowflakeOperator.
To resolve the issue, you will first need to confirm that the Snowflake provider package was successfully installed on the environment and then confirm that the code for your DAG is importing the correct operator.
For an MWAA environment using Airflow 2.5.1, you would want to use the following requirements file
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.5.1/constraints-3.10.txt"
apache-airflow-providers-snowflake==4.0.2
snowflake-connector-python==2.9.0
The requirements files for other supported Airflow versions would need to be updated to use the constraints file and package versions that are compatible.
Relevant content
- AWS OFFICIALUpdated 3 months ago
- AWS OFFICIALUpdated 7 months ago
