Skip to content

AWS Managed Apache Services

0

I am creating an environment in AWS MWAA and in my requirement.txt file I am specifying below: apache-airflow-providers-snowflake snowflake-connector-python The dag that I have created has a import statement from airflow.providers.snowflake.transfers.s3_to_snowflake import S3ToSnowflakeOperator but I am getting error while importing the dag as below. File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/usr/local/airflow/dags/S3-Snowflake-And-Copy-Operator-DAG.py", line 2, in <module> from airflow.providers.snowflake.transfers.s3_to_snowflake import S3ToSnowflakeOperator ModuleNotFoundError: No module named 'airflow.providers.snowflake.transfers.s3_to_snowflake'

can someone please help here

3 Answers
0

It looks like you're encountering an issue with importing the Snowflake operator in your Amazon MWAA environment. The error indicates that the module airflow.providers.snowflake.transfers.s3_to_snowflake cannot be found.

This issue is likely related to the version of Apache Airflow you're using. If you're using Apache Airflow v1, the import path is different from what you're trying to use.

For Apache Airflow v1, you should modify your imports as follows:

from airflow.contrib.hooks.snowflake_hook import SnowflakeHook
from airflow.contrib.operators.snowflake_operator import SnowflakeOperator

Additionally, update your requirements.txt file to include:

asn1crypto == 0.24.0
snowflake-connector-python == 1.7.2

The apache-airflow-providers-snowflake package is designed for Apache Airflow v2, not v1. If you're using Apache Airflow v1, this package won't work correctly.

If you're using Apache Airflow v2, then your import statement should be correct, but you might need to ensure that the requirements are properly installed and that there are no version conflicts.

I recommend testing your DAGs, custom plugins, and Python dependencies locally using the aws-mwaa-local-runner on GitHub before deploying to your MWAA environment. This can help identify and resolve dependency issues before they affect your production environment.
Sources
Troubleshooting: DAGs, Operators, Connections, and other issues in Apache Airflow v1 - Amazon Managed Workflows for Apache Airflow
Troubleshooting Amazon Managed Workflows for Apache Airflow - Amazon Managed Workflows for Apache Airflow

answered a year ago
0

I have tried with different version, but all are having same issue. These are the airflow versions I have been using in AWS MWAA 2.5.1, 2.7.2 and 2.10.3

answered a year ago
0

The most common common cause for the ModuleNotFoundError error message is the dependent Python package not being installed on the environment.

You can verify whether the apache-airflow-providers-snowflake package is installed by checking the requirement install logs in the scheduler, worker, and web server log groups.

If there are errors that occur during the requirement install process, it can prevent all packages from being installed. More details on installing additional packages on the environment can be found here.

Another common cause of this error is modules or operators being renamed in different versions of the provider package.

For Snowflake provider package, the version used with Airflow 2.5.1, apache-airflow-providers-snowflake==4.0.2 has the operator S3ToSnowflakeOperator. However, in the version used with Airflow 2.7.2 and the version used with Airflow 2.10.3 this operator is no longer available and was replaced with CopyFromExternalStageToSnowflakeOperator.

To resolve the issue, you will first need to confirm that the Snowflake provider package was successfully installed on the environment and then confirm that the code for your DAG is importing the correct operator.

For an MWAA environment using Airflow 2.5.1, you would want to use the following requirements file

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.5.1/constraints-3.10.txt"

apache-airflow-providers-snowflake==4.0.2
snowflake-connector-python==2.9.0

The requirements files for other supported Airflow versions would need to be updated to use the constraints file and package versions that are compatible.

AWS
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.