By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

from airflow.providers.amazon.aws.operators.s3_bucket import S3CreateBucketOperator, S3DeleteBucketOperator fails

0

Hi, from airflow.providers.amazon.aws.operators.s3_bucket import S3CreateBucketOperator, S3DeleteBucketOperator fails with

ModuleNotFoundError: No module named 'airflow.providers.amazon.aws.operators.s3_bucket'
Topics
Application Integration
Tags
Amazon Managed Workflows for Apache Airflow (MWAA)
Language
English
Andrew Bloss
asked a year ago396 views
2 Answers
0

Hi Santosh,

Thank you very much.

Where can I find out which providers version my airflow is using? I want to make sure I am looking at the right documents and continue to look at the right documents.

How can I tell that I should use apache-airflow-providers-amazon/6.0.0 and not the current stable version, https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/operators/s3/index.html?

Andrew Bloss
answered a year ago
  • Santosh_S_S SUPPORT ENGINEER
    a year ago

    MWAA environments with Airflow v2.2.2 (you can check the Airflow version in the console) comes with apache-airflow-providers-amazon package version 2.4.0. With Airflow v2.4.3, the default version installed in 6.0.0.

    You can check the installed versions by running the "pip3 list" command using BashOperator task similar to the example DAG below

    from airflow import DAG 
from airflow.operators.bash_operator import BashOperator 
from datetime import datetime, timedelta 
with DAG(dag_id='bash_dag', schedule_interval=None, start_date=datetime(2020, 1, 1), catchup=False) as dag: 
    bash_task = BashOperator(task_id='bash_task', bash_command="pip3 list") 
bash_task

    I hope the above information helps. Please feel free to write back if you any other queries or concerns, I will be more than happy to help.

    Thank you.

    I hope the above information helps. Please feel free to write back if you any other queries or concerns, I will be more than happy to help.

    Thank you.

0

Hi,

The module named 'airflow.providers.amazon.aws.operators.s3_bucket' was used until apache-airflow-providers-amazon version 2.4.0. Version 2.4.0 used to be the default amazon provider package version installed in MWAA with Airflow v2.0.2. Based on the error that you are seeing, I am suspecting that you are working on the MWAA environment with the latest version (2.4.3) where the he default amazon provider package version installed is 6.0.0.

s3_bucket module in v2.4.0 https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.4.0/_api/airflow/providers/amazon/aws/operators/s3_bucket/index.html

Same module is not present in v6.0.0 and in the immediate releases after 2.4.0, we can see the message that it is deprecated https://airflow.apache.org/docs/apache-airflow-providers-amazon/3.0.0/_api/airflow/providers/amazon/aws/operators/s3_bucket/index.html "This module is deprecated. Please use airflow.providers.amazon.aws.operators.s3."

Please use the operators of 6.0.0 as mentioned in the following document using the "airflow.providers.amazon.aws.operators.s3" instead. https://airflow.apache.org/docs/apache-airflow-providers-amazon/6.0.0/_api/airflow/providers/amazon/aws/operators/s3/index.html

If you are not using MWAA environment with the latest version 2.4.3, I would request you to check the specific operator documentations to see the correct modules as per the apache-airflow-providers-amazon version.

Thank you.

AWS
SUPPORT ENGINEER
Santosh_S_S
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content