MWAA Environment & DAGS cannot find airflow.providers.amazon.aws.operators

0

Have started using MWAA 2.0.2 and generally working well - but bizarrely - when trying to use any of the aws provider operators for some 'hello aws-world' DAGS - they cannot be found/imported for use. Is this a bug? The situation seems the same in the mwaa-local-runner image as well as the actual service itself

By way of example - the following (copied from the module documentation)

from airflow.providers.amazon.aws.operators.s3 import (
    S3CopyObjectOperator,
    S3CreateBucketOperator,
    S3CreateObjectOperator,
    S3DeleteBucketOperator,
    S3DeleteBucketTaggingOperator,
    S3DeleteObjectsOperator,
    S3FileTransformOperator,
    S3GetBucketTaggingOperator,
    S3ListOperator,
    S3ListPrefixesOperator,
    S3PutBucketTaggingOperator,
)

produces the import error

Broken DAG: [/usr/local/airflow/dags/s3_dag.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/airflow/dags/s3_dag.py", line 3, in <module>
    from airflow.providers.amazon.aws.operators.s3 import (
ModuleNotFoundError: No module named 'airflow.providers.amazon.aws.operators.s3'

I should add - I have tried adding the module to the requirements.txt to ensure it is installed but it appears to already be there and I receive 'requirement already sastified' which is what I would have expected given this is an aws service.

asked 2 years ago4610 views
1 Answer
1
Accepted Answer

airflow.providers.amazon.aws.operators.s3 module is available in Airflow version 2.6.0 onwards. However, most of these are available as part of multiple s3 modules in Airflow 2.0.0 as mentioned below -

from airflow.providers.amazon.aws.operators.s3_bucket import (S3CreateBucketOperator, S3DeleteBucketOperator)
from airflow.providers.amazon.aws.operators.s3_bucket_tagging import (S3GetBucketTaggingOperator, S3PutBucketTaggingOperator, S3DeleteBucketTaggingOperator)
from airflow.providers.amazon.aws.operators.s3_list import S3ListOperator
from airflow.providers.amazon.aws.operators.s3_file_transform import S3FileTransformOperator
from airflow.providers.amazon.aws.operators.s3_delete_objects import S3DeleteObjectsOperator

Here is a complete list of available aws operators in Airflow 2.0.0 - https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.0.0/_api/airflow/providers/amazon/aws/operators/index.html

AWS
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions