MWAA Environment & DAGS cannot find airflow.providers.amazon.aws.operators

0

Have started using MWAA 2.0.2 and generally working well - but bizarrely - when trying to use any of the aws provider operators for some 'hello aws-world' DAGS - they cannot be found/imported for use. Is this a bug? The situation seems the same in the mwaa-local-runner image as well as the actual service itself

By way of example - the following (copied from the module documentation)

from airflow.providers.amazon.aws.operators.s3 import (
    S3CopyObjectOperator,
    S3CreateBucketOperator,
    S3CreateObjectOperator,
    S3DeleteBucketOperator,
    S3DeleteBucketTaggingOperator,
    S3DeleteObjectsOperator,
    S3FileTransformOperator,
    S3GetBucketTaggingOperator,
    S3ListOperator,
    S3ListPrefixesOperator,
    S3PutBucketTaggingOperator,
)

produces the import error

Broken DAG: [/usr/local/airflow/dags/s3_dag.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/airflow/dags/s3_dag.py", line 3, in <module>
    from airflow.providers.amazon.aws.operators.s3 import (
ModuleNotFoundError: No module named 'airflow.providers.amazon.aws.operators.s3'

I should add - I have tried adding the module to the requirements.txt to ensure it is installed but it appears to already be there and I receive 'requirement already sastified' which is what I would have expected given this is an aws service.

gefragt vor 2 Jahren4686 Aufrufe
1 Antwort
1
Akzeptierte Antwort

airflow.providers.amazon.aws.operators.s3 module is available in Airflow version 2.6.0 onwards. However, most of these are available as part of multiple s3 modules in Airflow 2.0.0 as mentioned below -

from airflow.providers.amazon.aws.operators.s3_bucket import (S3CreateBucketOperator, S3DeleteBucketOperator)
from airflow.providers.amazon.aws.operators.s3_bucket_tagging import (S3GetBucketTaggingOperator, S3PutBucketTaggingOperator, S3DeleteBucketTaggingOperator)
from airflow.providers.amazon.aws.operators.s3_list import S3ListOperator
from airflow.providers.amazon.aws.operators.s3_file_transform import S3FileTransformOperator
from airflow.providers.amazon.aws.operators.s3_delete_objects import S3DeleteObjectsOperator

Here is a complete list of available aws operators in Airflow 2.0.0 - https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.0.0/_api/airflow/providers/amazon/aws/operators/index.html

AWS
beantwortet vor 2 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen