MWAA apache-airflow-providers-amazon DAGS can't import operator

2

I'm trying to use newest package for Amazon integration in MWAA.
In this particular case I want to use GlueJobOperator which is a part of the latest apache-airflow-providers-amazon package. (Link to the documentation)

MWAA Airflow version: 2.2.2

I added this to the requirements.txt:

apache-airflow-providers-amazon==4.0.0

and tried to import it and use it like in the examples:

from airflow.providers.amazon.aws.operators.glue import GlueJobOperator

glue_job = GlueJobOperator(
        task_id='airflow_id',
        job_name='job_name'
        wait_for_completion=True,
        script_location='bucket_script_prefix',
        s3_bucket='bucket_name',
        iam_role_name='iam_role'
        create_job_kwargs=job_arguments,
        script_args=script_arguments
    )

Unfortunately, whenever DAG is parsed I get this error:

...

    from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
ImportError: cannot import name 'GlueJobOperator' from 'airflow.providers.amazon.aws.operators.glue' (/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/glue.py)

It's not my first rodeo with MWAA and some extra packages, plugins etc. but I am lost. In this case, I tried many things, went through docs from cover to cover and I still couldn't find the reason. I verified in MWAA that packages were successfully installed both in the logs and in the UI with the version prompted in requirements.txt.

Package NameVersionDescription
apache-airflow-providers-amazon4.0.0Amazon integration (including Amazon Web Services (AWS)).

Fun fact: I'm using S3Hook in some other DAGs and it parses just fine.

from airflow.providers.amazon.aws.hooks.s3 import S3Hook

s3_hook = S3Hook()

...
1 Antwort
0

Hello,

By default MWAA version 2.2.2 comes up with few default providers , please check "Provider packages for Apache Airflow v2.2.2 connections" in below link . In the below link you can see MWAA version 2.2.2 comes up with apache-airflow-providers-amazon==2.4.0

https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html

Since your environment has already installed provider apache-airflow-providers-amazon==2.4.0 and glue operator is not available in 2.4.0 version. Please check available operator in for version 2.4.0 in below link

https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.4.0/operators/index.html

I would request to not upgrade the providers version as lt may upgrade Airflow version which lead to unhealthy scheduler

If you still want to use apache-airflow-providers-amazon==4.0.0 in MWAA version 2.2.2 ,please follow below link , it clearly explains how to specify newer package with out causing any version conflicts.

https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html#connections-packages-newer-packages

AWS
beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen