MWAA apache-airflow-providers-amazon DAGS can't import operator

2

I'm trying to use newest package for Amazon integration in MWAA.
In this particular case I want to use GlueJobOperator which is a part of the latest apache-airflow-providers-amazon package. (Link to the documentation)

MWAA Airflow version: 2.2.2

I added this to the requirements.txt:

apache-airflow-providers-amazon==4.0.0

and tried to import it and use it like in the examples:

from airflow.providers.amazon.aws.operators.glue import GlueJobOperator

glue_job = GlueJobOperator(
        task_id='airflow_id',
        job_name='job_name'
        wait_for_completion=True,
        script_location='bucket_script_prefix',
        s3_bucket='bucket_name',
        iam_role_name='iam_role'
        create_job_kwargs=job_arguments,
        script_args=script_arguments
    )

Unfortunately, whenever DAG is parsed I get this error:

...

    from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
ImportError: cannot import name 'GlueJobOperator' from 'airflow.providers.amazon.aws.operators.glue' (/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/glue.py)

It's not my first rodeo with MWAA and some extra packages, plugins etc. but I am lost. In this case, I tried many things, went through docs from cover to cover and I still couldn't find the reason. I verified in MWAA that packages were successfully installed both in the logs and in the UI with the version prompted in requirements.txt.

Package NameVersionDescription
apache-airflow-providers-amazon4.0.0Amazon integration (including Amazon Web Services (AWS)).

Fun fact: I'm using S3Hook in some other DAGs and it parses just fine.

from airflow.providers.amazon.aws.hooks.s3 import S3Hook

s3_hook = S3Hook()

...
1 Answer
0

Hello,

By default MWAA version 2.2.2 comes up with few default providers , please check "Provider packages for Apache Airflow v2.2.2 connections" in below link . In the below link you can see MWAA version 2.2.2 comes up with apache-airflow-providers-amazon==2.4.0

https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html

Since your environment has already installed provider apache-airflow-providers-amazon==2.4.0 and glue operator is not available in 2.4.0 version. Please check available operator in for version 2.4.0 in below link

https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.4.0/operators/index.html

I would request to not upgrade the providers version as lt may upgrade Airflow version which lead to unhealthy scheduler

If you still want to use apache-airflow-providers-amazon==4.0.0 in MWAA version 2.2.2 ,please follow below link , it clearly explains how to specify newer package with out causing any version conflicts.

https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html#connections-packages-newer-packages

AWS
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions