By using AWS re:Post, you agree to the Terms of Use

MWAA apache-airflow-providers-amazon DAGS can't import operator

1

I'm trying to use newest package for Amazon integration in MWAA.
In this particular case I want to use GlueJobOperator which is a part of the latest apache-airflow-providers-amazon package. (Link to the documentation)

MWAA Airflow version: 2.2.2

I added this to the requirements.txt:

apache-airflow-providers-amazon==4.0.0

and tried to import it and use it like in the examples:

from airflow.providers.amazon.aws.operators.glue import GlueJobOperator

glue_job = GlueJobOperator(
        task_id='airflow_id',
        job_name='job_name'
        wait_for_completion=True,
        script_location='bucket_script_prefix',
        s3_bucket='bucket_name',
        iam_role_name='iam_role'
        create_job_kwargs=job_arguments,
        script_args=script_arguments
    )

Unfortunately, whenever DAG is parsed I get this error:

...

    from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
ImportError: cannot import name 'GlueJobOperator' from 'airflow.providers.amazon.aws.operators.glue' (/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/glue.py)

It's not my first rodeo with MWAA and some extra packages, plugins etc. but I am lost. In this case, I tried many things, went through docs from cover to cover and I still couldn't find the reason. I verified in MWAA that packages were successfully installed both in the logs and in the UI with the version prompted in requirements.txt.

Package NameVersionDescription
apache-airflow-providers-amazon4.0.0Amazon integration (including Amazon Web Services (AWS)).

Fun fact: I'm using S3Hook in some other DAGs and it parses just fine.

from airflow.providers.amazon.aws.hooks.s3 import S3Hook

s3_hook = S3Hook()

...
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions