MWAA 2.5.1 requirements.txt does not recognize '--constraint ' tag anymore

Question

It appears that a modification was made to the MWAA setup around August 7-8, 2023. This change has caused a previously functional setup to fail.

I'm attempting to upgrade to a newer version of the Amazon provider: apache-airflow-providers-amazon==8.1.0, as opposed to the included version 7.1.0 in the deployed version.

Here's my process:

1. Duplicate the constraints, updating them to utilize version 8.1.0.
2. Include the revised constraints file in requirements.txt and request the installation of the upgraded version.

The requirements.txt is as follows:
```
--constraint "https://raw.githubusercontent.com/<***>/aws-mwaa/main/constraints-2.5.1/constraints-3.10-mod.txt"

apache-airflow-providers-amazon==8.1.0
apache-airflow-providers-slack==7.2.0
```

Recently, over the past day or two, I've encountered errors in processing the DAGS, with the `requirements_install_ip-**.us-east-2.compute.internal_***.log` having the following:
```
WARNING: Constraints should be specified for requirements.txt. Please see https://docs.aws.amazon.com/mwaa/latest/userguide/working-dags-dependencies.html#working-dags-dependencies-test-create
Forcing local constraints
ERROR: Cannot install apache-airflow-providers-amazon==8.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested apache-airflow-providers-amazon==8.1.0
    The user requested (constraint) apache-airflow-providers-amazon==7.1.0,==8.1.0

To fix this you could try to:...
```
From my observation, it now appears that I can't bypass the constraints.

I'm curious if this is a newly introduced bug (or possibly a fix that disallows constraints from loading). Modifying DAGS to accommodate the older AWS provider version seems to be an unreasonable solution at this point.

Thank you for your attention,
A

John_J · Answer

Hello...I have confirmed that`--constraint` is being detected as expected in Airflow 2.5.1 on Amazon MWAA.  Please double-check your requirements.txt and access to the constraint file from all MWAA containers (noting that private web servers do not have Internet access) and perform an environment update.  For private web servers, you may need to provide a local constraints file as described in https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html#connections-packages-newer-packages.

If not already, you should also test your requirements and constraints using https://github.com/aws/aws-mwaa-local-runner.

rePost-User-7482890 · Answer

Thank you for your reply; I appreciate it.

Please note that this was a previously working setup that stopped working (not a new setup).

Due to a timely release of 2.6.3 on MWAA, I have since upgraded my environment and am back to a working setup.

Thanks again.
A

MWAA 2.5.1 requirements.txt does not recognize '--constraint ' tag anymore

Add your answer

Relevant content