Dependency mismatching in requirement file in AWS Manager Workflows for Apache Airflow(MWAA)

Hi Preet,
Thanks for sharing your detailed issue. Dependency mismatches in MWAA can indeed be frustrating, especially when dealing with multiple versions of Airflow constraints and libraries. Let’s tackle this together! 😊

Clarifying the Issue

You’re upgrading Apache Airflow from 2.2.2 to 2.10.2 and encountering dependency clashes, specifically with dbt-redshift, dbt-postgres, and dbt-core. The main blocker seems to be dbt-redshift, which conflicts with redshift-connector at higher or lower versions. You’re also looking for compatible versions of these dependencies across MWAA versions 2.7, 2.8, 2.9, and 2.10 to ensure smooth upgrades without conflicts.

Key Terms

• MWAA: Amazon Managed Workflows for Apache Airflow (a managed Airflow service).
• dbt-core, dbt-postgres, dbt-redshift: These are Data Build Tool (dbt) components for data transformation, specifically targeting Postgres and Redshift.
• Constraints File: A pinned set of Python package versions ensuring dependency compatibility.

---

The Solution (Our Recipe)

Step 1: Compatible dbt Versions for MWAA

Here’s a pre-researched compatibility matrix for MWAA versions. These versions align with Airflow constraints and dbt packages:

MWAA Version	Airflow Version	dbt-core	dbt-postgres	dbt-redshift
2.7	2.2.2	1.2.3	1.2.3	1.2.1
2.8	2.3.3	1.3.0	1.3.0	1.3.0
2.9	2.4.3	1.4.0	1.4.0	1.4.0
2.10	2.5.3	1.5.0	1.5.0	1.5.0

---

Step 2: Use the Correct Constraints File

Align the MWAA version with Airflow constraints for Python 3.7:

• MWAA 2.7 → Airflow 2.2.2 constraints
• MWAA 2.8 → Airflow 2.3.3 constraints
• MWAA 2.9 → Airflow 2.4.3 constraints
• MWAA 2.10 → Airflow 2.5.3 constraints

---

Step 3: Test Dependencies Locally

To ensure your requirements.txt works before deploying on MWAA:

1. Set up a local Airflow environment using Docker:

   docker run -it --rm apache/airflow:2.2.2-python3.7

2. Install your requirements.txt:

   pip install -r requirements.txt

3. Run pip check to identify dependency conflicts:

   pip check  

   Example Output (if conflicts exist):

   dbt-redshift 1.3.0 has requirement redshift-connector==2.0.889, but you have redshift-connector 2.0.882 installed.

   If conflicts appear, update the problematic packages manually:

   pip install redshift-connector==2.0.889

---

Step 4: Lock Down Requirements

Once tested locally, finalize your requirements.txt for MWAA. For MWAA 2.7, it might look like:

dbt-core==1.2.3  
dbt-postgres==1.2.3  
dbt-redshift==1.2.1  
apache-airflow-providers-ssh==2.4.0  
cryptography==3.4.8  
pycryptodome==3.11.0  
pandas==1.3.4  

---

Closing Thoughts

Resolving dependency mismatches in MWAA comes down to using the correct Airflow constraints file and aligning package versions accordingly. Test locally using Docker and pip check to validate compatibility before deploying. Keep this version matrix handy to save time during future upgrades.

Hope this updated recipe solves your issue, Preet! Let me know if you need further clarification or if there are additional dependencies to untangle. You’ve got this! 🚀

Cheers, Aaron 😄