MWAA =2.9.2 incompatible with sagemaker dependency

Question

As mwaa is ending support for versions <2.6.3. We decided to upgrade our cluster to 2.9.2 or 2.10.3. And we have sagemaker as a dependency. Because we need to include related constraints for a version, we discovered that it's impossible to install sagemaker. case 2.10.3:

from const
boto3==1.35.36
attrs==24.2.0

from sage >= 2.246
"attrs>=24,<26",
"boto3>=1.35.75,<2.0",

from sage < 2.246
"attrs>=23.1.0,<24",
"boto3>=1.35.75,<2.0"

No mathing config.

case 2.9.2:

from const
boto3==1.34.106
attrs==23.2.0
dill==0.3.1.1
urllib3==2.2.1

from sage >= 2.246
"attrs>=24,<26",
"boto3>=1.35.75,<2.0",

from  sagemaker<2.206.0
"urllib3<1.27",


and sage versions between 2.206.0/2.246.0:
multiprocess(part of sagemaker) 0.70.11 depends on dill>=0.3.3
The user requested (constraint) dill==0.3.1.1

multiprocess(part of sagemaker)<= 0.70.10 (there is no binary), they cannot be built with Python 3.11(mwaa python version) because of this issue.

I tested it on the cluster - it stuck, and on my local env, and the results is the same. As a result, there is no viable configuration for sagemaker in MWAA

As of now, we reverted back to MWAA version 2.5.1, which is gonna have end of support soon.

We are wondering if there is any solution to it?

re:Post Agent · Answer

The dependency conflict you're experiencing between Amazon MWAA (Managed Workflows for Apache Airflow) and SageMaker is a common challenge when working with Python dependencies across AWS services.

You're correct that MWAA is ending support for versions below 2.6.3, and your analysis of the dependency conflicts is accurate. The core issues are:

1. For MWAA 2.10.3: The boto3 and attrs version constraints between MWAA and SageMaker are incompatible.
2. For MWAA 2.9.2: Multiple conflicts exist with boto3, attrs, dill, and urllib3 versions.

Here are some potential solutions:

1. Use a virtual environment approach: Consider creating a Lambda Layer or using a custom container that includes the specific versions of dependencies you need. This isolates your SageMaker dependencies from the MWAA environment.

2. Pin specific versions: Create a requirements.txt file that explicitly pins compatible versions of all dependencies, including transitive ones. This might require finding a specific SageMaker version that works with the MWAA constraints.

3. Use AWS CDK to automate environment setup: You could use AWS CDK to provision your MWAA environment with specific configurations that help manage these dependencies.

4. Consider using the AWS SDK directly: Instead of using the SageMaker Python SDK within your DAGs, you could use boto3 directly to interact with SageMaker. This might avoid some of the dependency conflicts.

5. Contact AWS Support: Since this appears to be a compatibility issue between two AWS services, AWS Support might be able to provide specific guidance or workarounds.

The dependency conflicts you're experiencing are particularly challenging because they involve core libraries like boto3 and urllib3 that are used by multiple AWS services. As MWAA continues to update its supported Airflow versions, AWS may provide more guidance on managing these types of dependency conflicts.  
**Sources**  
[Community | Automate your Apache Airflow Environments](https://community.aws/content/2dtuZnzL2ABCZK5fTQkfScDFNnJ/automating-mwaa-environments-and-workflows)  
[error pip install -t python requests | AWS re:Post](https://repost.aws/questions/QU4Stfj49SQYqVYEi5wsGc1g/error-pip-install-t-python-requests)  
[get_environment - Boto3 1.38.27 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/mwaa/client/get_environment.html)  
[Community | Automate the Provisioning of Your Apache Airflow Environments](https://community.aws/tutorials/automating-mwaa-environments-and-workflows)

MWAA >=2.9.2 incompatible with sagemaker dependency

Add your answer

Relevant content