Requirements installation timeout

Hi,

The error "requirements.txt installation timed out after 9 minutes" occurs because the environment was spending time for backtracking to find a compatible versions. For this, pip will try every possible combination that it needs to and determine that there is no compatible set. It looks like it could not find it in given time frame[1]. This is also mentioned in the AWS documentation [2] - "Although Amazon MWAA doesn't limit the size of a plugins.zip file explicitly, if dependencies can't be installed within ten minutes, the Fargate service will time-out and attempt to rollback the environment to a stable state".

In your case, I suspect if there is any package version incompatibility introduced in the environment due to airflow version difference that might be creating difficulties for pip to install the packages. I recommend you to perform a testing using the MWAA local runner [3] using the same plugins and requirements to find out the culprit dependency and try to either eliminate it or change the package version. Also, if you plan on changing any package version, make sure that it aligns with the latest constraint file [4].

References:

[1] https://pip.pypa.io/en/latest/topics/dependency-resolution/

[2] https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-import-plugins.html

[3] https://github.com/aws/aws-mwaa-local-runner

[4] https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.10.txt