MWAA 2.4.3 - can't install `mysqlclient==2.1.1` due to missing OS dependencies

Question

I'm trying to upgrade my Airflow MWAA Cluster to 2.4.3, but there's a crucial dependency for MySQL access that can't be installed due to missing OS dependencies.

Here is my `requirements.txt`:

```
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt"
crypto
plyvel
apache-airflow-providers-amazon
apache-airflow-providers-common-sql
apache-airflow-providers-cncf-kubernetes
snowflake-connector-python
snowflake-sqlalchemy
apache-airflow-providers-snowflake
mysql-connector-python
apache-airflow-providers-slack
croniter
apache-airflow-providers-mysql

```

And these are the logs I'm getting from Cloudwatch when I try to install those dependencies:

```

2023-02-09T14:37:06.229-05:00	Collecting mysqlclient>=1.3.6
2023-02-09T14:37:06.244-05:00	Downloading mysqlclient-2.1.1.tar.gz (88 kB)
2023-02-09T14:37:06.260-05:00	━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.1/88.1 kB 15.0 MB/s eta 0:00:00
2023-02-09T14:37:06.279-05:00	Preparing metadata (setup.py): started
2023-02-09T14:37:06.294-05:00	Preparing metadata (setup.py): finished with status 'error'
2023-02-09T14:37:06.309-05:00	error: subprocess-exited-with-error
2023-02-09T14:37:06.345-05:00	
2023-02-09T14:37:06.361-05:00	× python setup.py egg_info did not run successfully.
2023-02-09T14:37:06.375-05:00	│ exit code: 1
2023-02-09T14:37:06.391-05:00	╰─> [16 lines of output]
2023-02-09T14:37:06.405-05:00	mysql_config --version
2023-02-09T14:37:06.419-05:00	/bin/sh: mysql_config: command not found
2023-02-09T14:37:06.487-05:00	mariadb_config --version
2023-02-09T14:37:06.527-05:00	/bin/sh: mariadb_config: command not found
2023-02-09T14:37:06.542-05:00	mysql_config --libs
2023-02-09T14:37:06.556-05:00	/bin/sh: mysql_config: command not found
2023-02-09T14:37:06.575-05:00	Traceback (most recent call last):
2023-02-09T14:37:06.589-05:00	File "", line 2, in 
2023-02-09T14:37:06.655-05:00	File "", line 34, in 
2023-02-09T14:37:06.670-05:00	File "/tmp/pip-install-0q1pd5qd/mysqlclient_fcbf326f5008433ba2f5254795ee81da/setup.py", line 15, in 
2023-02-09T14:37:06.687-05:00	metadata, options = get_config()
2023-02-09T14:37:06.705-05:00	File "/tmp/pip-install-0q1pd5qd/mysqlclient_fcbf326f5008433ba2f5254795ee81da/setup_posix.py", line 70, in get_config
2023-02-09T14:37:06.722-05:00	libs = mysql_config("libs")
2023-02-09T14:37:06.745-05:00	File "/tmp/pip-install-0q1pd5qd/mysqlclient_fcbf326f5008433ba2f5254795ee81da/setup_posix.py", line 31, in mysql_config
2023-02-09T14:37:06.761-05:00	raise OSError("{} not found".format(_mysql_config_path))
2023-02-09T14:37:06.773-05:00	OSError: mysql_config not found
2023-02-09T14:37:06.790-05:00	[end of output]
2023-02-09T14:37:06.808-05:00	
2023-02-09T14:37:06.824-05:00	note: This error originates from a subprocess, and is likely not a problem with pip.
2023-02-09T14:37:06.842-05:00	error: metadata-generation-failed
2023-02-09T14:37:06.859-05:00	
2023-02-09T14:37:06.872-05:00	× Encountered error while generating package metadata.
2023-02-09T14:37:06.904-05:00	╰─> See above for output.
2023-02-09T14:37:06.917-05:00	
2023-02-09T14:37:06.932-05:00	note: This is an issue with the package mentioned above, not pip.
2023-02-09T14:37:06.949-05:00	hint: See above for details.
2023-02-09T14:37:06.963-05:00	
2023-02-09T14:37:06.990-05:00	[notice] A new release of pip available: 22.3.1 -> 23.0
2023-02-09T14:37:07.009-05:00	[notice] To update, run: pip install --upgrade pi
```

The PyPi description of the package talks about OS dependencies that must be present for this package to be installed - https://pypi.org/project/mysqlclient/ .

It's important to note that I don't currently have _any_ issues installing this package tree with MWAA `2.2.2`.

Given MySQL is such an essential piece of our tech stack, this makes us unable to upgrade airflow to the latest version. But why was that OS dependency removed? Why there's no mention of it anywhere in the docs? Am I doing something wrong? How can I fix this?

Any help is appreciated.

Answer

Hi,

I see that you are receiving the following error while installing "apache-airflow-providers-mysql" package during upgrading your Airflow MWAA Cluster to 2.4.3 version.

```
× python setup.py  egg_info did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      mysql_config --version
      /bin/sh: mysql_config: command not found
      mariadb_config --version
      /bin/sh: mariadb_config: command not found
      mysql_config --libs
      /bin/sh: mysql_config: command not found
      Traceback (most recent call last):
        File "", line 2, in 
        ......
        not found".format(_mysql_config_path))
      OSError: mysql_config not found
```

Fix for installing apache-airflow-provider-mysql in MWAA environment 2.4.3 is now deployed to all regions. Please update your MWAA environments so the web server, schedulers, and workers can restart with the fix.

On the console you need to click 'Edit' for your MWAA environment then click 'Next' on the 'Specify details' and 'Configure advanced settings' pages, then 'Save' on the 'Review and save' page, then the environment will update. Or the you can use the update-environment mwaa cli command for your environment and not use any additional options for the command.

I had tested the same recently while deploying a MWAA cluster with 2.4.3 version in us-west-2 region which deployed successfully along with the mysql provider package. I was able to install below requirement file on MWAA without any issue.

```
requirements.txt
=============
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.4.3/constraints-3.10.txt "
apache-airflow[amazon,docker,mysql]==2.4.3
pendulum==2.1.2
pydantic==1.10.4
pyaml==21.10.1
mysql-connector-python==8.0.31
sqlparse==0.4.3
deepmerge==1.1.0
```

Thank you.

MWAA 2.4.3 - can't install `mysqlclient==2.1.1` due to missing OS dependencies

Relevant content