How do I enable the custom connection types in the Airflow UI on Amazon MWAA with a private web server?

Description

Amazon Managed Workflows for Apache Airflow (MWAA) supports both Public and Private network access mode for the Apache Airflow UI. MWAA enables custom connection types through provider packages, some are available out-of-the-box, while others (e.g., Snowflake, MySQL) require installing the corresponding provider package. If you’re using MWAA with a private web server and notice that the custom connection type doesn't appear in the Airflow UI, it's likely because the relevant provider package is not installed correctly on the Airflow web server. This issue can impact both public and private web servers.

Several factors can cause this issue:

• MWAA environments without internet access(private mode) cannot reach pypi.org repository to download necessary packages.
• Dependency conflicts between packages in the requirements.txt or plugins.zip files may prevent successful installation.
• Beginning with Apache Airflow version 2.7.2, MWAA enforces package constraints. If your requirements.txt file doesn't include a --constraint line, MWAA will automatically apply one to ensure compatibility. Any dependency conflicts may cause the installation process to fail, leading to missing provider packages.
   

Resolution:

Amazon MWAA environments with private web servers (no public internet access) require all Python dependencies to be pre-packaged and hosted in Amazon S3. You can resolve this by pre-packaging the Python dependencies into a plugins.zip using the AWS MWAA local runner, uploading the file to your S3 bucket, and updating your environment configuration.

In this example, we will walk through enabling the Snowflake connection type in your MWAA environment. However, the same steps apply for other custom provider packages like MySQL, etc.

Use the aws-mwaa-local-runner tool to prepackage dependencies

Since the MWAA environment is in a private subnet, you need to manually build and prepackage dependencies using the aws-mwaa-local-runner tool. Download the version of aws-mwaa-local-runner that matches your Airflow version to an Amazon EC2 instance with internet access (Amazon Linux 2 recommended) or your local machine.

$ git clone --branch v2.10.1 https://github.com/aws/aws-mwaa-local-runner.git
$ cd aws-mwaa-local-runner

Build the local runner image

Make sure docker is installed, then run:

$ ./mwaa-local-env build-image  

Update the requirements.txt with necessary packages

In the requirements/requirements.txt file, list your dependencies along with the proper Airflow constraint. Be sure to choose versions that are compatible with your Airflow version and Python interpreter.

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.10.1/constraints-3.11.txt"

apache-airflow-providers-snowflake==5.7.0
snowflake-connector-python==3.12.1

Package the requirements into plugins.zip

Run the following command to generate the zip file. This will generate a plugins.zip file under the requirements/ folder containing all necessary wheel files.

$ ./mwaa-local-env package-requirements

Prepare your requirements.txt file to install from local plugins

Because the MWAA environment uses a private web server without internet access, you should remove the --constraint line (since it references an internet-hosted URL) and instead add the following lines to your requirements.txt, as recommended in the official documentation.

This version of requirements.txt will ensure MWAA installs from the local plugins.zip instead of relying on the internet (PyPI):

--find-links /usr/local/airflow/plugins
--no-index

apache-airflow-providers-snowflake==5.7.0
snowflake-connector-python==3.12.1

Upload and reference the files in your MWAA Environment

1. Upload both requirements.txt and plugins.zip to the S3 bucket configured for your MWAA environment.
2. Update your MWAA environment configuration to point to the new:
   • requirements.txt path
   • plugins.zip path

Validate Deployment

Once the update completes, go to the Amazon MWAA Airflow UI:  

• Navigate to Admin → Connections
• Click + to add a new connection
• You should now see Snowflake listed in the connection types
• Additionally, check the requirement_install logs in the MWAA web server log group in CloudWatch to verify that the packages installed correctly.

Note: The Amazon MWAA documentation also includes alternative methods to create a plugins.zip file without using the local runner.
 

Best Practices

• When updating requirements.txt and plugins.zip, ensure that your MWAA environment configuration references the latest versions of these files.
• Amazon MWAA recommends enabling versioning on your Amazon S3 bucket to track changes and allow referencing specific versions if needed

Related Information

Troubleshoot Amazon MWAA connection timeout errors
Install libraries in Amazon MWAA
Community-developed Airflow connection types
Hosting MWAA Local Runner on Amazon ECS Fargate