How do I install custom packages in my Amazon MWAA environment?

Resolution

Create a plugins.zip file and a requirements.txt file

Create a plugins.zip file, and then upload your plugins.zip file to Amazon Simple Storage Service (Amazon S3).

Create a requirements.txt file, and then upload your requirements.txt file to Amazon S3.

Install libraries with Python wheels

To install libraries with Python wheels, see How do I install Python libraries in my Amazon MWAA environment?

When you install a Python wheel, make sure that the WHL file is in your plugins.zip file. Then, include a path to the WHL file in your requirements.txt file. If you update the environment after you add the WHL file to the plugins.zip file, then the environment sends the WHL file to /usr/local/airflow/plugins/.

Install custom plugins

Amazon MWAA supports Apache Airflow's built-in plugin manager that allows you to use custom Apache Airflow operators, hooks sensors, or interfaces. To install custom plugins, see Installing custom plugins.

Use your own custom plugin

Complete the following steps:

1. Zip your custom plugin code into plugins.zip.
2. Upload your plugins.zip file to Amazon S3.
3. Install your custom plugin on your environment.

For examples of custom plugin creation, see Example use cases for plugins.zip. For information about how to set up constraints in your requirements file, see Create the requirements.txt. For information on how to upgrade the Constraint file, see Constraints file.

Export CRT, PEM, and configuration files

If you have files that don't need continuous updates when your environment runs, then zip these files into plugins.zip. Also, zip files that you don't need to grant access to users that write DAGs into plugins.zip. These files include certificate (CRT) files, PEM files, and configuration YAML files.

To zip CRT, PEM, or YAML files into the plugins.zip file, run the following command:

zip plugins.zip your-file.txt

Note: Replace your-file.txt with your CRT, PEM, or YAML file name and extension.

After you zip all files into plugins.zip, upload plugins.zip to Amazon S3. Then, update your environment. The files replicate with required permissions to access /usr/local/airflow/plugins.

Troubleshoot the installation process

If you can't install these packages, then use amazon-mwaa-docker-images to locally test your DAGs, custom plugins, or Python dependencies. For more information, see amazon-mwaa-docker-images on the GitHub website.

To troubleshoot Python package installation issues in the plugins.zip file, view the requirements_install_ip log file from either the Apache Airflow Worker or Scheduler log groups.

Related information

Plugins