Secrets backend support in MWAA

2

Is any other secrets backend besides AWS Secrets manager supported in MWAA? I would like to use Hashicorp Vault, and the Airflow provider loads fine, but when i try to set secrets.backend_kwargs it seems fail.

BenO
asked a year ago690 views
2 Answers
1

Facing same issue, when I set Vault as secret backend MWAA env. stuck for couple of hours in create/update state and then fails after couple of hours

Using AWS Secrets manager works fine

Also, manually initiating VaultBackend class in Python operator to get secrets works secrets fine

So, I assume there is no connectivity issue in my case

It is very complicated to analyse such failure, as there is no log and Error message is pointing to documentation

Grisha
answered a year ago
-1

Hello, As you may already aware, Amazon MWAA is a managed service for Apache Airflow that lets you use your current, familiar Apache Airflow platform to orchestrate your workflows. MWAA runs open source Apache Airflow. So if Airflow supports a third party integration, then usually you can be able to use it with MWAA.

Airflow supports Hashicorp Vault as shown here. https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/secrets-backends/hashicorp-vault.html. The possible reasons for this failure could be IAM permissions issues or networking issues. If the vault is stored externally then please ensure whether you are able to connect it from the subnet you are using in MWAA. Please make sure to have a route to connect to vault if is placed inside EC2.

Usage of a custom secrets backend is outside of the scope of the MWAA service team, and in general we cannot provide specific guidance. Note that the secrets backend may need to be accessible from the web server, depending on how it's implemented, and web servers in MWAA cannot access VPCs outside AWS.

AWS
answered a year ago
  • One follow on question, when the MWAA is in Private network mode, is it possible for the web servers to initiate traffic back into my VPC? Our vault is in our VPC.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions