Some Airflow Configuration Options lead to broken MWAA environment

Hi!

The MWAA team is working on a fix that will enable the AWS secrets manager backend and should be available soon.

Thanks!

That's good news! Thanks.

Hi John,

is there any ETA for the fix? I had the same issue today. It is poorly documented on AWS side and the 2+ hours updating state with Scheduler down was also very unpleasant.

Thank you in advance,
Petr

Yes secrets manager is now supported.

You can use the AWS Secrets manager backend by:

1. set the airflow configuration override secrets.backend to airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend

2. Add the connections/variables to AWS secrets manager, for example:
   a. for a variable called max_metadb_storage_days you would add airflow/variables/max_metadb_storage_days with a value of 14 to AWS Secrets Manager
   b. for a connection called my_db_connection you would add airflow/connections/my_db_connection with a value of 14 to AWS Secrets Manager

3. Add AWS secrets manager read policy to your MWAA environment’s execution role

backend_kwargs is not supported, however a workaround is to override the SecretsManager function call by adding the following to your DAGs (in this case adding a "2" to the prefix):

from airflow.contrib.secrets.aws_secrets_manager import SecretsManagerBackend

def get_variable(self, key):
return self._get_secret('airflow/variables2', key)
SecretsManagerBackend.get_variable=get_variable

def get_conn_uri(self, key):
return self._get_secret('airflow/connections2', key)
SecretsManagerBackend.get_conn_uri=get_conn_uri

def get_config(self, key):
return self._get_secret('airflow/config2', key)
SecretsManagerBackend.get_config=get_config