Some Airflow Configuration Options lead to broken MWAA environment

0

Creating a new MWAA env with some additional Airflow config options creates an env that's marked as "Available", but non-functioning. No env component logs are posted in CloudWatch and the Airflow UI is unreachable (returns empty response when any HTTP request is made to it). For example, when "CreateEnvironment" is called with "secrets.backend": "airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend" in AirflowConfigurationOptions", such non-functioning MWAA env is created.

Additional observation:

When you create an MWAA env without additional config options, it comes up and is functioning. Then update the env with the config options: takes long time to finish the update, eventually the environment is running and is marked as "Available", but the configuration is unaffected. If you look at "conf.as_dict()", it shows:

"secrets": {
"backend": "",
"backend_kwargs": ""
}

There appears to be a problem with MWAA and how it handles the additional config options. Is this something MWAA team looking into? Any time frame for the resolution if so?

Edited by: levahim on Jan 11, 2021 12:04 PM

已提問 4 年前檢視次數 822 次
4 個答案
0

Hi!

The MWAA team is working on a fix that will enable the AWS secrets manager backend and should be available soon.

Thanks!

AWS
已回答 4 年前
0

That's good news! Thanks.

已回答 4 年前
0

Hi John,

is there any ETA for the fix? I had the same issue today. It is poorly documented on AWS side and the 2+ hours updating state with Scheduler down was also very unpleasant.

Thank you in advance,
Petr

已回答 4 年前
0

Yes secrets manager is now supported.

You can use the AWS Secrets manager backend by:

  1. set the airflow configuration override secrets.backend to airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend

  2. Add the connections/variables to AWS secrets manager, for example:
    a. for a variable called max_metadb_storage_days you would add airflow/variables/max_metadb_storage_days with a value of 14 to AWS Secrets Manager
    b. for a connection called my_db_connection you would add airflow/connections/my_db_connection with a value of 14 to AWS Secrets Manager

  3. Add AWS secrets manager read policy to your MWAA environment’s execution role

backend_kwargs is not supported, however a workaround is to override the SecretsManager function call by adding the following to your DAGs (in this case adding a "2" to the prefix):

from airflow.contrib.secrets.aws_secrets_manager import SecretsManagerBackend

def get_variable(self, key):
return self._get_secret('airflow/variables2', key)
SecretsManagerBackend.get_variable=get_variable

def get_conn_uri(self, key):
return self._get_secret('airflow/connections2', key)
SecretsManagerBackend.get_conn_uri=get_conn_uri

def get_config(self, key):
return self._get_secret('airflow/config2', key)
SecretsManagerBackend.get_config=get_config

AWS
已回答 4 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南