How do I resolve Secrets Manager integration issues with Amazon MWAA?

Short description

To store airflow connections and variables, a Secrets Manager secret can be configured as a secrets backend for Amazon MWAA. Incorrect airflow configurations might result in issues when you create or update an Amazon MWAA environment.

The following are common causes of Secrets Manager integration issues with Amazon MWAA:

• The Amazon MWAA role doesn't have the required permissions for Secrets Manager.
• The Amazon MWAA private subnets don't have the correct routes to Secrets Manager.
• The airflow configurations options are incorrect for Secrets Manager. For example, the class name is incorrect. The Class name key must be secrets.backend and the value must be airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend.
• The Amazon MWAA role doesn't have permissions for the AWS Key Management Service (AWS KMS) key that's used for Secrets Manager encryption.

Resolution

Resolve your Secrets Manager integration issues with Amazon MWAA based on the following scenarios:

Amazon MWAA isn't in an Available state

The following are common reasons that your Amazon MWAA environment isn't in an Available state:

• Incorrect airflow configuration options with the Secrets backend class on Amazon MWAA.
• The network is missing for Secrets Manager.
• Permissions are missing for Secrets Manager.
• The airflow connection aws_default was modified.

To resolve these issues, complete the following steps:

1. To check that the Amazon provider package is installed, complete the following:
   Open the Airflow UI.
   Under the Admin tab, select Provider.
   Check the apache-airflow-providers-amazon provider package.
2. Check that the correct airflow configuration class is specified for secrets.backend:
   Key: secrets.backend
   Value: airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
3. Check the Secrets Manager connection from the Amazon MWAA private subnets. The connection can be from either a NAT gateway or a Secrets Manager Amazon Virtual Private Cloud (Amazon VPC) endpoint.
4. Check that the Amazon MWAA role has the correct permissions for Secrets Manager and the AWS KMS key that's used with Secrets Manager.
5. Check that the airflow connection aws_default isn't modified. Also, don't create Secrets Manager with the aws_default name.

AWS CloudTrail shows an error for the GetSecretValue API call that's made by Amazon MWAA

The following error might appear when you run the GetSecretValue API by Amazon MWAA:

"errorCode": "ResourceNotFoundException",
"errorMessage": "Secrets Manager can't find the specified secret."

To resolve this error, complete the following steps:

1. To retrieve the details of the GetSecretValue API, use the CloudTrail API that's called by the user AmazonMWAA-airflow.
2. Check the Secrets Manager that's configured on the Amazon MWAA airflow connection or variable. The connections_prefix:airflow/connections prefix and name must match the Secrets name.
3. Check if the airflow connection or variable exists in Secrets Manager.

The Amazon MWAA role doesn't have the required permissions for Secrets Manager

The following error appears when the Amazon MWAA role doesn't have the required permissions for Secrets Manager:

"botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::123456789:assumed-role/rjf-use2-dev-demo-mwaa-env-mwaa-execution-role/AmazonMWAA-airflow is not authorized to perform: secretsmanager:GetSecretValue on resource: <connection_prefix>/secret_name because no identity-based policy allows the secretsmanager:GetSecretValue action"

To resolve this error, see Step one: Provide Amazon MWAA with permission to access Secrets Manager secret keys.

Large number of Secrets Manager API calls for aws_default by Amazon MWAA

To reduce the number API calls that Amazon MWAA makes to Secrets Manager, use a lookup pattern. For more information, see Optional lookup on the Apache Airflow website. When you specify a lookup pattern, you can lower the number of paths that Apache Airflow looks for. As a result, this lowers your costs when you use Secrets Manager with Amazon MWAA.

To specify a lookup pattern, specify the connections_lookup_pattern and variables_lookup_pattern parameters. These parameters accept a RegEx string as input. For example, to specify an aws_default lookup, enter the following for secrets.backend_kwargs:

"connections_lookup_pattern" : "^(?!aws-default).*"

Note: To use the connections_lookup_pattern and variables_lookup_pattern parameters, apache-airflow-providers-amazon version 7.3.0 or later must be installed. For more information, see Specifying newer provider packages.

Can't access the Secrets Manager connection

When Secrets Manager is configured with Amazon MWAA, the connection created at the configured connections_prefix path can be accessed. To access the connection, use the Secrets name by the airflow operator.

Example:
Note: Replace example-mysql-task with your MySQL task and example-connection_prefix/mysql-connection with your connection prefix and MySQL connection ID.

mysql_uri_operator = MySqlOperator(
        task_id='example-mysql-task',
        mysql_conn_id = '<example-connection_prefix>/example-mysql-connection
        sql=CREATE_SQL
    )

Related information

Configuring an Apache Airflow connection using a Secrets Manager secret

Move your Apache Airflow connections and variables to Secrets Manager

Source code for airflow.providers.amazon.aws.secrets.secrets_manager on the Apache Airflow website