AWS SecretsManager backend configuration for MWAA

Is it possible that this was resolved on 1.10.12 but not on 2.0.2? I've been running into this exact issue trying to set up the secrets manager on a 2.0.2 environment.

Hi!

Please try omitting the secrets.backend_kwargs entry as JSON is not currently supported for overrides.

Thanks!

Hi, thank you for the reply. Unfortunately that did give me the same result (gateway errors). I will double check again when I rebuild the dev environment the next time.

I also have a workaround by inheriting from the operator which needs the secret to set the config environment variables in the constructor with os.environment to enable the secrets manager backend. It is not nice, but as I just need it for one operator right now it is OK until there is a cleaner solution. Also allows me to set the prefixes.

Hi!

There may be an issue with the Web Server retrieving the secrets from the backend. The MWAA team is investigating and will push a fix if required.

Thanks!

Cool thank you, and also to confirm I tried again without kwargs and the webserver does not seem to come up if I do. Will keep using my workaround with a custom operator for now, which handles updating the backend config.

I will mark this question as answered, as we have established that I can't do what I want to do right now.

Overall though pretty happy with it MWAA in general.

Having same issue. Additional observations that may be helpful:

• When you create a new MWAA env with additional Airflow config options:

{
"secrets.backend": "airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend"
}

the environment does not come up, even though it gets the status "Available". No logs are created in CloudWatch for any of the components (scheduler, webserver, etc.), the UI webserver returns empty response when called via HTTP.

• When you create a MWAA env without addoitional config options, it comes up. Then update the env with the config options: takes long time to finish the update, the environment is running, but the configuration is unaffected. If you look at "conf.as_dict()", it shows:

"secrets": {
"backend": "",
"backend_kwargs": ""
}

There's a clear problem with MWAA and how it handles this. Also, this should be a very typical use-case: I can imagine lots of folks will want to configure connections using SecretsManager or SSM Param Store in MWAA.

Edited by: levahim on Jan 8, 2021 4:00 PM

This issue has been resolved. Please see https://docs.aws.amazon.com/mwaa/latest/userguide/troubleshooting.html#access-secrets-manager

Verified. It works.

We've also published a new guide for this topic - Configuring an Apache Airflow connection using a Secrets Manager secret key: https://docs.aws.amazon.com/mwaa/latest/userguide/connections-secrets-manager.html.

Thanks!

Hello,

when using secrets backend, updating an environment takes a very long time (usually about 2 hours). When looking at Cloudtrail, I can see that there are many "CreateNetworkInterface" and "DeleteNetworkInterface" events - seems like MWAA recreating network instances over and over again.

The environment is created with default settings (other than the secrets backend). VPC created using default cloudformation template as well. Execution role is also default (created when setting up MWAA) and of course this execution role has SecretsBackendReadWrite policy attached to it.

Any explanation why this is happening? Thanks

When using backend for airflow 2.0.2, I'm still getting the same error. The env is in 'Available' state but I'm getting gateway error while accessing the airfow UI. Does that mean this is not yet resolved for 2.0.2?

As of 12/28/2003 I am still seeing similar errors with 2.7, creating environment succeeds bur the moment the airflow config is included the environment fails to be created after a long time in pending.