Restart MWAA environment programmatically

0

Hi,

We use custom packages as dependencies of some of our PythonOperators. Therefore, when we update these packages, we'd like to reflect this update when used by a PythonOperator in MWAA.

The current solution today is to "restart" the MWAA in the UI (by simply editing anything in the MWAA environment configuration).

But we wondered if there was any solution in order to trigger this restart of the MWAA env programmatically, so it is possible to integrate this an extra step of a CodePipeline for instance.

Thanks

asked 3 years ago4838 views
2 Answers
0

Hi!

You can call update-environment, either with the new S3 version ID for your new plugins/requirements.txt or with an additional configuration override, such as custom.lastupdate = "<current time>", as part of your CI/CD pipeline that will force MWAA to restart.

Thanks!

AWS
John_J
answered 3 years ago
  • Not sure how to use the configuration override. I tried aws mwaa update-environment --name dev custom.lastupdate = "<current time>" but got an error 'Unknown options: =, <current time>, custom.lastupdate'

0

Note that, as of Airflow 2.0, you do not need to package custom operators as plugins and should simply place them somewhere in your S3 dags folder. These will be synced automatically by MWAA to each container and no restart is required. See https://airflow.apache.org/docs/apache-airflow/stable/plugins.html .

AWS
John_J
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions