What's the best way to migrate DAGs between staging, prod environments?

0

I'm searching for a good way to automate migrating a DAG between multiple instances (staging/production) as part of a DevOps workflow. I would like to be able to run my DAGs in my staging environment with different configuration parameters (S3 bucket paths, etc.) and run the same DAG in my production environment without requiring a change to the DAG code (automate the migration).

Here is what I'm considering:

  1. Set an environment variable in Airflow/MWAA instance as part of initial setup (e.g. env=staging, env=prod)
  2. Create json configuration file with staging and production configuration parameters and store it with the DAGs
  3. Create a DAG that is a prerequisite for any DAGs which require configuration that checks Airflow environment variable and sets variables to staging/prod configuration parameters
  4. Use templated variables in DAGs requiring configuration

Is there a better way to approach this? Any advice is appreciated!

質問済み 1年前743ビュー
1回答
0

Hi, We've achieved this using the SSM Parameter store. created config in the parameter store and use plugins.py file to pull the configuration from SSM and set up environment variables.

https://docs.aws.amazon.com/mwaa/latest/userguide/samples-env-variables.html

回答済み 1年前
  • Thanks, your answer was helpful. Do the plugins, and subsequently the environment variables, get re-loaded only when the environment is built/re-built?

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ