使用 AWS re:Post 即表示您同意 AWS re:Post 使用條款
AWS re:Postre:Post

mwaa pipeline scheduling issue

0

hi all,

I have a Dag that I schedule to run every quarterly using: 'schedule_interval='@quarterly', but for some reason it runs every day. I don't see the run in managed airflow UI but the operator is been executed (the pipeline delete some file and I can see its been deleted everyday). can you help me find the reason it happen?

additional information: the start date is schedule to the begging of the year and the catchup=False.

  • start_date: datetime(2022, 1, 1),
  • catchup=False,
主題
分析應用程式整合無伺服器運算
標籤
AWS Data PipelineAmazon Managed Workflows for Apache Airflow (MWAA)工作排程
語言
English
Shir
已提問 2 年前檢視次數 602 次
1 個回答
0

Hi!

A newly added DAG will generally run once, even with catchup=False, if there was a valid run time since start_date. To avoid this, set start date to the next run time you wish.

If there were failures and retries are enabled, that may account for the addition downstream executions. To confirm you can check your dag run history to see if there were any additional runs.

Cheers!

AWS
John_J
已回答 2 年前
  • Shir
    2 年前

    hey, thank you for your response. this isn't a new DAG, it's an old DAG that had a cron schedule (0 0 1 */3 *) and I changed it to @quarterly, and just to make sure it aligned with all the other DAGS I changed the start date to -datetime(2022, 1, 1). is it possible that the dag runs without any documentation in the airflow UI? I attached a snippet that shows that no runs were executed even dough I know that it did run. *UPDATE- it seems that there is no place to add snippet.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋，同時有助於提問者的專業成長。

回答問題指南

相關內容