mwaa pipeline scheduling issue

hi all,

I have a Dag that I schedule to run every quarterly using: 'schedule_interval='@quarterly', but for some reason it runs every day. I don't see the run in managed airflow UI but the operator is been executed (the pipeline delete some file and I can see its been deleted everyday). can you help me find the reason it happen?

additional information: the start date is schedule to the begging of the year and the catchup=False.

start_date: datetime(2022, 1, 1),
catchup=False,

주제

분석 애플리케이션 통합 서버리스 컴퓨팅

태그

AWS 데이터 파이프라인 아파치 에어플로우용 아마존 관리형 워크플로(MWAA)작업 일정 관리

언어

English

Shir

질문됨 2년 전601회 조회

1개 답변

최신
최다 투표
가장 많은 댓글

이 답변이 도움이 되었나요?커뮤니티가 여러분의 지식을 활용할 수 있도록 정답을 찬성하세요.

Hi!

A newly added DAG will generally run once, even with catchup=False, if there was a valid run time since start_date. To avoid this, set start date to the next run time you wish.

If there were failures and retries are enabled, that may account for the addition downstream executions. To confirm you can check your dag run history to see if there were any additional runs.

Cheers!

John_J

답변함 2년 전

Shir
2년 전
hey, thank you for your response. this isn't a new DAG, it's an old DAG that had a cron schedule (0 0 1 */3 *) and I changed it to @quarterly, and just to make sure it aligned with all the other DAGS I changed the start date to -datetime(2022, 1, 1). is it possible that the dag runs without any documentation in the airflow UI? I attached a snippet that shows that no runs were executed even dough I know that it did run. *UPDATE- it seems that there is no place to add snippet.

mwaa pipeline scheduling issue

관련 콘텐츠