Can't run airflow backfill via cli

0

I am trying to run the equivalent of backfill -t TASKNAME DAGNAME

I copied the script from https://docs.aws.amazon.com/mwaa/latest/userguide/access-airflow-ui.html#call-mwaa-apis-cli , but it looks to me as if the environment running the cli does not have the same packages loaded as per my requirements.txt (Airflow can parse the DAGs)

Result is this

  
Traceback (most recent call last):  
  File "/usr/local/bin/airflow", line 37, in <module>  
    args.func(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 76, in wrapper  
    return f(*args, **kwargs)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 187, in backfill  
    dag = dag or get_dag(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 164, in get_dag  
    'parse.'.format(args.dag_id))  
airflow.exceptions.AirflowException: dag_id could not be found: sentinel-hourly. Either the dag did not exist or it failed to parse.  
  
\[2021-04-01 21:40:27,020] {{__init__.py:50}} INFO - Using executor CeleryExecutor  
\[2021-04-01 21:40:27,020] {{dagbag.py:417}} INFO - Filling up the DagBag from /usr/local/airflow/dags  
\[2021-04-01 21:40:27,023] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/test.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/test.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
\[2021-04-01 21:40:27,026] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/sentinel-hourly.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/sentinel-hourly.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
tth-pl
質問済み 3年前1423ビュー
2回答
0

Hi!

CLI commands run on the Airflow web server, which for security reasons does not install plugins or requirements. There is an open Airflow issue https://github.com/apache/airflow/issues/15306 to address this.

In the interim, while not officially supported, a workaround is to run "airflow backfill -t TASKNAME DAGNAME" from a BashOperator. See https://github.com/aws-samples/amazon-mwaa-examples/tree/main/dags/bash_operator_script for an example.

Thanks!

AWS
John_J
回答済み 3年前
0

Thanks for the answer, I was able to work around it by clearing or failing the earliest of the ones I wanted to re-run and selecting "future" in case its useful to anyone else.

tth-pl
回答済み 3年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ