Can't run airflow backfill via cli

0

I am trying to run the equivalent of backfill -t TASKNAME DAGNAME

I copied the script from https://docs.aws.amazon.com/mwaa/latest/userguide/access-airflow-ui.html#call-mwaa-apis-cli , but it looks to me as if the environment running the cli does not have the same packages loaded as per my requirements.txt (Airflow can parse the DAGs)

Result is this

  
Traceback (most recent call last):  
  File "/usr/local/bin/airflow", line 37, in <module>  
    args.func(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 76, in wrapper  
    return f(*args, **kwargs)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 187, in backfill  
    dag = dag or get_dag(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 164, in get_dag  
    'parse.'.format(args.dag_id))  
airflow.exceptions.AirflowException: dag_id could not be found: sentinel-hourly. Either the dag did not exist or it failed to parse.  
  
\[2021-04-01 21:40:27,020] {{__init__.py:50}} INFO - Using executor CeleryExecutor  
\[2021-04-01 21:40:27,020] {{dagbag.py:417}} INFO - Filling up the DagBag from /usr/local/airflow/dags  
\[2021-04-01 21:40:27,023] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/test.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/test.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
\[2021-04-01 21:40:27,026] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/sentinel-hourly.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/sentinel-hourly.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
tth-pl
已提問 3 年前檢視次數 1424 次
2 個答案
0

Hi!

CLI commands run on the Airflow web server, which for security reasons does not install plugins or requirements. There is an open Airflow issue https://github.com/apache/airflow/issues/15306 to address this.

In the interim, while not officially supported, a workaround is to run "airflow backfill -t TASKNAME DAGNAME" from a BashOperator. See https://github.com/aws-samples/amazon-mwaa-examples/tree/main/dags/bash_operator_script for an example.

Thanks!

AWS
John_J
已回答 3 年前
0

Thanks for the answer, I was able to work around it by clearing or failing the earliest of the ones I wanted to re-run and selecting "future" in case its useful to anyone else.

tth-pl
已回答 3 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南