Can't run airflow backfill via cli

0

I am trying to run the equivalent of backfill -t TASKNAME DAGNAME

I copied the script from https://docs.aws.amazon.com/mwaa/latest/userguide/access-airflow-ui.html#call-mwaa-apis-cli , but it looks to me as if the environment running the cli does not have the same packages loaded as per my requirements.txt (Airflow can parse the DAGs)

Result is this

  
Traceback (most recent call last):  
  File "/usr/local/bin/airflow", line 37, in <module>  
    args.func(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 76, in wrapper  
    return f(*args, **kwargs)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 187, in backfill  
    dag = dag or get_dag(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 164, in get_dag  
    'parse.'.format(args.dag_id))  
airflow.exceptions.AirflowException: dag_id could not be found: sentinel-hourly. Either the dag did not exist or it failed to parse.  
  
\[2021-04-01 21:40:27,020] {{__init__.py:50}} INFO - Using executor CeleryExecutor  
\[2021-04-01 21:40:27,020] {{dagbag.py:417}} INFO - Filling up the DagBag from /usr/local/airflow/dags  
\[2021-04-01 21:40:27,023] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/test.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/test.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
\[2021-04-01 21:40:27,026] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/sentinel-hourly.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/sentinel-hourly.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
tth-pl
已提问 3 年前1368 查看次数
2 回答
0

Hi!

CLI commands run on the Airflow web server, which for security reasons does not install plugins or requirements. There is an open Airflow issue https://github.com/apache/airflow/issues/15306 to address this.

In the interim, while not officially supported, a workaround is to run "airflow backfill -t TASKNAME DAGNAME" from a BashOperator. See https://github.com/aws-samples/amazon-mwaa-examples/tree/main/dags/bash_operator_script for an example.

Thanks!

AWS
John_J
已回答 3 年前
0

Thanks for the answer, I was able to work around it by clearing or failing the earliest of the ones I wanted to re-run and selecting "future" in case its useful to anyone else.

tth-pl
已回答 3 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则