Can't run airflow backfill via cli

0

I am trying to run the equivalent of backfill -t TASKNAME DAGNAME

I copied the script from https://docs.aws.amazon.com/mwaa/latest/userguide/access-airflow-ui.html#call-mwaa-apis-cli , but it looks to me as if the environment running the cli does not have the same packages loaded as per my requirements.txt (Airflow can parse the DAGs)

Result is this

  
Traceback (most recent call last):  
  File "/usr/local/bin/airflow", line 37, in <module>  
    args.func(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 76, in wrapper  
    return f(*args, **kwargs)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 187, in backfill  
    dag = dag or get_dag(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 164, in get_dag  
    'parse.'.format(args.dag_id))  
airflow.exceptions.AirflowException: dag_id could not be found: sentinel-hourly. Either the dag did not exist or it failed to parse.  
  
\[2021-04-01 21:40:27,020] {{__init__.py:50}} INFO - Using executor CeleryExecutor  
\[2021-04-01 21:40:27,020] {{dagbag.py:417}} INFO - Filling up the DagBag from /usr/local/airflow/dags  
\[2021-04-01 21:40:27,023] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/test.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/test.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
\[2021-04-01 21:40:27,026] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/sentinel-hourly.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/sentinel-hourly.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
tth-pl
asked 2 years ago316 views
2 Answers
0

Hi!

CLI commands run on the Airflow web server, which for security reasons does not install plugins or requirements. There is an open Airflow issue https://github.com/apache/airflow/issues/15306 to address this.

In the interim, while not officially supported, a workaround is to run "airflow backfill -t TASKNAME DAGNAME" from a BashOperator. See https://github.com/aws-samples/amazon-mwaa-examples/tree/main/dags/bash_operator_script for an example.

Thanks!

John_J
answered 2 years ago
0

Thanks for the answer, I was able to work around it by clearing or failing the earliest of the ones I wanted to re-run and selecting "future" in case its useful to anyone else.

tth-pl
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions