Can't run airflow backfill via cli

0

I am trying to run the equivalent of backfill -t TASKNAME DAGNAME

I copied the script from https://docs.aws.amazon.com/mwaa/latest/userguide/access-airflow-ui.html#call-mwaa-apis-cli , but it looks to me as if the environment running the cli does not have the same packages loaded as per my requirements.txt (Airflow can parse the DAGs)

Result is this

  
Traceback (most recent call last):  
  File "/usr/local/bin/airflow", line 37, in <module>  
    args.func(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 76, in wrapper  
    return f(*args, **kwargs)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 187, in backfill  
    dag = dag or get_dag(args)  
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 164, in get_dag  
    'parse.'.format(args.dag_id))  
airflow.exceptions.AirflowException: dag_id could not be found: sentinel-hourly. Either the dag did not exist or it failed to parse.  
  
\[2021-04-01 21:40:27,020] {{__init__.py:50}} INFO - Using executor CeleryExecutor  
\[2021-04-01 21:40:27,020] {{dagbag.py:417}} INFO - Filling up the DagBag from /usr/local/airflow/dags  
\[2021-04-01 21:40:27,023] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/test.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/test.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
\[2021-04-01 21:40:27,026] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/sentinel-hourly.py  
Traceback (most recent call last):  
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file  
    m = imp.load_source(mod_name, filepath)  
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source  
    module = _load(spec)  
  File "<frozen importlib._bootstrap>", line 696, in _load  
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked  
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module  
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed  
  File "/usr/local/airflow/dags/sentinel-hourly.py", line 3, in <module>  
    from airflow.providers.amazon.aws.operators.ecs import ECSOperator  
ModuleNotFoundError: No module named 'airflow.providers'  
tth-pl
demandé il y a 3 ans1368 vues
2 réponses
0

Hi!

CLI commands run on the Airflow web server, which for security reasons does not install plugins or requirements. There is an open Airflow issue https://github.com/apache/airflow/issues/15306 to address this.

In the interim, while not officially supported, a workaround is to run "airflow backfill -t TASKNAME DAGNAME" from a BashOperator. See https://github.com/aws-samples/amazon-mwaa-examples/tree/main/dags/bash_operator_script for an example.

Thanks!

AWS
John_J
répondu il y a 3 ans
0

Thanks for the answer, I was able to work around it by clearing or failing the earliest of the ones I wanted to re-run and selecting "future" in case its useful to anyone else.

tth-pl
répondu il y a 3 ans

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions