I am trying to run the equivalent of backfill -t TASKNAME DAGNAME
I copied the script from https://docs.aws.amazon.com/mwaa/latest/userguide/access-airflow-ui.html#call-mwaa-apis-cli , but it looks to me as if the environment running the cli does not have the same packages loaded as per my requirements.txt (Airflow can parse the DAGs)
Result is this
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 37, in <module>
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 76, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 187, in backfill
dag = dag or get_dag(args)
File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 164, in get_dag
'parse.'.format(args.dag_id))
airflow.exceptions.AirflowException: dag_id could not be found: sentinel-hourly. Either the dag did not exist or it failed to parse.
\[2021-04-01 21:40:27,020] {{__init__.py:50}} INFO - Using executor CeleryExecutor
\[2021-04-01 21:40:27,020] {{dagbag.py:417}} INFO - Filling up the DagBag from /usr/local/airflow/dags
\[2021-04-01 21:40:27,023] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/test.py
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file
m = imp.load_source(mod_name, filepath)
File "/usr/lib64/python3.7/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/airflow/dags/test.py", line 3, in <module>
from airflow.providers.amazon.aws.operators.ecs import ECSOperator
ModuleNotFoundError: No module named 'airflow.providers'
\[2021-04-01 21:40:27,026] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/sentinel-hourly.py
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file
m = imp.load_source(mod_name, filepath)
File "/usr/lib64/python3.7/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/airflow/dags/sentinel-hourly.py", line 3, in <module>
from airflow.providers.amazon.aws.operators.ecs import ECSOperator
ModuleNotFoundError: No module named 'airflow.providers'