- Newest
- Most votes
- Most comments
MWAA has a maximum of 4 vCPUs (for a large environment). As such, you essentially have 4 parallel processes. Exceeding the Airflow default scheduler.parsing_processes of 2 will leave no resources left to actually schedule any tasks or update the heartbeat, as you're encountering with your error.
Creating session with aws_access_key_id
is an INFO level log item in the Airflow Amazon provider package and not specific to MWAA nor attached to any error condition.
Airflow is highly sensative to DAG parsing, as it runs Python import on every Python file. Recommend using .airflowignore to avoid parsing files that do not create top-level DAG objects and remove top-level code wherever possible (i.e. most code should be executed in a task, rather than by the parsing action--see example here)
Relevant content
- asked 2 years ago
- asked a year ago
- asked 2 years ago
- AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 8 months ago
- AWS OFFICIALUpdated 2 years ago