How do I resolve SIGTERM and SIGKILL errors in my Amazon MWAA environment?

Short description

"SIGTERM" and "SIGKILL" errors can occur consistently or intermittently during task execution. When a task requires more resources than what's available, the operating system (OS) sends a SIGTERM signal to gracefully shut down the task. When the SIGTERM signal can't shut down the task, the OS sends a SIGKILL signal to forcefully shut down the task.

You receive one of the following "SIGTERM" or "SIGKILL" error messages:

• "Task exited with return code Negsignal.SIGKILL"
• "{process_utils.py:133} INFO - Sending Signals.SIGTERM to group 198. PIDs of all processes in the group: [198]"
• "{process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 198"
• "{taskinstance.py:1479} ERROR - Received SIGTERM. Terminating subprocesses."
• "2023-07-26 13:00:49,356: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM) Job: 1049.')"
• "Error: airflow.exceptions.AirflowException: Task received SIGTERM signal"

You can view the preceding error messages in the Apache Airflow task logs and worker logs. If task logs are missing from the failing tasks, then "SIGTERM" errors might be in the worker logs.

Resolution

Reduce resource usage on the worker

Transfer large operations to other AWS services

Use Amazon CloudWatch to check whether a worker's CPUUtilization is above 90%. "SIGTERM" errors might occur when you run operations on the MWAA workers rather than through the MWAA environment. To resolve this issue, transfer large operations to different AWS services such as AWS Glue, Amazon EMR, or AWS Lambda for computation instead of MWAA workers.

Note: If there's high CPU usage on the metadata database, then use the TABLES_TO_CLEAN Directed Acyclic Graphs (DAG) code sample to delete metadata database entries.

Scale up the environment

Increase the environment class to allocate more resources.

Note: When you scale up the environment, tasks might still fail because of resource constraints. Tasks still might use all resources in a larger class environment.

Modify DAG scheduling

High worker CPU utilization can occur when you start a large number of DAGs at the same time. To more evenly distribute the workload, stagger the start times for the DAGs by a few minutes. If there are multiple DAGs with large tasks, then schedule them so that they aren't running at the same time.

Note: When you modify the DAG schedule, tasks might still fail because of resource constraints.

Reduce core.parallellism

Use the MWAA console to reduce the core.parallelism value on the worker so that more resources are available for fewer tasks.

Adjust the celery.worker_autoscale value

Apache Airflow can't assign a task to a specific worker, and a worker doesn't check resource utilization before it accepts a task. When you schedule tasks, a worker accepts the tasks according to the value that you set for celery.worker_autoscale. Unless you set the celery.worker_autoscale value to (1, 1), the same worker can still select multiple large tasks. The tasks immediately fail when the worker doesn't have enough resources to complete the tasks.

Use the MWAA console to reduce the celery.worker_autoscale maximum and minimum values so that they're less than the default values for your environment class. For example, the number of default concurrent tasks for the mw1.large environment class is 20 and can run up to 500 concurrent tasks. If you reduce the maximum value to 10, then your environment concurrently runs 250 tasks. It's a best practice to change the maximum and minimum values for resource-intensive tasks to allow more worker capacity for each task.

The worker can run the minimum number of tasks, and scale up to the maximum when needed.

Resolve the "SIGTERM" error for failed tasks

Check for a task instance heartbeat timeout

Task failure occurs when the workers that are running tasks no longer have the resources to run the task. The worker can't send a heartbeat to the metadata database because of a broken link between the task instance and worker. When there's no heartbeat for longer than the value that you set for scheduler_zombie_task_threshold, the scheduler sends the SIGTERM signal to shut down the task.

To check whether the schedular shut down the unresponsive task, use the CloudWatch console to view the ZombiesKilled Apache Airflow metric. Or, use scheduler_job.py in the scheduler logs. For more information, see scheduler_job.py on the GitHub website.

To prevent this issue, increase the value for scheduler_zombie_task_threshold from 5 minutes to 20 minutes or more. Also, it's a best practice to configure core.default_task_retries. Set the task retry parameter to more than or equal to 2. For more information, see default_task_retries on the Apache Airflow website.

Don't duplicate task instances

When the scheduler detects a duplicate task, it sends a SIGTERM signal to terminate the duplicate. To resolve this issue, don't duplicate task instances.

Check for SIGTERM signals on DagFileProcessorManager

The scheduler might send SIGTERM signals to the DagFileProcessorManager process. 

Reduce memory usage

If a task uses too much memory, then the system might terminate the task so that other tasks aren't affected.

To reduce memory usage, take the following actions:

• Modify tasks and DAG code so that they use less memory. For more information, see Best practices on the Apache Airflow website.
• Assign more resources to your environment.

Modify core.killed_task_cleanup_time

When a LocalTaskJob detects an unresponsive task, it sends a heartbeat signal to the scheduler to gracefully shut down the task. The value that you set for killed_task_cleanup_time determines the amount of time that the task has to clean up before the scheduler forcefully shuts down the task. 

For more information, see killed_task_cleanup_time on the Apache Airflow website.

Database connection timeouts

When running Airflow tasks in your MWAA environment try to connect to databases, such as Snowflake, PostgreSQL, or Amazon Redshift, connections might time out and cause "SIGTERM" errors.

To resolve this issue, set core.execute_tasks_new_python_interpreter to True in the Airflow configurations.

Note: The core.execute_tasks_new_python_interpreter parameter is activated by default.

Break long-running tasks into smaller jobs

MWAA uses Amazon Simple Queue Service (Amazon SQS) as the celery broker that has a maximum visibility timeout of 12 hours. If a task runs longer than 12 hours, then its Amazon SQS message might time out and cause Apache Airflow to forcefully shut down the task. During a task retry, the task tries to reprocess the SQS message. To resolve this issue, break long-running tasks into smaller jobs to complete within 12 hours.

Resolve CloudWatch PutLogEvents throttling

If the PutLogEvents CloudWatch API throttles, then Apache Airflow might experience "SIGTERM" errors.

To resolve this issue, identify why PutLogEvents API calls are throttling. PutLogEvents actions are throttled on a per-second per-account quota. If necessary, you can request a quota increase.

Avoid reliance on transit gateways

Network delays can intermittently cause "SIGTERM" errors, especially in MWAA environments that use Amazon VPC Transit Gateways. To minimize network delays, create AWS PrivateLink virtual private cloud (VPC) endpoints.

Note: VPC endpoints that you configure for private MWAA environments must use the same security group of the MWAA environment. Or, the VPC endpoints must have a route between the MWAA security group and the endpoint security group.

Make sure that DAG landing times don't exceed celery.task_adoption_timeout

If the landing time is greater than the value that you set for celery.task_adoption_timeout, then tasks fail. For more information see, Landing times and task_adoption_timeout on the Apache Airflow website.

To resolve this issue, make sure that landing times don't exceed the default value of 600 seconds for the task_queued_timeout parameter. For more information, see task_queued_timeout on the Apache Airflow website.

Note: In earlier versions of Apache Airflow, task_queue_timeout is task_adoption_timeout.

Related information

Scheduler_health_check_threshold on the Apache Airflow website

Configuring Amazon MWAA worker automatic scaling

Scheduler on the Apache Airflow website

Troubleshooting Amazon Managed Workflows for Apache Airflow

How do I troubleshoot the "Was the task killed externally" error in Amazon MWAA?