ec2 availability error,task kcompactd0:33 blocked for more than 120 seconds

Question

My Singapore ec2 is not accessible via ssh and the /var/log/syslog logs are as follows:
"

```
INFO: task kcompactd0:33 blocked for more than 120 seconds.
Not tainted 5.19.0-1025-aws #26~22.04.1-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kcompactd0      state:D stack:    0 pid:   33 ppid:     2 flags:0x00004000
Call Trace:

Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018165]  __schedule+0x254/0x5a0
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018169]  schedule+0x5d/0x100
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018171]  io_schedule+0x46/0x80
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018173]  ? wbt_exit+0x40/0x40
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018177]  rq_qos_wait+0xd0/0x170
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018180]  ? wbt_rqw_done+0x120/0x120
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018182]  ? sysv68_partition+0x350/0x350
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018184]  ? wbt_exit+0x40/0x40
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018187]  wbt_wait+0xad/0x100
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018190]  __rq_qos_throttle+0x28/0x40
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018192]  blk_mq_get_new_requests+0xa6/0x190
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018196]  blk_mq_submit_bio+0x1eb/0x450
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018199]  __submit_bio+0xf6/0x190
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018201]  submit_bio_noacct_nocheck+0xc2/0x120
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018204]  submit_bio_noacct+0x209/0x560
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018206]  submit_bio+0x40/0xf0
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018207]  ext4_io_submit+0x24/0x50
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018210]  ext4_writepages+0x28a/0x870
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018214]  do_writepages+0xd4/0x1e0
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018219]  ? radix_tree_lookup+0xd/0x20
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018222]  ? balance_dirty_pages_ratelimited+0x169/0x340
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018225]  filemap_fdatawrite_wbc+0x7c/0xb0
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018228]  __filemap_fdatawrite_range+0x54/0x80
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018231]  file_write_and_wait_range+0x5b/0xc0
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018234]  ext4_sync_file+0x109/0x340
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018237]  vfs_fsync_range+0x49/0xa0
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018241]  __x64_sys_fsync+0x38/0x70
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018242]  do_syscall_64+0x5c/0x90
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018244]  ? exit_to_user_mode_prepare+0x3b/0xd0
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018246]  ? exit_to_user_mode_prepare+0x3b/0xd0
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018247]  ? irqentry_exit_to_user_mode+0x9/0x20
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018250]  ? irqentry_exit+0x21/0x40
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018251]  ? exc_page_fault+0x92/0x190
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018253]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018256] RIP: 0033:0x7f2c8411ba5b
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018258] RSP: 002b:00007f2c7a5fec50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018260] RAX: ffffffffffffffda RBX: 000055901a425a80 RCX: 00007f2c8411ba5b
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018261] RDX: 0000000000000000 RSI: 00007f2c8464ea14 RDI: 000000000000000a
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018262] RBP: 00007f2c846502a0 R08: 0000000000000000 R09: 00007fff6687dd7f
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018263] R10: 00007f2c8409499a R11: 0000000000000293 R12: 0000000000000002
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018264] R13: 0000000000000002 R14: 00007f2c84094850 R15: 00007fff6687ddc0
Aug 19 08:11:14 zmq-projects-15 kernel: [6795865.018266]  
```

"

Answer

According to this Red Hat knowledge article (requires a RH login) https://access.redhat.com/solutions/31453 these messages typically mean that the system is experiencing resource congestion like memory,I/O,network and semaphore,etc. and processes are being starved of available resources.

I notice you're running Ubuntu but this will still be applicable here.

If you don't have Red Hat login then this covers much the same ground https://www.thegeekdiary.com/info-task-processpid-blocked-for-more-than-120-seconds-system-becomes-unresponsive/

In the AWS EC2 Console look in the Monitoring tab for the instance, is there anything suggesting excessive resource consumption?

ec2 availability error,task kcompactd0:33 blocked for more than 120 seconds

Relevant content