Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

EC2 instance nvme disk suddenly Buffer I/O error

0

EC2 server (VM) disk, nvme, looks fault and remounted as read only. Is this ssd disk problem? Please let me know how to resolve. Here's extract of dmesg log.

[Sun Aug 4 15:26:36 2024] IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link becomes ready [Mon Aug 12 18:40:31 2024] nvme nvme1: I/O 0 QID 1 timeout, aborting [Mon Aug 12 18:40:31 2024] nvme nvme1: I/O 20 QID 1 timeout, aborting [Mon Aug 12 18:41:01 2024] nvme nvme1: Abort status: 0x0 [Mon Aug 12 18:41:02 2024] nvme 0000:00:1f.0: irq 34 for MSI/MSI-X [Mon Aug 12 18:41:32 2024] blk_update_request: I/O error, dev nvme1n1, sector 28721152 [Mon Aug 12 18:41:32 2024] EXT4-fs warning (device nvme1n1): ext4_end_bio:302: I/O error -5 writing to inode 3937974 (offset 0 size 1048576 starting block 3590208) [Mon Aug 12 18:41:32 2024] Buffer I/O error on device nvme1n1, logical block 3590144 Mon Aug 12 18:41:33 2024] nvme nvme1: failed to mark controller state 1 [Mon Aug 12 18:41:33 2024] nvme nvme1: Removing after probe failure status: 0 [Mon Aug 12 18:41:33 2024] Aborting journal on device nvme1n1-8. [Mon Aug 12 18:41:33 2024] nvme1n1: detected capacity change from 107374182400 to 0 [Mon Aug 12 18:41:33 2024] Buffer I/O error on dev nvme1n1, logical block 13139968, lost sync page write [Mon Aug 12 18:41:33 2024] JBD2: Error -5 detected when updating journal superblock for nvme1n1-8. [Mon Aug 12 18:41:33 2024] Buffer I/O error on dev nvme1n1, logical block 0, lost sync page write [Mon Aug 12 18:41:33 2024] EXT4-fs error (device nvme1n1): ext4_journal_check_start:56: Detected aborted journal [Mon Aug 12 18:41:33 2024] EXT4-fs (nvme1n1): Remounting filesystem read-only [Mon Aug 12 18:41:33 2024] EXT4-fs (nvme1n1): previous I/O error to superblock detected [Mon Aug 12 18:41:33 2024] EXT4-fs (nvme1n1): Remounting filesystem read-only [Mon Aug 12 18:41:33 2024] EXT4-fs (nvme1n1): previous I/O error to superblock detected

Topics
Compute
Tags
Amazon EC2
Language
English

asked 2 years ago990 views

1 Answer
0
Accepted Answer

Hello.

If you look at the log, a timeout error has occurred, so why not try increasing the timeout value for I/O operations?
Please check whether "/sys/module_nvme_core/parameters_io_timeout" is set to "4294967295".
https://docs.aws.amazon.com/ebs/latest/userguide/nvme-ebs-volumes.html#timeout-nvme-ebs-volumes

The URL below introduces a similar error and explains that it is a hardware problem.
If it is truly a hardware failure, the user will not be able to resolve the issue and will need to contact AWS Support.
https://support.teradata.com/knowledge?id=kb_article_view&sys_kb_id=b02e21ec875fc998c693baa7cebb3560

EXPERT

answered 2 years ago

EXPERT

reviewed a year ago

  • mac-dpam
    2 years ago

    Thank you so much for the comment. io_timeout was 30, so increased it to larger number, rebooted. There's no more issue. It's likely that the same problem will occur again as 30 seconds, so it would be best to set it to a larger value.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content