Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

EKS: StorageReady warnings every 10 minute on managed nodes

1

Maybe someone can help me. I have created a new EKS cluster with a managed node group of 4 nodes. On each nodes I am seeing the following warning every 10 minute. Don't understand what could be the problem, since the nodes are fully managed by AWS. Maybe there is a problem with aws-ebs-csi-driver add-on configuration?

The warning message is IODelays: Process (jbd2/nvme1n1-8) (PID 116703) incurred 290.9 seconds of I/O delay

Warnings

Topics
Containers
Tags
Amazon Elastic Kubernetes Service
Language
English

asked 2 years ago394 views

3 Answers
1

The warnings you're seeing about StorageReady every 10 minutes on your managed nodes in EKS are likely related to the node monitoring and auto repair features of Amazon EKS. These features are designed to continuously monitor the health of nodes and automatically react to detected problems.

The StorageReady condition is one of several conditions that the node monitoring agent checks, along with others like KernelReady and NetworkingReady. These checks help ensure the overall health and availability of your cluster.

It's important to note that these warnings don't necessarily indicate a problem with your aws-ebs-csi-driver add-on configuration. Instead, they are part of the normal health checking process for EKS managed nodes.

However, if these warnings persist or cause issues, you may want to investigate further. Here are a few steps you can take:

  1. Check the status of your nodes using kubectl commands to get more detailed information about the node conditions.

  2. Verify that the ebs-csi-controller-sa service account is correctly set up in the kube-system namespace of your cluster. This service account is crucial for managing EBS volumes in Kubernetes.

  3. Ensure that the EBS CSI Driver Add-On is properly installed and configured in your cluster.

  4. Review the events reported by the node monitoring agent to get more context about these warnings.

If the issue persists after these checks, you may want to contact AWS support for further assistance, as the nodes are fully managed by AWS and there might be an underlying issue that needs their attention.
Sources
Enable node auto repair and investigate node health issues - Amazon EKS
Community | Dynamic Database Storage with the Amazon EBS CSI Driver for Amazon EKS
View the health status of your nodes - Amazon EKS

answered 2 years ago

EXPERT

reviewed 2 years ago

  • piercosta
    a year ago

    HI, i performed kubectl events check on the nodes and it seems to me there are a lot of errors. I am not experiencing any specific performance issue, but I am afraid that this can lead to intermittent issues. Do you think the problem is serious? How can I check if the EBS add on configuration is correct?

    root@PIERRS:/home/pier# kubectl events node ip-172-22-116-165.eu-south-1.compute.internal
LAST SEEN               TYPE      REASON            OBJECT                                               MESSAGE
57m (x11 over 3h17m)    Warning   StorageReady      Node/ip-172-22-116-165.eu-south-1.compute.internal   IODelays: Process (systemd-journal) (PID 1252) incurred 25.8 seconds of I/O delay
57m                     Warning   StorageReady      Node/ip-172-22-116-165.eu-south-1.compute.internal   IODelays: Process (postgres) (PID 10885) incurred 163.9 seconds of I/O delay
57m                     Warning   StorageReady      Node/ip-172-22-116-165.eu-south-1.compute.internal   IODelays: Process (xfsaild/nvme0n1) (PID 1193) incurred 138.7 seconds of I/O delay
57m                     Warning   StorageReady      Node/ip-172-22-40-186.eu-south-1.compute.internal    IODelays: Process (systemd-journal) (PID 1251) incurred 49.3 seconds of I/O delay
51m                     Warning   StorageReady      Node/ip-172-22-66-189.eu-south-1.compute.internal    IODelays: Process (xfsaild/nvme0n1) (PID 1194) incurred 236.1 seconds of I/O delay
51m                     Warning   StorageReady      N
0

I am seeing the same thing. Attached screenshot from one of the events (from Datadog's side).Enter image description here

answered a year ago

0

This is pretty noisy. Can we disable these events?

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content