Need help troubleshooting a t3.xlarge that is not accruing CPU credits and hence is always throttled.

0

One of my t3.xlarge instances in my EKS cluster is not accruing CPU credits and hence impacting the response of my services. The processes running on this particular t3.xlarge instance had a mis-configuration for an extended period that I'm thinking it's now being punished for that extended period (2 months) of high (between 50 and 60%) CPU usage.

I've since reconfigured the application and now CPU load is just under 5% and it's been under 5% for at least 24hrs yet my CPU Credits balance is still at 0 - see attached screenshot. Any pointers? Would a reboot resolve the issue?

Enter image description here Enter image description here

已提問 1 年前檢視次數 597 次
1 個回答
2
已接受的答案

Thank you for the detailed description.

Wondering if your instance is in unlimited mode, so it's possible to have higher-than-baseline utilization for two months. After the utilization is below the baseline, you might be paying down the surplus credits [1], hence not accruing "CPU Credit Balance." In this case, review the metric CPUSurplusCreditBalance and see if it's gradually decreasing.

It also looks like the instance is not being throttled by the underlying infrastructure, as according to [2], t3.xlarge earned 96 credits per hour, while the "CPU Credit Usage" is at around 0.6 (per 5 minutes). Fine-tuning the cluster or the application would help to utilize its 40% baseline utilization per t3.xlarge vCPU.

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances-unlimited-mode-concepts.html#unlimited-mode-surplus-credits [2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html#earning-CPU-credits

AWS
weidi
已回答 1 年前
  • Thanks for enlightening me, @weidi.

    I deployed the cluster using kops and I believe it sets up the cluster in unlimited mode by default as I now see all T3.xlarge instances are setup as unlimited. Cloudwatch shows my CPUSurplusCreditBalance has dropped to below 280 so I suspect I'll soon start to see it accrue as expected. In any case, I think it it safe to say that I throttled myself by running a misconfigured service for so long - :(.

    How do I mark this as resolved?

  • Just posting back to say that credit accrual is back on track. Now at 800 and climbing.

    Thanks again for the enlightenment, @weidi.

  • Thank you for the update. I'm glad it helps :) @rePost-User-0768007

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南