Need help troubleshooting a t3.xlarge that is not accruing CPU credits and hence is always throttled.
One of my t3.xlarge instances in my EKS cluster is not accruing CPU credits and hence impacting the response of my services. The processes running on this particular t3.xlarge instance had a mis-configuration for an extended period that I'm thinking it's now being punished for that extended period (2 months) of high (between 50 and 60%) CPU usage.
I've since reconfigured the application and now CPU load is just under 5% and it's been under 5% for at least 24hrs yet my CPU Credits balance is still at 0 - see attached screenshot. Any pointers? Would a reboot resolve the issue?
- 최신
- 최다 투표
- 가장 많은 댓글
Thank you for the detailed description.
Wondering if your instance is in unlimited mode, so it's possible to have higher-than-baseline utilization for two months. After the utilization is below the baseline, you might be paying down the surplus credits [1], hence not accruing "CPU Credit Balance." In this case, review the metric CPUSurplusCreditBalance and see if it's gradually decreasing.
It also looks like the instance is not being throttled by the underlying infrastructure, as according to [2], t3.xlarge earned 96 credits per hour, while the "CPU Credit Usage" is at around 0.6 (per 5 minutes). Fine-tuning the cluster or the application would help to utilize its 40% baseline utilization per t3.xlarge vCPU.
[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances-unlimited-mode-concepts.html#unlimited-mode-surplus-credits [2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html#earning-CPU-credits
관련 콘텐츠
- 질문됨 한 달 전
AWS 공식업데이트됨 일 년 전- AWS 공식업데이트됨 일 년 전
Thanks for enlightening me, @weidi.
I deployed the cluster using kops and I believe it sets up the cluster in unlimited mode by default as I now see all T3.xlarge instances are setup as unlimited. Cloudwatch shows my CPUSurplusCreditBalance has dropped to below 280 so I suspect I'll soon start to see it accrue as expected. In any case, I think it it safe to say that I throttled myself by running a misconfigured service for so long - :(.
How do I mark this as resolved?
Just posting back to say that credit accrual is back on track. Now at 800 and climbing.
Thanks again for the enlightenment, @weidi.
Thank you for the update. I'm glad it helps :) @rePost-User-0768007