How to resolve very high CPU steal time on a t3.large instance without using dedicated and reserved instances

t3* families are burstable instance types, that operate on a credit system, you can read more about this here. Each t3 instance comes with a certain baseline CPU utilization. If your instance has a CPU utilization below the baseline, then it will earn CPU credits, and if it is above the baseline it will use credits. The default for new T3 instances is to run in unlimited mode for an additional hourly fee when the average utilization is above the baseline utilization.

A non-zero steel time, means that your application is demanding more CPU than the hypervisor is allowing you to have at that moment. This could be due to your instance not running in unlimited mode, and having exhausted its credits. If you are running in unlimited mode, then you could always perform an stop-start on the instance, which would cause you to move to a different underlying host, and may resolve the issue.

Your top output shows the CPU at 30% utilisation, which is the baseline value for t3.large. It can burst higher than that, but only until you've depleted your accrued credits. You can find it in the "baseline utilization per vCPU" column in this table: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html#earning-CPU-credits

If you consistently need more than 30% of the nominal power of the CPU cores you have, your options are either to reduce CPU utilisation or switch to a more powerful instance.

You can use the "Instance Types" view in the EC2 console to browse available instance types. You can filter based on Architecture = x86_64, expose the "cores" and "sustained clock speed" columns, and sort in ascending order by "on-demand linux pricing", if you're using a Linux distribution that doesn't require additional licences. Looking at the eu-west-1 region (Ireland) as an example, r7a.medium would give 100% of the capacity of a 3.7-GHz processor core with 8 GiB of memory, while costing slightly less (at least in Ireland) than your current t3.large that only gives 30% of a 2.5-GHz processor core and the same 8 GiB of memory.

If you're running commodity software and are willing to put in more effort, you could get an even better price by switching from Intel's processors to AWS's own Graviton series of processors. They are based on the Arm processor architecture, as opposed to the Intel/AMD x86_64 architecture, so you'd have to install a new operating system from a fresh AMI and also reinstall all your applications from packages compiled for Arm processors. However, that would be a one-time operation that would permanently allow you to use Graviton-based processors that usually give a substantially better price-performance ratio over Intel/AMD processors.

For example, still looking at eu-west-1, the instance type r7g.medium would have the same 8 GiB of memory and give 100% of a 2.6-GHz Graviton3 processor core, which is more powerful than an Intel processor core running at the same clock frequency. In eu-west-1, it's about 30% cheaper than the r7a.medium, which was about 6.7% cheaper than the original t3.large.

You can find the exact prices and available instance types for your region in the Instance Types view in the EC2 console. The "Architecture" column, based on which you can also filter, distinguishes between Arm-based and x86_64-based processors.