Long running ec2 instances are slower than fresh instances.

Question

Hello   
  
I have many jobs that should roughly take about 10mins. Initially, on a fresh ec2 instance they do take 10 mins, but as that instance is reused for hours, jobs start taking 30mins. When I increased the compute environment size and new instances are created, their jobs are back down to 10 mins.  
  
When looking at old vs new instances, the new instances use about 90% of the cpu, while the older on is maybe 20%.  
  
Any idea, what is going on? and how I can fix this without shutting down running instances?

Answer

I didnt know EBS had burst, so thank you for that. You are most likely right. It would make sense why a new volume would be back to the original speed.  
  
Thank you

Answer

Don't know if you've sorted out your problems yet, but I'd suspect that your jobs are running out of their I/O allowance.  
  
Every EC2 instance - and, in particular, their attached EBS storage - has an I/O budget. If you consume I/O quickly, you eventually run out of your 'burst allowance', and get throttled back.  
  
You can verify that by using the CloudWatch stats to monitor the burst allowance for your EBS devices.

Long running ec2 instances are slower than fresh instances.

Relevant content