Long running ec2 instances are slower than fresh instances.



I have many jobs that should roughly take about 10mins. Initially, on a fresh ec2 instance they do take 10 mins, but as that instance is reused for hours, jobs start taking 30mins. When I increased the compute environment size and new instances are created, their jobs are back down to 10 mins.

When looking at old vs new instances, the new instances use about 90% of the cpu, while the older on is maybe 20%.

Any idea, what is going on? and how I can fix this without shutting down running instances?

asked 5 years ago330 views
2 Answers

Don't know if you've sorted out your problems yet, but I'd suspect that your jobs are running out of their I/O allowance.

Every EC2 instance - and, in particular, their attached EBS storage - has an I/O budget. If you consume I/O quickly, you eventually run out of your 'burst allowance', and get throttled back.

You can verify that by using the CloudWatch stats to monitor the burst allowance for your EBS devices.

answered 5 years ago

I didnt know EBS had burst, so thank you for that. You are most likely right. It would make sense why a new volume would be back to the original speed.

Thank you

answered 5 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions