Why instances attached to auto scaling restarts every day at a specific time automatically?
I have 2 auto-scaling groups (ASG) which have been configured to maintain one healthy instance. One for production API and another one for development API. Both auto-scaling groups are connected to CodePipeline and Target groups. If the instances or the target groups detect unhealthy instances, ASG will deploy new instances to match the minimum number of servers configured in ASG.
The issue here is that the CPU utilization goes above 50% at the same time every day between 6 AM and 7:30 AM UTC and because of that the ASG deploys a new server due to unhealthy status indicated by the target groups. I even disconnected the target group from the ASG for a day but still faced the same issue. Nothing unusual while checking the API logs. I need to know why the instances go unhealthy at the same time every day,
- Newest
- Most votes
- Most comments
I'd strongly recommend you to check event history of ASG.
Please follow this re:Post Knowledge Center Article step by step, you'll most likely find the cause.
Assuming you already have strict inbound rules in attached security groups, most likely this would be coming from within application, some scheduled job/load.
Additionally take a look at following two posts, which would clarify little more:
Relevant content
- Accepted Answerasked 2 months ago
- Accepted Answerasked a year ago
- asked a year ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated 9 months ago
- AWS OFFICIALUpdated 3 months ago
Were you able to find the cause?
Not yet. When running the same API application on an ec2 instance it works fine but deploying the application on ASG something gets restarted exactly around 7 AM UTC. The CPU utilization and other EBS-related metrics spike up.
can you simply create a shell script and schedule it for that time to see what are the processes running, something as below: echo "------ps-------" ps -ef echo "------top-------" top echo "------sar-------" sar
Put these commands in a script and schedule it at the interval of 2 minutes for an hour
*/5 7 * * * load.sh>load.dmp
You'll get some clue, what's going on.