Skip to content

Autoscaler Actions by "user request"

0

We have an autoscaler which scales in and out based on policies such as NetworkIn, Healthy Host, etc. We also have some scheduled actions set up. All these actions are properly logged and can be identified in the Actions Log.

However, I keep seeing Scale-In and Scale-Out actions logged "in response to a user request" (see below example). What can cause these and how can I pinpoint them? I promise none of us is manually changing desired or minimum settings)

At 2024-07-10T23:51:48Z instance i-0c705997aaa028d42 was taken out of service in response to a user request, shrinking the capacity from 5 to 4.

3 Answers
1

This is most often from the TerminateInstanceInAutoScalingGroup API. Its commonly used by some services (such as EKS, Batch, or CloudFormation), as well as some 3rd party applications which are managing instances in an ASG. As the other replies recommended, go to CloudTrail and search around the start time of one of these activities by either: ReadOnly=False or EventSource=autoscaling.amazonaws.com

You'll want to look at the UserAgent of the call in cloudtrail to see where its coming from

EXPERT
answered 2 years ago
  • thank you, I have activated cloudtrail now. We found some issues with our load balancer health checks which target this autoscaler group, could it be that the load balancer does these "user requests" to make sure we have healthy instances in the autoscaler?

  • No, failed ELB healthchecks would have an explicit message in the Activity History. Its possible some other system is noticing the failed ELB healthchecks and sending these termination requests to AutoScaling, but the ELB itself won't do that

    You should be able to see 90 days of past CloudTrail events by default, even without a Trail setup to store them externally. You can see these past events on the CloudTrail console, or via the lookup-events API

1

Hello.

If the change was made by a user, I think you could check which IAM user changed the EC2 capacity by checking the CloudTrail event history.
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-console.html

EXPERT
answered 2 years ago
EXPERT
reviewed 2 years ago
EXPERT
reviewed 2 years ago
0

Hii

Check these troubleshoot steps to resolve issue:

  • Identify Specific Actions: Look for entries in the Actions Log with "in response to a user request" next to the scaling action (scale-in or scale-out).
  • Review User Activity: Check if any users made scaling requests through the autoscaler's user interface or API during the logged times..

Consider Integrations:

  • Do you have any integrations with other tools that might trigger scaling actions?
  • Investigate if these integrations can initiate scaling and might be misidentified as user requests.
  • Check Scheduled Actions: Double-check your scheduled scaling actions to ensure they aren't causing these events

.

EXPERT
answered 2 years ago
EXPERT
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.