Questions tagged with AWS Auto Scaling

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Scheduled Action triggering at time specified in another action

I have a CloudFormation setup with Scheduled Actions to autoscale services based on times. There is one action that scales up to start the service, and another to scale down to turn it off. I also occasionally add an additional action to scale up if a service is needed at a different time on a particular day. I'm having an issue where my service is being scaled down instead of up when I specify this additional action. Looking at the console logs I get an event that looks like: ``` 16:00:00 -0400 Message: Successfully set min capacity to 0 and max capacity to 0 Cause: scheduled action name ScheduleScaling_action_1 was triggered ``` However the relevant part of the CloudFormation Template for the Scheduled Action with the name in the log has a different time, e.g.: ``` { "ScalableTargetAction": { "MaxCapacity": 0, "MinCapacity": 0 }, "Schedule": "cron(0 5 ? * 2-5 *)", "ScheduledActionName": "ScheduleScaling_action_1" } ``` What is odd is that the time this action is triggering matches exactly with the Schedule time for another action. E.g. ``` { "ScalableTargetAction": { "MaxCapacity": 1, "MinCapacity": 1 }, "Schedule": "cron(00 20 ? * 2-5 *)", "ScheduledActionName": "ScheduleScaling_action_2" } ``` I am using CDK to generate the CloudFormation template, which doesn't appear to allow me to specify a timezone. So my understanding is that the times here should be UTC. What could cause the scheduled action to trigger at the incorrect time like this?
1
answers
0
votes
56
views
Jacques
asked 8 months ago

Cloudformation - Autoscaling: how to get the summary (not average) of the metrics from all nodes?

I set my treshold to scale-up when cpu usage is 80% and scale-in when there is below 70% of usage. And the problem is that (AFAIK) for autoscaling group the average value is taken. Why its a problem? Example situation: 1. There is one node, i make 100% cpu load 2. Alarm is triggered, another instance is created 3. Now metric is divided by 2 so `(100% + 0%) / 2 = 50%` which is below 70% -> scale-in alarm is triggered and even though one node is still loaded with 100%, one node is being destroyed. Ideally for scale down i would use not average but SUMMARY of all loads on the nodes. There is `AWS::CloudWatch::Alarm/Properties/Statistic` settings with average or sum values but these are for Evaluation periods, not for ammount of factors in given dimension? https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cw-alarm.html#cfn-cloudwatch-alarms-statistic my template ``` { "AWSTemplateFormatVersion":"2010-09-09", "Description" : "Creates Autoscaling group. Used securitygroup ids and subnets ids are hardcoded.", "Parameters" : { "myprojectAmiId": { "Description": "New AMI ID which will be used to create/update autoscaling group", "Type": "AWS::EC2::Image::Id" }, "myprojectNodesDefaultQuantity":{ "Type": "Number", "MinValue" : "1" } }, "Resources" : { "myprojectLaunchTemplate":{ "Type":"AWS::EC2::LaunchTemplate", "Properties":{ "LaunchTemplateData":{ "IamInstanceProfile":{ "Arn": "arn:aws:iam::censored6:instance-profile/myproject-ec2" }, "ImageId": { "Ref":"myprojectAmiId" }, "InstanceType" : "t3a.small", "KeyName" : "my-ssh-key", "SecurityGroupIds" : [ "sg-censored", "sg-censored", "sg-censored5", "sg-censored" ] } } }, "myprojectAutoScalingGroup": { "Type":"AWS::AutoScaling::AutoScalingGroup", "UpdatePolicy" : { "AutoScalingRollingUpdate" : { "MaxBatchSize" : "1", "MinInstancesInService" : "1", "PauseTime" : "PT5M", "WaitOnResourceSignals": "true" } }, "Properties": { "MinSize":{ "Ref":"myprojectNodesDefaultQuantity" }, "MaxSize":"3", "HealthCheckGracePeriod":300, "LaunchTemplate": { "LaunchTemplateId": { "Ref":"myprojectLaunchTemplate" }, "Version":{ "Fn::GetAtt":[ "myprojectLaunchTemplate", "LatestVersionNumber" ] } }, "VPCZoneIdentifier" : [ "subnet-censored", "subnet-0censoredc" ], "TargetGroupARNs" : [ "arn:aws:elasticloadbalancing:us-west-2:censored:targetgroup/autoscaling-tests-targetgroup/censored" ], "Tags" : [ {"Key" : "Name", "Value" : "myproject-cloudformation-ascaling-tests", "PropagateAtLaunch" : true}, {"Key" : "Stack", "Value" : "dev-staging","PropagateAtLaunch" : true}, {"Key" : "CreatedBy", "Value" : "cloudformation", "PropagateAtLaunch" : true} ] } }, "myprojectScaleUpPolicy":{ "Type" : "AWS::AutoScaling::ScalingPolicy", "Properties" : { "AdjustmentType" : "ChangeInCapacity", "AutoScalingGroupName" : { "Ref" : "myprojectAutoScalingGroup" }, "Cooldown" : "60", "ScalingAdjustment" : 1 } }, "myprojectScaleDownPolicy":{ "Type" : "AWS::AutoScaling::ScalingPolicy", "Properties" : { "AdjustmentType" : "ChangeInCapacity", "AutoScalingGroupName" : { "Ref" : "myprojectAutoScalingGroup" }, "Cooldown" : "60", "ScalingAdjustment" : -1 } }, "myprojectCPUAlarmHigh": { "Type" : "AWS::CloudWatch::Alarm", "Properties" : { "AlarmActions" : [ { "Ref" : "myprojectScaleUpPolicy" } ], "AlarmDescription" : "Scale-up if CPU > 80% for 5 minutes", "ComparisonOperator" : "GreaterThanThreshold", "Dimensions" : [ { "Name": "AutoScalingGroupName", "Value": { "Ref" : "myprojectAutoScalingGroup" }} ], "EvaluationPeriods" : 2, "MetricName" : "CPUUtilization", "Namespace" : "AWS/EC2", "Period" : 30, "Statistic" : "Average", "Threshold" : 80 } }, "myprojectCPUAlarmLow": { "Type" : "AWS::CloudWatch::Alarm", "Properties" : { "AlarmActions" : [ { "Ref" : "myprojectScaleDownPolicy" } ], "AlarmDescription" : "Scale-down if CPU < 70% for 10 minutes", "ComparisonOperator" : "LessThanThreshold", "Dimensions" : [ { "Name": "AutoScalingGroupName", "Value": { "Ref" : "myprojectAutoScalingGroup" }} ], "EvaluationPeriods" : 2, "MetricName" : "CPUUtilization", "Namespace" : "AWS/EC2", "Period" : 600, "Statistic" : "Average", "Threshold" : 70 } } } } ```
0
answers
0
votes
22
views
asked 8 months ago

LoadBalancer health check fails but instance is not terminating

Hello, I have a load balancer which as you know keeps the health check for the web app/website. I have deployed nothing in my instance means no app/site so when anyone visits the Loadbalancer URL they see a 502 Bad gateway error which is fine. and also in the target group, it shows that an instance has failed the health check but the thing is that the auto-scaling group is not terminating the failed health check instance and replacing it. Below is the Cloudformation code ``` AutoScailingGroup: Type: AWS::AutoScaling::AutoScalingGroup Properties: VPCZoneIdentifier: - Fn::ImportValue: !Sub ${EnvironmentName}-PR1 - Fn::ImportValue: !Sub ${EnvironmentName}-PR2 LaunchConfigurationName: !Ref AppLaunchConfiguration MinSize: 1 MaxSize: 4 TargetGroupARNs: - Ref: WebAppTargetGroup AppLoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Properties: SecurityGroups: - Ref: ApplicationLoadBalancerSecurityGroup Subnets: - Fn::ImportValue: !Sub ${EnvironmentName}-PU1 - Fn::ImportValue: !Sub ${EnvironmentName}-PU2 Tags: - Key: Name Value: !Ref EnvironmentName Listener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: DefaultActions: - Type: forward TargetGroupArn: !Ref WebAppTargetGroup LoadBalancerArn: !Ref AppLoadBalancer Port: "80" Protocol: HTTP LoadBalancerListenerRule: Type: AWS::ElasticLoadBalancingV2::ListenerRule Properties: Actions: - Type: forward TargetGroupArn: !Ref WebAppTargetGroup Conditions: - Field: path-pattern Values: [/] ListenerArn: !Ref Listener Priority: 1 WebAppTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup Properties: HealthCheckIntervalSeconds: 10 HealthCheckPath: / HealthCheckProtocol: HTTP HealthCheckTimeoutSeconds: 8 HealthyThresholdCount: 2 Port: 80 Protocol: HTTP UnhealthyThresholdCount: 5 VpcId: Fn::ImportValue: Fn::Sub: "${EnvironmentName}-VPCID" ```
1
answers
0
votes
434
views
Ashish
asked 8 months ago

EKS HPA's apiVersion fails to stay at v2beta2

When I deploy my HPA's I am choosing ``apiVersion: autoscaling/v2beta2`` but kubernetes is making them autoscaling/v2beta1 For example: If I deploy this. ``` apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: surething-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: surething minReplicas: 2 maxReplicas: 4 behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 100 periodSeconds: 15 scaleUp: stabilizationWindowSeconds: 0 policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 4 periodSeconds: 15 selectPolicy: Max metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 100 ``` I will get this ``` apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: surething-hpa namespace: dispatch-dev uid: 189cee35-c000-410b-954e-c164a08809e1 resourceVersion: '404150989' creationTimestamp: '2021-04-04T17:30:48Z' labels: app: dispatch deployment: dev microservice: surething annotations:... selfLink:... status:... spec: scaleTargetRef: kind: Deployment name: surething apiVersion: apps/v1 minReplicas: 2 maxReplicas: 4 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 100 ``` All the documentation I can find on EKS and HPA's says that I should be able to use ``apiVersion: autoscaling/v2beta2``. My cluster is version 1.21 and my nodegroup is as well. When I run ``kubectl api-versions`` I can find ``autoscaling/v2beta2`` in the list. I'm at wits end on this one. Can someone tell me what I am doing wrong?
0
answers
0
votes
11
views
Kay
asked 9 months ago

how to set up autoscaling for async sagemaker endpoint?

working with an example documented here -> https://github.com/aws/amazon-sagemaker-examples/blob/main/async-inference/Async-Inference-Walkthrough.ipynb. I was able to set up the sagemaker model, config and aync endpoint via lambda, now I'm trying to re-create the stack via terraform. based on the documentation on terraform, i was able to set up the model, config and the endpoint but couldn't find how to go about setting up the auto scaling ( sample code below). is this possible? ``` client = boto3.client( "application-autoscaling") resource_id = ( "endpoint/" + endpoint_name + "/variant/" + "variant1") response = client.register_scalable_target( ServiceNamespace="sagemaker", ResourceId=resource_id, ScalableDimension="sagemaker:variant:DesiredInstanceCount", MinCapacity=0, MaxCapacity=5, ) response = client.put_scaling_policy( PolicyName="Invocations-ScalingPolicy", ServiceNamespace="sagemaker", ResourceId=resource_id, # Endpoint name ScalableDimension="sagemaker:variant:DesiredInstanceCount", PolicyType="TargetTrackingScaling", # 'StepScaling'|'TargetTrackingScaling' TargetTrackingScalingPolicyConfiguration={ "TargetValue": 5.0, SageMakerVariantInvocationsPerInstance "CustomizedMetricSpecification": { "MetricName": "ApproximateBacklogSizePerInstance", "Namespace": "AWS/SageMaker", "Dimensions": [{"Name": "EndpointName", "Value": endpoint_name}], "Statistic": "Average", }, "ScaleInCooldown": 600, .... }, ) ``` clean up ``` response = client.deregister_scalable_target( ServiceNamespace='sagemaker', ResourceId='resource_id', ScalableDimension='sagemaker:variant:DesiredInstanceCount' ) ```
1
answers
0
votes
264
views
asked 9 months ago