By using AWS re:Post, you agree to the Terms of Use
/AWS Auto Scaling/

Questions tagged with AWS Auto Scaling

Sort by most recent
  • 1
  • 90 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Scheduled Action triggering at time specified in another action

I have a CloudFormation setup with Scheduled Actions to autoscale services based on times. There is one action that scales up to start the service, and another to scale down to turn it off. I also occasionally add an additional action to scale up if a service is needed at a different time on a particular day. I'm having an issue where my service is being scaled down instead of up when I specify this additional action. Looking at the console logs I get an event that looks like: ``` 16:00:00 -0400 Message: Successfully set min capacity to 0 and max capacity to 0 Cause: scheduled action name ScheduleScaling_action_1 was triggered ``` However the relevant part of the CloudFormation Template for the Scheduled Action with the name in the log has a different time, e.g.: ``` { "ScalableTargetAction": { "MaxCapacity": 0, "MinCapacity": 0 }, "Schedule": "cron(0 5 ? * 2-5 *)", "ScheduledActionName": "ScheduleScaling_action_1" } ``` What is odd is that the time this action is triggering matches exactly with the Schedule time for another action. E.g. ``` { "ScalableTargetAction": { "MaxCapacity": 1, "MinCapacity": 1 }, "Schedule": "cron(00 20 ? * 2-5 *)", "ScheduledActionName": "ScheduleScaling_action_2" } ``` I am using CDK to generate the CloudFormation template, which doesn't appear to allow me to specify a timezone. So my understanding is that the times here should be UTC. What could cause the scheduled action to trigger at the incorrect time like this?
1
answers
0
votes
4
views
Jacques
asked a month ago

Cloudformation - Autoscaling: how to get the summary (not average) of the metrics from all nodes?

I set my treshold to scale-up when cpu usage is 80% and scale-in when there is below 70% of usage. And the problem is that (AFAIK) for autoscaling group the average value is taken. Why its a problem? Example situation: 1. There is one node, i make 100% cpu load 2. Alarm is triggered, another instance is created 3. Now metric is divided by 2 so `(100% + 0%) / 2 = 50%` which is below 70% -> scale-in alarm is triggered and even though one node is still loaded with 100%, one node is being destroyed. Ideally for scale down i would use not average but SUMMARY of all loads on the nodes. There is `AWS::CloudWatch::Alarm/Properties/Statistic` settings with average or sum values but these are for Evaluation periods, not for ammount of factors in given dimension? https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cw-alarm.html#cfn-cloudwatch-alarms-statistic my template ``` { "AWSTemplateFormatVersion":"2010-09-09", "Description" : "Creates Autoscaling group. Used securitygroup ids and subnets ids are hardcoded.", "Parameters" : { "myprojectAmiId": { "Description": "New AMI ID which will be used to create/update autoscaling group", "Type": "AWS::EC2::Image::Id" }, "myprojectNodesDefaultQuantity":{ "Type": "Number", "MinValue" : "1" } }, "Resources" : { "myprojectLaunchTemplate":{ "Type":"AWS::EC2::LaunchTemplate", "Properties":{ "LaunchTemplateData":{ "IamInstanceProfile":{ "Arn": "arn:aws:iam::censored6:instance-profile/myproject-ec2" }, "ImageId": { "Ref":"myprojectAmiId" }, "InstanceType" : "t3a.small", "KeyName" : "my-ssh-key", "SecurityGroupIds" : [ "sg-censored", "sg-censored", "sg-censored5", "sg-censored" ] } } }, "myprojectAutoScalingGroup": { "Type":"AWS::AutoScaling::AutoScalingGroup", "UpdatePolicy" : { "AutoScalingRollingUpdate" : { "MaxBatchSize" : "1", "MinInstancesInService" : "1", "PauseTime" : "PT5M", "WaitOnResourceSignals": "true" } }, "Properties": { "MinSize":{ "Ref":"myprojectNodesDefaultQuantity" }, "MaxSize":"3", "HealthCheckGracePeriod":300, "LaunchTemplate": { "LaunchTemplateId": { "Ref":"myprojectLaunchTemplate" }, "Version":{ "Fn::GetAtt":[ "myprojectLaunchTemplate", "LatestVersionNumber" ] } }, "VPCZoneIdentifier" : [ "subnet-censored", "subnet-0censoredc" ], "TargetGroupARNs" : [ "arn:aws:elasticloadbalancing:us-west-2:censored:targetgroup/autoscaling-tests-targetgroup/censored" ], "Tags" : [ {"Key" : "Name", "Value" : "myproject-cloudformation-ascaling-tests", "PropagateAtLaunch" : true}, {"Key" : "Stack", "Value" : "dev-staging","PropagateAtLaunch" : true}, {"Key" : "CreatedBy", "Value" : "cloudformation", "PropagateAtLaunch" : true} ] } }, "myprojectScaleUpPolicy":{ "Type" : "AWS::AutoScaling::ScalingPolicy", "Properties" : { "AdjustmentType" : "ChangeInCapacity", "AutoScalingGroupName" : { "Ref" : "myprojectAutoScalingGroup" }, "Cooldown" : "60", "ScalingAdjustment" : 1 } }, "myprojectScaleDownPolicy":{ "Type" : "AWS::AutoScaling::ScalingPolicy", "Properties" : { "AdjustmentType" : "ChangeInCapacity", "AutoScalingGroupName" : { "Ref" : "myprojectAutoScalingGroup" }, "Cooldown" : "60", "ScalingAdjustment" : -1 } }, "myprojectCPUAlarmHigh": { "Type" : "AWS::CloudWatch::Alarm", "Properties" : { "AlarmActions" : [ { "Ref" : "myprojectScaleUpPolicy" } ], "AlarmDescription" : "Scale-up if CPU > 80% for 5 minutes", "ComparisonOperator" : "GreaterThanThreshold", "Dimensions" : [ { "Name": "AutoScalingGroupName", "Value": { "Ref" : "myprojectAutoScalingGroup" }} ], "EvaluationPeriods" : 2, "MetricName" : "CPUUtilization", "Namespace" : "AWS/EC2", "Period" : 30, "Statistic" : "Average", "Threshold" : 80 } }, "myprojectCPUAlarmLow": { "Type" : "AWS::CloudWatch::Alarm", "Properties" : { "AlarmActions" : [ { "Ref" : "myprojectScaleDownPolicy" } ], "AlarmDescription" : "Scale-down if CPU < 70% for 10 minutes", "ComparisonOperator" : "LessThanThreshold", "Dimensions" : [ { "Name": "AutoScalingGroupName", "Value": { "Ref" : "myprojectAutoScalingGroup" }} ], "EvaluationPeriods" : 2, "MetricName" : "CPUUtilization", "Namespace" : "AWS/EC2", "Period" : 600, "Statistic" : "Average", "Threshold" : 70 } } } } ```
0
answers
0
votes
4
views
AWS-User-4322471
asked a month ago

LoadBalancer health check fails but instance is not terminating

Hello, I have a load balancer which as you know keeps the health check for the web app/website. I have deployed nothing in my instance means no app/site so when anyone visits the Loadbalancer URL they see a 502 Bad gateway error which is fine. and also in the target group, it shows that an instance has failed the health check but the thing is that the auto-scaling group is not terminating the failed health check instance and replacing it. Below is the Cloudformation code ``` AutoScailingGroup: Type: AWS::AutoScaling::AutoScalingGroup Properties: VPCZoneIdentifier: - Fn::ImportValue: !Sub ${EnvironmentName}-PR1 - Fn::ImportValue: !Sub ${EnvironmentName}-PR2 LaunchConfigurationName: !Ref AppLaunchConfiguration MinSize: 1 MaxSize: 4 TargetGroupARNs: - Ref: WebAppTargetGroup AppLoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Properties: SecurityGroups: - Ref: ApplicationLoadBalancerSecurityGroup Subnets: - Fn::ImportValue: !Sub ${EnvironmentName}-PU1 - Fn::ImportValue: !Sub ${EnvironmentName}-PU2 Tags: - Key: Name Value: !Ref EnvironmentName Listener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: DefaultActions: - Type: forward TargetGroupArn: !Ref WebAppTargetGroup LoadBalancerArn: !Ref AppLoadBalancer Port: "80" Protocol: HTTP LoadBalancerListenerRule: Type: AWS::ElasticLoadBalancingV2::ListenerRule Properties: Actions: - Type: forward TargetGroupArn: !Ref WebAppTargetGroup Conditions: - Field: path-pattern Values: [/] ListenerArn: !Ref Listener Priority: 1 WebAppTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup Properties: HealthCheckIntervalSeconds: 10 HealthCheckPath: / HealthCheckProtocol: HTTP HealthCheckTimeoutSeconds: 8 HealthyThresholdCount: 2 Port: 80 Protocol: HTTP UnhealthyThresholdCount: 5 VpcId: Fn::ImportValue: Fn::Sub: "${EnvironmentName}-VPCID" ```
1
answers
0
votes
5
views
Ashish
asked a month ago

EKS HPA's apiVersion fails to stay at v2beta2

When I deploy my HPA's I am choosing ``apiVersion: autoscaling/v2beta2`` but kubernetes is making them autoscaling/v2beta1 For example: If I deploy this. ``` apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: surething-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: surething minReplicas: 2 maxReplicas: 4 behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 100 periodSeconds: 15 scaleUp: stabilizationWindowSeconds: 0 policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 4 periodSeconds: 15 selectPolicy: Max metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 100 ``` I will get this ``` apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: surething-hpa namespace: dispatch-dev uid: 189cee35-c000-410b-954e-c164a08809e1 resourceVersion: '404150989' creationTimestamp: '2021-04-04T17:30:48Z' labels: app: dispatch deployment: dev microservice: surething annotations:... selfLink:... status:... spec: scaleTargetRef: kind: Deployment name: surething apiVersion: apps/v1 minReplicas: 2 maxReplicas: 4 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 100 ``` All the documentation I can find on EKS and HPA's says that I should be able to use ``apiVersion: autoscaling/v2beta2``. My cluster is version 1.21 and my nodegroup is as well. When I run ``kubectl api-versions`` I can find ``autoscaling/v2beta2`` in the list. I'm at wits end on this one. Can someone tell me what I am doing wrong?
0
answers
0
votes
3
views
Kay
asked 2 months ago

how to set up autoscaling for async sagemaker endpoint?

working with an example documented here -> https://github.com/aws/amazon-sagemaker-examples/blob/main/async-inference/Async-Inference-Walkthrough.ipynb. I was able to set up the sagemaker model, config and aync endpoint via lambda, now I'm trying to re-create the stack via terraform. based on the documentation on terraform, i was able to set up the model, config and the endpoint but couldn't find how to go about setting up the auto scaling ( sample code below). is this possible? ``` client = boto3.client( "application-autoscaling") resource_id = ( "endpoint/" + endpoint_name + "/variant/" + "variant1") response = client.register_scalable_target( ServiceNamespace="sagemaker", ResourceId=resource_id, ScalableDimension="sagemaker:variant:DesiredInstanceCount", MinCapacity=0, MaxCapacity=5, ) response = client.put_scaling_policy( PolicyName="Invocations-ScalingPolicy", ServiceNamespace="sagemaker", ResourceId=resource_id, # Endpoint name ScalableDimension="sagemaker:variant:DesiredInstanceCount", PolicyType="TargetTrackingScaling", # 'StepScaling'|'TargetTrackingScaling' TargetTrackingScalingPolicyConfiguration={ "TargetValue": 5.0, SageMakerVariantInvocationsPerInstance "CustomizedMetricSpecification": { "MetricName": "ApproximateBacklogSizePerInstance", "Namespace": "AWS/SageMaker", "Dimensions": [{"Name": "EndpointName", "Value": endpoint_name}], "Statistic": "Average", }, "ScaleInCooldown": 600, .... }, ) ``` clean up ``` response = client.deregister_scalable_target( ServiceNamespace='sagemaker', ResourceId='resource_id', ScalableDimension='sagemaker:variant:DesiredInstanceCount' ) ```
1
answers
0
votes
5
views
clouduser
asked 2 months ago

Modify Auto Scaling Health Check setting for Elastic Beanstalk Enviornment

Hi I have been go through the documentation for update the "`AWS:AutoScaling:AutoSclaingGroup`" in my existing EB Environment. My current environment is setup using `Docker-compose.yml `file to refer to the default .env file. I decided to go with update environment setting through AWS CLI. But i keep getting the error as below. ``` aws elasticbeanstalk update-environment --environment-name Testenv-env --template-name v1 An error occurred (ConfigurationValidationException) when calling the UpdateEnvironment operation: Configuration validation exception: Invalid option specification (Namespace: 'aws:autoscaling:autoscalinggroup', OptionName: 'HealthCheckType'): Unknown configuration setting. ``` In my v1.json file, I have the setup in below ``` aws:autoscaling:autoscalinggroup: HealthCheckType: ELB ``` Then I researched online a bit, found ppl are discussing this is no longer working, and I checked the latest API documentation, which is conflicting from the instruction in below link. Apparent "HealthCheckType" is no longer in the Support field for EB. Then I turned to another solution mentioned in below link: [https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environmentconfig-autoscaling-healthchecktype.html](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environmentconfig-autoscaling-healthchecktype.html) It suggested that I use the autoscaling.config in the .ebextensions folder. However I found quite not clear instruction to how to use the .ebextensions folder with my current environment setup where I use docer-compose to refer to the default .env file when deploying. Can someone help in pointing to me a clear instruction. Another thought is, since the environment already exist, if I directly go to the ASG group in my environment, and update from there, would new deployment (with upgrade version) into my EB environment override this ASG change each time I do upgrade? Or the change will be honored and kept in the ASG group as long as it was not deleted.
1
answers
0
votes
4
views
AWS-User-3610377
asked 3 months ago

How do I make an ECS cluster spawn GPU instances with more root volume than default?

I need to deploy an ML app that needs GPU access for its response times to be acceptable (since it uses some heavy networks that run too slowly on CPU). The app is containerized and uses an nvidia/cuda base image, so that it can make use of its host machine's GPU. The image alone weighs ~10GB, and during startup it pulls several ML models and data which takes up about another ~10GB of disk. We were previously running this app on Elastic Beanstalk, but we realized it doesn't support GPU usage, even if specifying a Deep Learning AMI, so we migrated to ECS, which provides more configurability that the former. However, we soon ran into a new problem: **selecting a g4dn instance type when creating a cluster, which defaults the AMI to an ECS GPU one, turns the Root EBS Volume Size field into a Data EBS Volume Size field.** This causes the instance's 22GB root volume (which is the only one that comes formatted and mounted) to be too small for pulling our image and downloading the data it needs during startup. The other volume (of whatever size I specify during creation in the new Data EBS Volume Size field) is not mounted and therefore not accessible by the container. Additionally, the g4dn instances come with a 125GB SSD, that is not mounted either. If either of these were usable or it was possible to enlarge the root volume (which it is if using the default non-GPU AMI) ECS would be the perfect solution for us at this time. At the moment, we worked around this issue by creating an *empty* cluster in ECS, and the manually creating and attaching an Auto Scaling group to it, since when using a Launch configuration or template the root volume's size can be correctly specified, even if using the same exact ECS GPU AMI as ECS does. However, this is a tiresome process, and makes us lose valuable ECS functionality such as automatically spawning a new instance during a rolling update to maintain capacity. Am I missing something here? Is this a bug that will be fixed at some point? If its not, is there a simpler way to achieve what I need? Maybe by specifying a custom launch configuration to the ECS cluster or by automatically mounting the SSD on instance launch? Any help is more than appreciated. Thanks in advance!
0
answers
0
votes
4
views
Ian
asked 3 months ago

Elastic Beanstalk´s ASG cannot create EC2 instances

**The problem** I´m using Elastic Beanstalk to deploy applications alongside GitHub Actions, when the action gets activated, Beanstalk creates an ASG where the desired capacity creates at least 1 instance with the containerized application. For some reason, the ASG provided by Beanstalk started to set as unhealthy the instances almost in an immediate way and terminates them. This process repeats 5 or 6 times and then returns an error state to the beanstalk application. The ASG remains in *Provisioning state* and when I looked at the ASG activity history log I got the following: | Status | Description | Cause | | --- | --- | --- | | Cancelled | Launching a new EC2 instance. Status Reason: Instance Became unhealthy while waiting for instance to be in inService state. Termination Reason: Client.InternalError: Client error on launch | At 2022-01-13 an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1 | And the EB environment events throw 4 errors: 1. Creating Auto Scaling group named: awseb-[...]-AWSEBAutoScalingGroup-1V0R8Z9EJ8G8J failed. Reason: Group did not stabilize. {current/minSize/maxSize} group size = {0/1/1}. 2. Service:AmazonCloudFormation, Message:Stack named 'awseb-e-rvjtnttttf-immutable-stack' aborted operation. Current state: 'CREATE_FAILED' Reason: The following resource(s) failed to create: [AWSEBAutoScalingGroup]. 3. Failed to deploy application. 4. Cannot complete command execution on the following instances as they are no longer running: [i-03449eff8756123c2]. **The following steps have been taken at the moment** 1. Review of IAM role permissions to allow creating EC2 instances. 2. Review of SG and secure connection between the load balancer and target group **Identified activities before the problem** 1. Enable ELB health check with 300 grace period for two of our ASG **Personal point of view** The problem seems to be not directly with Beanstalk but between TG and the instances, maybe a VPC endpoint is needed to return health status from EC2 to TG? `Client -> LB -> TG -[HERE]> EC2`
3
answers
0
votes
6
views
Osain Abitia
asked 4 months ago

AutoScaling using an AMI created in another account

When attempting to use Autoscale to launch Instances in one account (acct2) that are using an AMI in another account (acct1) , I run into the following error (seen on the terminated instance) **State transition reason**: Server.InternalError **State transition message**: Client.InternalError: Client error on launch Is there an exact location I can put the suggested policy? which will allow me to run the 'aws kms create-grant ' as documented? I see they give the policy, but it's not 100% clear where this applies and all my attempts have failed. Keep in mind, These AMI's are successfully launched in acct1 if I do it manually. Below are the details of everything I have done ------------------------------------------------------------------- Hello, I am attempting to use an autoscaling group in account (acct2) 051878XXXXXX which is using an encrypted AMI ami-04eeff30998ec5863 which belongs to account (acct1) 535298XXXXXX I am having problems with getting the AMI to unencrypt. We are able to use the AMI in that acct2 for regular EC2 launches, just not autoscaling. The details for some of the steps taken are below. The initial and current issue is our servers end up with the following error: State transition reason Server.InternalError State transition message Client.InternalError: Client error on launch Uppon hitting that error, we search for Amazon provided documentation on what needs to be done for autoscaling https://docs.aws.amazon.com/autoscaling/ec2/userguide/key-policy-requirements-EBS-encryption.html https://aws.amazon.com/premiumsupport/knowledge-center/kms-launch-ec2-instance/ This Auto Scaling Group arn:aws:autoscaling:us-west-2:051878XXXXXX:autoScalingGroup:ed7799a5-7a00-4667-93be-44317df73960:autoScalingGroupName/prod-marvin-2021-44-2-AutoScalingGroup-LO2L18YDTGR7 Several attempts were made to get this to work 1) **The policy key for the principle** "arn:aws:sts::051878XXXXXX:assumed-role/OrganizationAccountAccessRole/awsmfa_20211104T135016" action "kms:Create*" 2) **I added to arn:aws:iam::05187XXXXXX:role/OrganizationAccountAccessRole (Since REmoved)** The policy KMSPolicyForAutoScaling "Statement": \[ { "Sid": "Test", "Effect": "Allow", "Action": "kms:CreateGrant", "Resource": "arn:aws:kms:us-west-2:535298XXXXXX:key/b9b06a39-f963-4786-a66b-df5f1aXXXXXX" } 3) **I added this policy directly to our MFA policy (Since removed)** arn:aws:iam::535298XXXXXX:policy/SwitchboardAdministratorWithMfa { "Sid": "testkms", "Effect": "Allow", "Action": "kms:CreateGrant", "Resource": "arn:aws:kms:us-west-2:535298XXXXXX:key/b9b06a39-f963-4786-a66b-df5f1ab5a97f" } 4) **Here is the attempt to grant the permission from account 051878XXXXXX** I get the failure An error occurred (AccessDeniedException) when calling the CreateGrant operation: aws kms create-grant --region us-west-2 --key-id arn:aws:kms:us-west-2:535298XXXXXX:key/b9b06a39-f963-4786-a66b-df5f1aXXXXXX --grantee-principal arn:aws:iam::051878507259:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling --operations "Encrypt" "Decrypt" "ReEncryptFrom" "ReEncryptTo" "GenerateDataKey" "GenerateDataKeyWithoutPlaintext" "DescribeKey" "CreateGrant" Here is the cloudtrail showing the failure. "eventSource": "kms.amazonaws.com", "eventName": "CreateGrant", "awsRegion": "us-west-2", "sourceIPAddress": "99.88.41.224", "userAgent": "aws-cli/2.0.48 Python/3.7.4 Darwin/20.6.0 exe/x86_64 command/kms.create-grant", "errorCode": "AccessDenied", "errorMessage": "User: arn:aws:sts::051878XXXXXX:assumed-role/OrganizationAccountAccessRole/awsmfa_20211104T142204 is not authorized to perform: kms:CreateGrant on resource: arn:aws:kms:us-west-2:535298210113:key/b9b06a39-f963-4786-a66b-df5f1aXXXXXX because no resource-based policy allows the kms:CreateGrant action", 5) **I also attempted to grant the permission** (step 4) from the root login to account 051878507259 as well but to no avail.
1
answers
0
votes
0
views
zsoltszabo
asked 6 months ago
  • 1
  • 90 / page