Questions tagged with Spot Instances

Hi Dears, I am building ML model using DeepAR Algorithm. I faced this error while i reached to this point : Error : ClientError: An error occurred (UnknownOperationException) when calling the CreateHyperParameterTuningJob operation: The requested operation is not supported in the called region. ------------------- Code: from sagemaker.tuner import ( IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner, ) from sagemaker import image_uris container = image_uris.retrieve(region= 'af-south-1', framework="forecasting-deepar") deepar = sagemaker.estimator.Estimator( container, role, instance_count=1, instance_type="ml.m5.2xlarge", use_spot_instances=True, # use spot instances max_run=1800, # max training time in seconds max_wait=1800, # seconds to wait for spot instance output_path="s3://{}/{}".format(bucket, output_path), sagemaker_session=sess, ) freq = "D" context_length = 300 deepar.set_hyperparameters( time_freq=freq, context_length=str(context_length), prediction_length=str(prediction_length) ) Can you please help in solving the error? I have to do that in af-south-1 region. Thanks Basem hyperparameter_ranges = { "mini_batch_size": IntegerParameter(100, 400), "epochs": IntegerParameter(200, 400), "num_cells": IntegerParameter(30, 100), "likelihood": CategoricalParameter(["negative-binomial", "student-T"]), "learning_rate": ContinuousParameter(0.0001, 0.1), } objective_metric_name = "test:RMSE" tuner = HyperparameterTuner( deepar, objective_metric_name, hyperparameter_ranges, max_jobs=10, strategy="Bayesian", objective_type="Minimize", max_parallel_jobs=10, early_stopping_type="Auto", ) s3_input_train = sagemaker.inputs.TrainingInput( s3_data="s3://{}/{}/train/".format(bucket, prefix), content_type="json" ) s3_input_test = sagemaker.inputs.TrainingInput( s3_data="s3://{}/{}/test/".format(bucket, prefix), content_type="json" ) tuner.fit({"train": s3_input_train, "test": s3_input_test}, include_cls_metadata=False) tuner.wait()

Hi, I am working on Hugging Face example in Sagemaker labs, and I got the below error "ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateTrainingJob operation: The account-level service limit 'ml.p3.2xlarge for training job usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please contact AWS support to request an increase for this limit." Code: Can you please enable 1 instance to help me continue with Sagemaker Lab example of Hugging face model. Thank you

I do have an EC2 instance with a gitlab-manager runinc (it was working for the past year until this morning). The gitlab-manager is configuerd to always keep 1 machine running on a spot instance. `` "amazonec2-request-spot-instance=true",`` This morning the Instances created from the manager aint working, It starts the instance but it stays with the "Status Check" Initializing. After some time the instance is shuted-down and a new request to create a new instance is made. I tryed to debug the gitlab-manager but it seems that the problem is happening during the instance inicialization on AWS. What can I do? How can I debug the reason for the instance to not initialize?

We run a number of EC2/Ubuntu 20.04 instances in our VPC, as spot instances. All but one are fine, and the last keeps rebooting on an irregular basis. It looks (from journalctl) as though it's doing a controlled (rather than kp) shutdown and then reboots. We simply cannot work out why this is happening, when other instances have no such problems. Is there a particular parameter I should be looking at?

looking at few examples, for training in sagemaker . are there some guidelines based on the model size, data to be trained , what type of instance cpu/gpu to use? also, can one use spot instances ( may be with multiple gpu cores)?

Is it possible to deploy spot inf1 instances on sagemaker? We run an API 24/7, and it's costly to keep it up, considering we only have 2 hours of peak performance a day. We don't shut off those machines because we might have random bursts of traffic during the day that CPU instances can't hold. Alternatively, we could deploy spot EC2 inf machines; however, I'm unsure how I would invoke them from gateway and lambda. Does anybody have a tip or recommendation for our case? Thanks!

How can we make sure what is the spot instance price boundaries as preflight? specifically the min price

I have a question about my instances. I did switch off some of them because I didn't use. But now I cannot enter them, as there is a notification "You have been disconnected because another connection was made to the remote PC". Is it because I switched it off? But I read that it should not affect the business. What it may be? Thank you in advance

I am currently using an EC2 instance (t3.xlarge) with on-demand pricing. How can I switch the instance to spot pricing, without creating a new instance?

I would like to resize an EC2 spot instance from c5.large to c5.xlarge. Has AWS added support for this yet?

I have a launch template that specifies `t3.medium` as the instance to launch via SPOT. I want to override this in ASG configuration. To do that I edit ASG config and do the following: * Under instance type requirements, I choose "override launch template" * I select 2 instance types that I want: `t3.medium` and `t3a.medium` * Under instance purchase options, I choose 100% Spot, 0% On-Demand and disable "On Demand base capacity" option * Under "Allocation strategies" choose "Lowest price" When I click on `Update` I get the error message: ``` Incompatible launch template: You cannot use a launch template that is set to request Spot Instances (InstanceMarketOptions) when you configure an Auto Scaling group with a mixed instances policy. Add a different launch template to the group and try again. ``` Is this a bug or am I missing something?

Hello, I created an ami on from an ubuntu 18.04 instance. AMI was generated without any errors. However, when I launch a new instance from the ami, instance reachability check fails and I unable to ssh into the instance. After looking at the logs, I got the following error message: ``` ... [FAILED] Failed to start Service for snap ap…amazon-ssm-agent.amazon-ssm-agent. See 'systemctl status snap.amazon-ssm-a….amazon-ssm-agent.service' for details. Failed to start Service for snap ap…amazon-ssm-agent.amazon-ssm-agent. ``` I cannot the command as the instance is not accessible by ssh. Can anyone please help? Thank you.

My EC2 launch template doesn't work when using it to launch a Spot instance. The launch template is set to launch a c5.xlarge instance **associated to a pre-existing Elastic Network Interface** @ index 0. When launching a spot instance, I receive the following cryptic message, and the spot request fails: > c5.xlarge, ami-b2b55cd5, Linux/UNIX: A network interface may not specify both a network interface ID and a subnet First off, how can a **network interface** specify a network interface id? I believe this error means to say "a spot instance may not specify both a network interface ID and a subnet", but I can't be sure. Secondly, my launch template *doesn't* specify a subnet directly - it only specifies a network interface ID, which in turn specifies the subnet. As a troubleshooting step, I've tried launching an on-demand EC2 instance directly using the same launch template, via "**Launch Templates -> Actions -> Launch Instance from Template**" - when I do this, the EC2 instance launches successfully. I've been able to reproduce this error consistently for over 9 months now, and am surprised that no one else has brought this up. What gives? Here is my Spot config: ``` "MySpotFleet" : { "Type" : "AWS::EC2::SpotFleet", "Properties" : { "SpotFleetRequestConfigData" : { "AllocationStrategy" : "lowestPrice", "IamFleetRole" : {"Fn::GetAtt" : ["MyIAMFleetRole", "Arn"]}, "InstanceInterruptionBehavior" : "stop", "LaunchTemplateConfigs": [ { "LaunchTemplateSpecification": { "LaunchTemplateId": { "Ref" : "MyLaunchTemplate" }, "Version": { "Fn::GetAtt" : [ "MyLaunchTemplate", "LatestVersionNumber" ]} } } ], "ReplaceUnhealthyInstances" : false, "SpotMaxTotalPrice" : "5.01", "SpotPrice" : "5.01", "TargetCapacity" : 1, "TerminateInstancesWithExpiration" : false, "Type" : "maintain", "ValidFrom" : "2021-01-01T00:00:00Z", "ValidUntil" : "2050-12-31T23:59:59Z" } }, "DependsOn": [ "MyLaunchTemplate" ] } ``` If I replace the above Spot config with this on-demand instance config, it works: ``` "MyInstance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "LaunchTemplate" : { "LaunchTemplateId": { "Ref" : "MyLaunchTemplate" }, "Version": { "Fn::GetAtt" : [ "MyLaunchTemplate", "LatestVersionNumber" ]} } }, "DependsOn": [ "MyLaunchTemplate" ] } ``` If it helps, here is my Launch Template: ``` "MyLaunchTemplate" : { "Type" : "AWS::EC2::LaunchTemplate", "Properties" : { "LaunchTemplateName":"MyLaunchTemplate", "LaunchTemplateData":{ "IamInstanceProfile" : { "Arn" : { "Fn::GetAtt" : ["MyEC2IAMInstanceProfile", "Arn"] } }, "ImageId" : "ami-b2b55cd5", "InstanceType": "c5.xlarge", "NetworkInterfaces" : [ { "NetworkInterfaceId" : {"Ref" : "MyENI00"}, "DeviceIndex" : "0" } ], "InstanceInitiatedShutdownBehavior" : "stop", "KeyName" : "my-keypair" } } ``` And the ENI in question: ``` "MyENI00": { "Type": "AWS::EC2::NetworkInterface", "Properties": { "Description" : "MyENI00", "GroupSet" : [ {"Ref" : "MySecurityGroup"} ], "PrivateIpAddresses": [ { "Primary" : true, "PrivateIpAddress" : "172.16.0.100" }, { "Primary" : false, "PrivateIpAddress" : "172.16.0.101" } ], "SourceDestCheck": false, "SubnetId": { "Ref" : "MySubnet" } } } ```

What is the method used to select candidate instances when manually scaling down a spot fleet? I'm managing a render fleet and would like to terminate only idle instances when scaling in the fleet. Is there the ability to choose based on Cloud Watch metrics, or the ability to insert a custom termination policy like the Auto Scaling Group has?

I'm trying to launch a spot instance with the following settings: AMI: ami-06f2f779464715dc5 Instance type: c5.18xlarge (72 vCPUs and 144 GiB of memory) Storage: root EBS with 100 GiB and added an encryption key Volume type: General Purpose SSD (gp2) As far as I can tell the above settings meet all the requirements for hibernation interruption behavior as described in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Hibernate.html#hibernating-prerequisites but still I see the message: "Selected instance type does not currently support 'hibernate' interruption behavior." Changing the instance type to be c5.9xlarge (36 vCPUs and 72 GiB of memory) with the same settings does allow me to choose hibernate as interruption behavior. Any idea what is missing? Thanks.

Customer is looking at using CloudFormation for launching a mix of On-Demand and Spot Instances for their application, and wanted to know what to put into the MarketType option to achieve that. It seems that the only acceptable value is Spot, so how would they handle doing a mix of On-Demand and Spot Instances? If they put Spot in for that value, it will only deploy Spot Instances. Is there a way to have it try Spot Instances and if the conditions aren't me, then launch On-Demand instances?

Customer is running at-scale processing of large datasets with Spot Instances. They want to know their limits for vCPUs per region and per account, however this information does not appear in Service Quota. However, "All <insert-instance-type-here> Spot Instance Requests" are available, but this limit is different from the "vCPU limit". What is the correct way to understand the limit on Spot Instances that the customer can launch?

A SaaS provider offers a service that enables customers to launch long-running jobs. These jobs are placed in containers and deployed to EC2 Spot instances using EKS. The provider does not want to terminate the jobs and restart them from scratch if the Spot instance is terminated, but instead have its state persisted and resume where it left off on the new Spot instance using hibernation. Do we have any working examples of this? The challenges I envision are related to configuring K8s to not replace the hibernated Spot instance or its containers, as we want to instead resume them. Also handling any side-effects related to have K8s processes running on the resumed EC2 instance. I suspect an external system will need to then watch for job completion and terminate the node accordingly. K8s thus would only be concerned with creating new Spot nodes and placing new jobs, without recovering/replacing failed ones.

If creating an Auto Scaling group with multiple instance types, the [documentation ](https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-purchase-options.html#asg-spot-best-practices) states: "If you don't specify a maximum price, the default maximum price is the On-Demand price, but you still receive the steep discounts provided by Spot Instances." Is the default price the On-Demand price of the instance type that will be spun up? Or is it the highest/lowest/other On-Demand price of instance types in the ASG?

Using CodeDeploy for deploying to an EC2 Auto Scaling group, CodeDeploy could disable ASG'S scaling activities by setting HANDLE_PROCS=true. However, what happens if a Spot Instance is interrupted during a deployment?

While creating a spot instance using console, it is not compulsory to choose a subnet. I can select ... Subnet: No preference (default subnet in any Availability Zone) But while creating a spot instance using cloudformation template, I need to choose a single subnet and I am not sure which one to choose for best price. Does it mean that using web interface is better than cloudformation template (for spot instances)?

Customer who is deploying a workload (stateless web services) on ECS and would like to leverage Spot Instances to reduce the cost. Customer would like an ECS cluster that has a base number of On-Demand Instances and then AutoScale based on demand by trying to add Spot Instances first and then On-Demand Instances if Spot Request fails. What is the recommended way to achieve this objective? One option I can see is to use multiple AutoScaling Groups for On-Demand & Spot Instances with alarms based on Service/Cluster Utilization. But I would also like to explore the possibility of using SpotFleet with ECS directly. But I don't see a way to make sure that On-demand instances are added to the cluster if Spot Requests fail with a SpotFleet. Is there any way to achieve this with SpotFleet?

Customer is asking if their Batch job that execute on spot is interrupted due to spot price increasing beyond their threshold, will Batch resubmit the job when the spot price falls again, or will the customer have to manually resubmit?

Has anyone ever used spot instances for HPC workload? Do we have any public reference use-case? IFAIK, Spot can be used if the job can checkpoint his status (so we can restart in case of a termination). Otherwise if the job runs for many hours, you run the risk to have your job terminated before it’s completed. Looking forward to your comments. I'll appreciate if you can share links to public use-cases or guides.