- Newest
- Most votes
- Most comments
Hello,
The Desired capacity value of your ASG can change for several reasons. For example, if you’ve configured auto-scaling based on metrics like CPU or memory usage, AWS will adjust the Desired capacity accordingly.
To make sure you always have 6 tasks running, you should set the Min desired capacity field of your ASG to 6.
Here is an AWS doc explaining scalability.
Hope this helps! 😊
This ECS Service doesn't have an ASG, and I haven't configured auto-scaling for it. Is this still expected behavior?
Do I need to use Service Auto Scaling to prevent the desired tasks from being set to 0?
Did you try to modify the Service auto scaling section by setting the Minimum number of tasks field to 6? You can set this field by updating your ECS service
The behavior you're experiencing with your ECS Service using Fargate Spot capacity is not typical and doesn't align with the expected functionality of Fargate Spot. While it's normal for Fargate Spot tasks to be interrupted due to AWS reclaiming capacity, the Desired Tasks count should not automatically change to 0.
Fargate Spot is designed to run interruption-tolerant tasks at a discounted rate compared to regular Fargate pricing. When AWS needs the capacity back, tasks running on Fargate Spot can be interrupted with a two-minute warning. However, this interruption should not affect the desired task count of your service.
In a normal scenario:
- When Fargate Spot capacity becomes unavailable, tasks may be stopped.
- The ECS service scheduler should detect this and attempt to launch new tasks to maintain the desired count.
- If capacity is not immediately available, the service will continue to retry launching tasks until the required capacity becomes available.
The fact that your Desired Tasks count is being set to 0 suggests that there might be an issue with your service configuration or an unexpected behavior in the ECS service scheduler.
To address this issue:
- Check your service configuration to ensure there are no auto-scaling policies or other automation that might be modifying the desired count.
- Review your CloudTrail logs to see if there are any API calls modifying the desired count of your service.
- Consider using a mix of Fargate and Fargate Spot capacity providers in your service's capacity provider strategy. This can help maintain some level of availability even when Spot capacity is scarce.
- Implement monitoring and alerting for your service's desired and running task counts.
- If the issue persists, it would be advisable to contact AWS support, as this behavior is not expected and may indicate a deeper problem.
Remember, while Fargate Spot can offer significant cost savings, it's important to design your applications to be resilient to interruptions and to have proper monitoring in place to detect and respond to unexpected changes in your service's state.
Sources
Amazon ECS clusters for the Fargate launch type - Amazon Elastic Container Service
AWS Fargate for Amazon ECS - Amazon Elastic Container Service
Using Amazon EC2 Spot and FARGATE_SPOT - Amazon Elastic Container Service
Definitely 1 & 2 on the last section. I'd assume there's either an AutoScaling policy or some automation setting the Desired to 0 on the ECS Service (I assume these aren't standalone tasks and there's a Service here, since you mentioned a Desired?)
@Shahad_C yes there is an ECS Service with Desired Tasks set to 6. There is no auto scaling policy configured for this ECS Service. I don't have any automation set up to change the desired tasks either.
Relevant content
- asked a year ago
- asked a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated a year ago
I figured out the issue -- it was related to an automated deploy I wasn't aware of.
Nice! In general, I suggest always checking Cloud Trail when something unexpected happens. It logs most API calls in your account, and can be invaluable to troubleshoot issues where an unknown change happened. In this case, you would google "AWS API call to change ECS Desired", and it should return the results "UpdateService" - you would then search for the EventName=UpdateService to see where those API calls came from (and presumably it would show your automation pipeline)