Is there any way to limit active requests per ELB target?

Question

I have a small deployment where 2 instances is usually enough to handle the load. However, requests are very resource intensive (CPU and RAM), with individual requests taking up to 30 seconds to complete while pegging the CPU. Occasionally I see spikes in requests that result in the user waiting way too long for the system to catch up. I want to utilize auto scaling to add instances in these situations. The problem is by the time new instances have spun up, there are already too many requests in flight, and no way for the original instances to share the work backlog.

Is there a way with the Application Load balancer to limit the number of active requests to each target, effectively moving the work queue from the instances to the ELB? That way when new instances come up they can immediately be given a bunch of work to do, and the original instances will never be overloaded.

Answer

Unfortunately, there is no direct way at the ALB level to put a hard restriction on the number of connections a target will receive.

As you mentioned, Auto Scaling is an option, use Auto Scaling Group to dynamically adjust the number of instances based on traffic. If you haven't already explored Target Tracking(dynamic) and Predictive scaling policies, I'd suggest you to see which fits bets in your use case. I've been in same situation in past and I chose predictive scaling for my workloads as I could predict some of the traffic pattern.

More details, refer this [AWS Documentation: Dynamic Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scale-based-on-demand.html) and [AWS Documentation: Predictive Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-predictive-scaling.html).

Comment here if you have additional questions, happy to help.

Abhishek

Answer

ELB doesn't directly support queuing like you're looking for.  You'd need to implement your own custom load balancer layer using something like Nginx or HAProxy to do that (and even with those, it might be complicated to setup)

I think a better option might be to setup a queue based system where you're able to make things a bit more async.

There's a bunch of StackOverflow discussions on it, but I've never implemented it, so not going to link to a specific one since I can't vouch for them.  In general, the way you set this up would depend on how much control you have over the client.  
1) If the client is just a web browser or something else you can't control, you'd send a reply back to the client synchronously saying "we received your request, check back here for results" - and let the end user check back on their own.  
2) If you control the client, you could synchronously pass back a token, and have the client know to automatically poll back periodically using that token

For the ELB/ASG side of things, you could also then split into a 2 tier setup, where there's a small frontend taking in these requests and responding to the followup polling requests from clients, but isn't actually processing the requests.  The frontend just puts the request info in an SQS (or other) queue.  The backend workers then pull jobs from that queue and you autoscale on messages visible in the queue, or on [AverageMessagesPerWorker](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-using-sqs-queue.html) (Note: You can do something similar to the approach in this doc with metric math on a single scaling policy without needing to go through all the steps of publishing a custom metric)

Is there any way to limit active requests per ELB target?

Relevant content