AWS sagemaker endpoints dont accept concurrent calls?

Question

I have a pytorch model deployed on p3.2xlarge, connected to lambda and api gateway.

When multiple requests are sent to it, based on logs, it accepts them sequentially. There is no concurrency. But Shouldn't endpoints be able to handle like 200 calls concurrently?

Do we need to set something up?

Please let me know.

This is a sample concurrent call:

import threading
import requests

def send_request():
    data = {"image": encoded_image}
    response = requests.post("https://idr263lxri.something", json=data)
    print(response.text)

threads = []
for _ in range(8):
    thread = threading.Thread(target=send_request)
    thread.start()
    threads.append(thread)

for thread in threads:
    thread.join()
I tried various concurrent calling and they are handled sequentially.

I used various concurrency calls same thing.

It doesnt make sense that an endpoint only serves 1 call at a time....

Are we missing something?

Thank you

Answer

Hi,

We use Sagemaker model endpoints on a constant basis with multiple parallel queries all the time to increase our global inference throughput: it works fine.

You need to disclose more of your setup to obtain more detailled support.

In the meantime, you can read this pair of blog posts to see what's possible: https://aws.amazon.com/blogs/machine-learning/scale-foundation-model-inference-to-hundreds-of-models-with-amazon-sagemaker-part-1/

Best,

Didier

AWS sagemaker endpoints dont accept concurrent calls?

Relevant content