2 Risposte
- Più recenti
- Maggior numero di voti
- Maggior numero di commenti
1
The maximum concurrency for a single endpoint is the limit of simultaneous invocations that one particular endpoint can handle, set up to 200. The maximum total concurrency across all serverless endpoints is the sum of concurrent invocations that all endpoints combined can handle, capped at 10 for the entire account. Even though a single endpoint can handle up to 200 invocations, the overall limit of 10 invocations across the account is the overriding constraint.
0
I confirm the Service Quotas for total concurrency of Serverless Inference also dropped from 200 to 10!!!
con risposta 2 mesi fa
Contenuto pertinente
AWS UFFICIALEAggiornata 3 anni fa
Thanks Marte, in AWS document it said "You can set the maximum concurrency for a single endpoint up to 200, and the total number of serverless endpoint variants you can host in a Region is 50. The total concurrency you can share between all serverless endpoints per Region in your account is 200".
So each endpoint can handle 200 concurrent invocations. Additionally, all endpoints together can handle 200 concurrent invocations (not 10). I think AWS should update their dev document to avoid this misleading.
https://sagemaker-examples.readthedocs.io/en/latest/serverless-inference/huggingface-serverless-inference/huggingface-text-classification-serverless-inference.html#:~:text=You%20can%20set%20the%20maximum,in%20your%20account%20is%20200