Make inner queue of an async SageMaker endpoint FIFO
Hi there,
I was wondering if an inner queue of an async endpoint can be FIFO, because I send a lot of invocations to the endpoint and as the amount of requests grow so does the total processing time. It's still fast, but it'll be much better if it could process in a FIFO way so the oldest requests don't time out.
If that queue can't be changed, is there another way to make it process as a FIFO?
Beforehand, thank you
- 新しい順
- 投票が多い順
- コメントが多い順
Hi there,
The internal queue is essentially FIFO, requests that are sent to your endpoint are queued in the order they are received by the endpoint. If you only have one instance behind this endpoint then your requests are processed in that same order, if you have multiple instances then requests are distributed evenly amongst them. In this case not all requests may be processed at the same time as the time to perform inference will inevitably vary across all of these instances so there can be discrepancies in the order in which you receive your responses.
The internal queue of an Asynchronous Endpoint maintains a FIFO request ordering on a best effort basis only and we do not guarantee this to be case all the time.
If your requests are getting timed out then I suggest that you increase your InvocationTimeoutSeconds (refer to the documentation here) to prevent this from happening, however if you have already set this parameter to its maximum value of 3600 seconds then you should consider adding an Autoscaling policy that monitors a metric such as ApproximateBacklogSizePerInstance to scale up the total number of instances you have. Having more instances would mean your requests are processed much faster and would reduce the likelihood of timeout.
