Does the SageMaker Inference Endpoint support SSE (Streaming Response, text/event-stream)?

Question

For Now, I use SageMaker inference endpoint to deploy LLM and use the TGI container to provide services, now I want to use the TGI stream function, but I didn't find anything about mentioning this from AWS documentation, I want to confirm whether it is supported, if so, How should it be used with SDK, if not supported, is there any workaround?

Thanks

Answer

Hi,

This post says that it was not supported as of May-June: https://discuss.huggingface.co/t/streaming-output-text-when-deploying-on-sagemaker/39611/5

Best,

Didier

Answer

Wow, good timing to ask...

**Yes**, SageMaker does now support response streaming as of just last Friday. You can find the:

- [AWS Blog post with example walkthroughs](https://aws.amazon.com/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/) (code [here on GitHub](https://github.com/aws-samples/sagemaker-hosting/tree/main/GenAI-Hosting/Large-Language-Model-Hosting/LLM-Streaming/Falcon-40b-and-7b))
- [What's new post](https://aws.amazon.com/about-aws/whats-new/2023/09/sagemaker-real-time-inference-response-streaming/) including links to the relevant API docs and developer guide pages

As mentioned in the blog, you'll probably want to use the AWS Large Model Inference (LMI) or Hugging Face Text Generation Interface (TGI) containers to take advantage of this feature... The 'vanilla' framework containers for e.g. PyTorch and Hugging Face (particularly when used in script mode with an `inference.py`, custom `input_fn`, `predict_fn`, etc) use some synchronous patterns that I'm not sure translate nicely to streaming contexts so far.

Does the SageMaker Inference Endpoint support SSE (Streaming Response, text/event-stream)?

Relevant content