Custom Sagemaker Batch "Request Splitter"


Batch Transform has a number of pre-defined "SplitType" / "BatchStrategy" options that allow certain file-types to be automatically split into batches that are < 64MB. These include CSV and JSONL but not JSON.

I'm using Triton with Batch Transform, it's possible to automate the splitting/reassembling of Triton (if you assume the first dimension of each tensor is the batch dimension) but this requires custom code. Is this possible with batch transform? Can I provide my own transform?

질문됨 10달 전331회 조회
2개 답변
수락된 답변

Thanks, I don't think those links are relevant to Triton - there's already an AWS Triton container. What I ended up doing that worked fine was to create a Triton pipeline using a Python backend step that performed the batching and a onnx backend for the model.

답변함 9달 전

The input to batch transforms must be of a format that can be split into smaller files to process in parallel. These formats include CSV, JSON, JSON Lines, TFRecord and RecordIO.

The SplitType parameter indicates how to split the records in the input dataset. To split input files into mini-batches when you create a batch transform job, set the SplitType parameter value to Line. If SplitType is set to None or if an input file can't be split into mini-batches, SageMaker uses the entire input file in a single request. You can control the size of the mini-batches by using the BatchStrategy and MaxPayloadInMB parameters. MaxPayloadInMB must not be greater than 100 MB.

In your use case where you are using Triton with Batch Transform and want to automate the splitting/reassembling of Triton by assuming the first dimension of each tensor is the batch dimension, In order to achieve the same you can Use Your Own Inference Code with Batch Transform and implement the same. You can refer below links for steps and example to bring your own code with batch transform.

[1] [2]

[3] [4]

답변함 9달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠