Running a request against all variants in an endpoint

Question

I have a customer asking me about the [Rendezvous architecture](https://towardsdatascience.com/rendezvous-architecture-for-data-science-in-production-79c4d48f12b). What I'm thinking is, we could implement this in a number of ways, all using endpoint variants:

- Lambda (and probably SQS) around the endpoint;
- A custom monitoring job;
- Step Functions

Without going into details of the above options or of how the evaluation and SLA check will be done, it looks like the several models would fit very well as variants of an endpoint. The thing is, the architecture expects to call them all. Is there a way to directly call all variants of a model, or will a wrapper to identify the variants, call them all and process the results be needed?

Accepted Answer

When I last looked into it, it was not possible to query all versions/variants of the model automatically. You can specify what variant to use when using the `invoke_endpoint` method. I would therefore write a lambda function to invoke each of the endpoints one-by-one (see here: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html). To be especially rigorous about it, you can add a function in your lambda code that first retrieves all the endpoint variants (see here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.describe_endpoint) then queries them one-by-one, and returns all the results.

Running a request against all variants in an endpoint

Relevant content