How can you deploy a TensorFlow model to an async endpoint in Amazon SageMaker while including an inference.py script?

1/Create a TensorFlow serving model using the Model class from sagemaker.tensorflow.serving, specifying the model_data, entry_point ('inference.py'), and source_dir ('code')

2/Create an AsyncInferenceConfig object to configure the async endpoint

3/Deploy the model to an async endpoint using the deploy() method and passing the AsyncInferenceConfig

Link: https://github.com/aws/amazon-sagemaker-examples/tree/default/%20%20%20%20%20deploy_and_monitor

To deploy a TensorFlow model to an asynchronous endpoint in Amazon SageMaker while including an inference.py script, you can follow these steps:

1. Create a model in SageMaker using the CreateModel API. You'll need to specify the location of your TensorFlow model artifacts in Amazon S3 and the Docker registry path for the TensorFlow serving container.

2. Include your inference.py script along with your model artifacts. This script should contain the logic for processing input data and generating predictions using your TensorFlow model.

3. Create an endpoint configuration using the CreateEndpointConfig API. In this configuration, specify that you want to use asynchronous inference.

4. Create the asynchronous HTTPS endpoint using the CreateEndpoint API, referencing the endpoint configuration you created in step 3.

5. Once the endpoint is created, you can send inference requests to it using the InvokeEndpointAsync API.

It's important to note that when including a custom inference script like inference.py, you may need to create a custom Docker container that includes both your TensorFlow model and the inference script. This container should be able to run your inference.py script and handle the asynchronous nature of the requests.

If you want to simplify this process, you could consider using a tool like the ezsmdeploy Python SDK. While it's not explicitly mentioned for async endpoints, it generally allows you to deploy models by passing in model files and an inference script without dealing directly with Docker containers. This could potentially streamline the process of including your inference.py script with your TensorFlow model deployment.
Sources
How to create an Asynchronous Inference Endpoint - Amazon SageMaker
Deploy machine learning models to Amazon SageMaker using the ezsmdeploy Python package and a few lines of code | AWS Open Source Blog