Hello there! I am creating a pytorch realtime inference endpoint using sagemaker. My folder structure is as following:
endpoint
|_code
|_inference.py
|_requirements.txt
|_model.pt
I am using a base docker image provided by AWS (----.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:1.8.1-gpu-py36) which includes the pytorch version I need. The problem is, the python version is 3.6, and I would need it to be 3.8 because when I add other dependencies in the requirements.txt file, for example the imageio library, the endpoint fails to launch because the library doesn't work with a python version below 3.8.
What is the best practice in this case for managing complex dependencies, where I need a custom python version, coupled with specific libraries versions? Should I create another docker image that has the libraries I need and the python version I require? If so, can I extend the base image to install python3.8, eg:
FROM ----.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:1.8.1-gpu-py36
RUN apt-get update && \
apt-get install -y software-properties-common && \
add-apt-repository -y ppa:deadsnakes/ppa && \
apt-get update && \
apt install -y python3.8
If I would then push this image to ECR and use it for inference, would it run torchserve and spin up everything needed for inference? Or do I have to add these commands to the dockerfile myself?
I'm a bit confused on how to manages this and would really appreciate some guidance! Thank you so much!
AWS OFFICIALUpdated 2 years ago
Thank you so much for your answer dear Christian! Appreciate it! I will try to extend the pre-built container and let you know. Do you know by any chance if there's any documentation / tutorial I can follow for it?
Furthermore, can I generally assume that this approach is best practice? For example, in my work I often have the task to find suitable pre-trained ML models on github and implement them in endpoints. The repos generally share a conda environment with a specific python version and a set of libraries. In order for me to use these exact same requirements of python version + libraries, I would need to extend a pre-built container as mentioned above with the right python version and its requirements. Right?
And this also removes the need to have a
requirements.txtfile alongside myinference.pyfileIt will all depends on the specific use case. Baking the extra dependencies in the container increase the side of your docker image and bigger images can take longer to be downloaded for ECR. There may be scenario where it is better to install the libraries at run-time via a
requirements.txt. Similarly forinference.pyyou might have a use case that requires you to just make changes to the model loading method (model_fn()) without the need to extend the pre-built containers. To sum it up, it will depends on the use-case.Here is an example of how to extend our PyTorch container. Also have a look at the documentation here
Hope it helps.
Sounds good, thank you so much! Appreciate your help!