Is it possible to compile a neuron model in my local machine?

2

I'm following some guides and from my understanding this should be possible. But I've been trying for hours to compile a yolov5 model into a neuron model with no success. Is it even possible to do this in my local machine or do I have to be in an inferentia instance?

This is what my environment looks like:

# packages in environment at /miniconda3/envs/neuron:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
absl-py                   1.2.0                    pypi_0    pypi
astor                     0.8.1                    pypi_0    pypi
attrs                     22.1.0                   pypi_0    pypi
backcall                  0.2.0              pyhd3eb1b0_0  
ca-certificates           2022.07.19           h06a4308_0  
cachetools                5.2.0                    pypi_0    pypi
certifi                   2022.9.24                pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
debugpy                   1.5.1            py37h295c915_0  
decorator                 5.1.1              pyhd3eb1b0_0  
dmlc-nnvm                 1.11.1.0+0               pypi_0    pypi
dmlc-topi                 1.11.1.0+0               pypi_0    pypi
dmlc-tvm                  1.11.1.0+0               pypi_0    pypi
entrypoints               0.4              py37h06a4308_0  
fonttools                 4.37.3                   pypi_0    pypi
gast                      0.2.2                    pypi_0    pypi
google-auth               2.12.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
gputil                    1.4.0                    pypi_0    pypi
grpcio                    1.49.1                   pypi_0    pypi
h5py                      3.7.0                    pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-metadata        4.12.0                   pypi_0    pypi
inferentia-hwm            1.11.0.0+0               pypi_0    pypi
iniconfig                 1.1.1                    pypi_0    pypi
ipykernel                 6.15.2           py37h06a4308_0  
ipython                   7.34.0                   pypi_0    pypi
ipywidgets                8.0.2                    pypi_0    pypi
islpy                     2021.1+aws2021.x.16.0.bld0          pypi_0    pypi
jedi                      0.18.1           py37h06a4308_1  
jupyter_client            7.3.5            py37h06a4308_0  
jupyter_core              4.10.0           py37h06a4308_0  
jupyterlab-widgets        3.0.3                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libsodium                 1.0.18               h7b6447c_0  
libstdcxx-ng              11.2.0               h1234567_1  
llvmlite                  0.39.1                   pypi_0    pypi
markdown                  3.4.1                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
matplotlib                3.5.3                    pypi_0    pypi
matplotlib-inline         0.1.6            py37h06a4308_0  
ncurses                   6.3                  h5eee18b_3  
nest-asyncio              1.5.5            py37h06a4308_0  
networkx                  2.4                      pypi_0    pypi
neuron-cc                 1.11.7.0+aec18907e          pypi_0    pypi
numba                     0.56.2                   pypi_0    pypi
numpy                     1.19.5                   pypi_0    pypi
oauthlib                  3.2.1                    pypi_0    pypi
opencv-python             4.6.0.66                 pypi_0    pypi
openssl                   1.1.1q               h7f8727e_0  
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3               pyhd3eb1b0_0  
pandas                    1.3.5                    pypi_0    pypi
parso                     0.8.3              pyhd3eb1b0_0  
pexpect                   4.8.0              pyhd3eb1b0_3  
pickleshare               0.7.5           pyhd3eb1b0_1003  
pillow                    9.2.0                    pypi_0    pypi
pip                       22.2.2                   pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
prompt-toolkit            3.0.31                   pypi_0    pypi
protobuf                  3.20.3                   pypi_0    pypi
psutil                    5.9.2                    pypi_0    pypi
ptyprocess                0.7.0              pyhd3eb1b0_2  
py                        1.11.0                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pygments                  2.13.0                   pypi_0    pypi
pyparsing                 3.0.9            py37h06a4308_0  
pytest                    7.1.3                    pypi_0    pypi
python                    3.7.13               h12debd9_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
pytz                      2022.2.1                 pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     23.2.0           py37h6a678d5_0  
readline                  8.1.2                h7f8727e_1  
requests                  2.28.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scipy                     1.4.1                    pypi_0    pypi
seaborn                   0.12.0                   pypi_0    pypi
setuptools                59.8.0                   pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.39.3               h5082296_0  
tensorboard               1.15.0                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                1.15.0                   pypi_0    pypi
tensorflow-estimator      1.15.1                   pypi_0    pypi
termcolor                 2.0.1                    pypi_0    pypi
thop                      0.1.1-2209072238          pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
tomli                     2.0.1                    pypi_0    pypi
torch                     1.11.0                   pypi_0    pypi
torch-neuron              1.11.0.2.3.0.0           pypi_0    pypi
torchvision               0.12.0                   pypi_0    pypi
tornado                   6.2              py37h5eee18b_0  
tqdm                      4.64.1                   pypi_0    pypi
traitlets                 5.4.0                    pypi_0    pypi
typing-extensions         4.3.0                    pypi_0    pypi
urllib3                   1.26.12                  pypi_0    pypi
wcwidth                   0.2.5              pyhd3eb1b0_0  
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.37.1                   pypi_0    pypi
widgetsnbextension        4.0.3                    pypi_0    pypi
wrapt                     1.14.1                   pypi_0    pypi
xz                        5.2.6                h5eee18b_0  
zeromq                    4.3.4                h2531618_0  
zipp                      3.8.1                    pypi_0    pypi
zlib                      1.2.12               h5eee18b_3  

asked 2 months ago54 views
1 Answer
2

It is possible to compile a model to inferentia anywhere. You don't need an inf1 to compile it. Inf1 is only required to run the model later. To compile a Yolov5 model, just make sure you have the correct versions of the libraries/frameworks. The following combination works well:

I tested it using Python 3.6 and 3.7 on an x86_64 instance (CPU only). First prepare the env:

pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com
pip install -r https://raw.githubusercontent.com/ultralytics/yolov5/master/requirements.txt  # install dependencies
pip install -U --force-reinstall torch-neuron==1.10.1.2.2.0.0 neuron-cc[tensorflow] "protobuf<4" torchvision==0.11.2

Then compile the model:

import torch
import torch_neuron

model_type='l'
assert(model_type in ['n', 's', 'm', 'l', 'x'])

x = torch.rand([1, 3, 640, 640], dtype=torch.float32)
model = torch.hub.load('ultralytics/yolov5', f'yolov5{model_type}', pretrained=True)
model.eval()

y = model(x) # warmup

torch.neuron.analyze_model(model, example_inputs=x)
model_neuron = torch.neuron.trace( model, example_inputs=x)

## Export to saved model
model_neuron.save("yolov5_neuron.pt")

Then, copy the model to an inf1 instance, prepared with the same libraries versions you compiled the model and run:

import torch
import torch.neuron

model = torch.load('yolov5_neuron.pt')
model.eval()
x = torch.rand([1, 3, 640, 640], dtype=torch.float32)
y = model(x)

ADDITIONAL INFORMATION

You need to post process the predictions externally, to extract the bounding boxes. You can use the following sample to see how to run pre and post processing routines. https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuron/inference/yolov5/Yolov5.ipynb

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions