Is it possible to compile a neuron model in my local machine?


I'm following some guides and from my understanding this should be possible. But I've been trying for hours to compile a yolov5 model into a neuron model with no success. Is it even possible to do this in my local machine or do I have to be in an inferentia instance?

This is what my environment looks like:

# packages in environment at /miniconda3/envs/neuron:
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
absl-py                   1.2.0                    pypi_0    pypi
astor                     0.8.1                    pypi_0    pypi
attrs                     22.1.0                   pypi_0    pypi
backcall                  0.2.0              pyhd3eb1b0_0  
ca-certificates           2022.07.19           h06a4308_0  
cachetools                5.2.0                    pypi_0    pypi
certifi                   2022.9.24                pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
debugpy                   1.5.1            py37h295c915_0  
decorator                 5.1.1              pyhd3eb1b0_0  
dmlc-nnvm                      pypi_0    pypi
dmlc-topi                      pypi_0    pypi
dmlc-tvm                       pypi_0    pypi
entrypoints               0.4              py37h06a4308_0  
fonttools                 4.37.3                   pypi_0    pypi
gast                      0.2.2                    pypi_0    pypi
google-auth               2.12.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
gputil                    1.4.0                    pypi_0    pypi
grpcio                    1.49.1                   pypi_0    pypi
h5py                      3.7.0                    pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-metadata        4.12.0                   pypi_0    pypi
inferentia-hwm                 pypi_0    pypi
iniconfig                 1.1.1                    pypi_0    pypi
ipykernel                 6.15.2           py37h06a4308_0  
ipython                   7.34.0                   pypi_0    pypi
ipywidgets                8.0.2                    pypi_0    pypi
islpy                     2021.1+aws2021.x.16.0.bld0          pypi_0    pypi
jedi                      0.18.1           py37h06a4308_1  
jupyter_client            7.3.5            py37h06a4308_0  
jupyter_core              4.10.0           py37h06a4308_0  
jupyterlab-widgets        3.0.3                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libsodium                 1.0.18               h7b6447c_0  
libstdcxx-ng              11.2.0               h1234567_1  
llvmlite                  0.39.1                   pypi_0    pypi
markdown                  3.4.1                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
matplotlib                3.5.3                    pypi_0    pypi
matplotlib-inline         0.1.6            py37h06a4308_0  
ncurses                   6.3                  h5eee18b_3  
nest-asyncio              1.5.5            py37h06a4308_0  
networkx                  2.4                      pypi_0    pypi
neuron-cc                 pypi_0    pypi
numba                     0.56.2                   pypi_0    pypi
numpy                     1.19.5                   pypi_0    pypi
oauthlib                  3.2.1                    pypi_0    pypi
opencv-python                    pypi_0    pypi
openssl                   1.1.1q               h7f8727e_0  
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3               pyhd3eb1b0_0  
pandas                    1.3.5                    pypi_0    pypi
parso                     0.8.3              pyhd3eb1b0_0  
pexpect                   4.8.0              pyhd3eb1b0_3  
pickleshare               0.7.5           pyhd3eb1b0_1003  
pillow                    9.2.0                    pypi_0    pypi
pip                       22.2.2                   pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
prompt-toolkit            3.0.31                   pypi_0    pypi
protobuf                  3.20.3                   pypi_0    pypi
psutil                    5.9.2                    pypi_0    pypi
ptyprocess                0.7.0              pyhd3eb1b0_2  
py                        1.11.0                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pygments                  2.13.0                   pypi_0    pypi
pyparsing                 3.0.9            py37h06a4308_0  
pytest                    7.1.3                    pypi_0    pypi
python                    3.7.13               h12debd9_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
pytz                      2022.2.1                 pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     23.2.0           py37h6a678d5_0  
readline                  8.1.2                h7f8727e_1  
requests                  2.28.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scipy                     1.4.1                    pypi_0    pypi
seaborn                   0.12.0                   pypi_0    pypi
setuptools                59.8.0                   pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.39.3               h5082296_0  
tensorboard               1.15.0                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                1.15.0                   pypi_0    pypi
tensorflow-estimator      1.15.1                   pypi_0    pypi
termcolor                 2.0.1                    pypi_0    pypi
thop                      0.1.1-2209072238          pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
tomli                     2.0.1                    pypi_0    pypi
torch                     1.11.0                   pypi_0    pypi
torch-neuron               pypi_0    pypi
torchvision               0.12.0                   pypi_0    pypi
tornado                   6.2              py37h5eee18b_0  
tqdm                      4.64.1                   pypi_0    pypi
traitlets                 5.4.0                    pypi_0    pypi
typing-extensions         4.3.0                    pypi_0    pypi
urllib3                   1.26.12                  pypi_0    pypi
wcwidth                   0.2.5              pyhd3eb1b0_0  
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.37.1                   pypi_0    pypi
widgetsnbextension        4.0.3                    pypi_0    pypi
wrapt                     1.14.1                   pypi_0    pypi
xz                        5.2.6                h5eee18b_0  
zeromq                    4.3.4                h2531618_0  
zipp                      3.8.1                    pypi_0    pypi
zlib                      1.2.12               h5eee18b_3  

asked 2 months ago54 views
1 Answer

It is possible to compile a model to inferentia anywhere. You don't need an inf1 to compile it. Inf1 is only required to run the model later. To compile a Yolov5 model, just make sure you have the correct versions of the libraries/frameworks. The following combination works well:

I tested it using Python 3.6 and 3.7 on an x86_64 instance (CPU only). First prepare the env:

pip config set global.extra-index-url
pip install -r  # install dependencies
pip install -U --force-reinstall torch-neuron== neuron-cc[tensorflow] "protobuf<4" torchvision==0.11.2

Then compile the model:

import torch
import torch_neuron

assert(model_type in ['n', 's', 'm', 'l', 'x'])

x = torch.rand([1, 3, 640, 640], dtype=torch.float32)
model = torch.hub.load('ultralytics/yolov5', f'yolov5{model_type}', pretrained=True)

y = model(x) # warmup

torch.neuron.analyze_model(model, example_inputs=x)
model_neuron = torch.neuron.trace( model, example_inputs=x)

## Export to saved model"")

Then, copy the model to an inf1 instance, prepared with the same libraries versions you compiled the model and run:

import torch
import torch.neuron

model = torch.load('')
x = torch.rand([1, 3, 640, 640], dtype=torch.float32)
y = model(x)


You need to post process the predictions externally, to extract the bounding boxes. You can use the following sample to see how to run pre and post processing routines.

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions