Compatibility issues on DLAMI with aws-neuron-dkms

0

I have setup a DLAMI (both with Deep Learning Base AMI (Amazon Linux 2) Version 57.2 (ID: ami-07d28e42a376a75fe) or Deep Learning Base AMI (Ubuntu 18.04) Version 56.6 (ID: ami-0be3ca7b9b68fa36c)) and machine inf1.6xlarge.

Then I installed everything according to https://awsdocs-neuron.readthedocs-hosted.com/en/v1.17.0/neuron-intro/pytorch-setup/pytorch-quickstart.html and to https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuron/setup/pytorch-install.html#install-neuron-pytorch.

Now when I go into the Jupyter Notebook of this one here https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/resnet50.html it seems to work nicely in the beginning but, at the end I get an error at

model_neuron = torch.jit.load('resnet50_neuron.pt')

saying

2023-Apr-21 11:17:54.0589  8816:8816  ERROR   NRT:nrt_init                                This Neuron Runtime (compatibility id: 6) is not compatible with the installed aws-neuron-dkms package.
2023-Apr-21 11:17:54.0589  8816:8816  ERROR   NRT:nrt_init                                Installed aws-neuron-dkms supports Runtime compatibility id range: 2 - 4.
2023-Apr-21 11:17:54.0589  8816:8816  ERROR   NRT:nrt_init                                Please make sure to upgrade to latest aws-neuron-dkms; for detailed installation instructions visit Neuron documentation.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[6], line 12
      9 output_cpu = model(image)
     11 # Load the compiled Neuron model
---> 12 model_neuron = torch.jit.load('resnet50_neuron.pt')
     14 # Run inference using the Neuron model
     15 output_neuron = model_neuron(image)

File ~/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch_neuron/jit_load_wrapper.py:13, in jit_load_wrapper.<locals>.wrapper(*args, **kwargs)
     11 @wraps(func)
     12 def wrapper(*args, **kwargs):
---> 13     script_module = jit_load(*args, **kwargs)
     14     found_neuron_function = False
     15     for node in script_module.graph.nodes():

File ~/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/jit/_serialization.py:162, in load(f, map_location, _extra_files)
    160 cu = torch._C.CompilationUnit()
    161 if isinstance(f, str) or isinstance(f, pathlib.Path):
--> 162     cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files)
    163 else:
    164     cpp_module = torch._C.import_ir_module_from_buffer(
    165         cu, f.read(), map_location, _extra_files
    166     )

RuntimeError: The PyTorch Neuron Runtime could not be initialized. Neuron Driver issues are logged
to your system logs. See the Neuron Runtime's troubleshooting guide for help on this
topic: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/

However, this is how it looks like on my Ubuntu machine

(aws_neuron_venv_pytorch) ubuntu@ip-172-31-13-82:~/aws-neuron-sdk/src/examples/pytorch$ sudo apt-get install aws-neuron-dkms -y
Reading package lists... Done
Building dependency tree       
Reading state information... Done
aws-neuron-dkms is already the newest version (2.3.26.0).
0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
(aws_neuron_venv_pytorch) ubuntu@ip-172-31-13-82:~/aws-neuron-sdk/src/examples/pytorch$ sudo apt-get install aws-neuron-tools -y
Reading package lists... Done
Building dependency tree       
Reading state information... Done
aws-neuron-tools is already the newest version (2.1.4.0).
0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.

Hence, it seems I have the most recent versions up and running. Do you know how to fix that?

1 Antwort
0

Hi there. Please try removing the aws-neuron-dkms (no longer used for most customers) package and install aws-neuronx-dkms. In general all packages should include the neuronx naming unless you are working with inf1, in that case you still need some -neuron package, such as torch-neuron

AWS
beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen