By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

SageMaker Neo Compilation - Unable to Neo Compile for FP16 and INT8 precision

0

I'm trying to Neo compile a Pytorch YoloV5 Large model for edge deployment on an Nvidia Jetson Xavier NX device. I'm able to do it using the default settings for FP32 precision but I'm unable to do it for FP16 or INT8 precision. I have tried passing it in "CompilerOptions" in the OutputConfig but the output of Neo compilation is still FP32.

How can I get a Neo compiled model for for FP16 and INT8 precision? Does Neo support these precision modes or not?

Topics
Machine Learning & AI
Tags
Machine Learning & AIAmazon SageMaker EdgeAmazon SageMaker Deployment
Language
English
shitijkarsolia
asked a year ago232 views
1 Answer
0

Unfortunately Neo doesn't support quantization for Jetson Devices. It means you can only compile FP32 models and they will be FP32 after compilation.

I know this is not what you're looking for, but FYI, Neo supports int8 model optimization only for TFLite and targeting CPU not GPU. Check here some supported models: https://docs.amazonaws.cn/en_us/sagemaker/latest/dg/neo-supported-edge-tested-models.html

AWS
Samir Araujo
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content