Inf2 compilation outside of Inf2 instances (i.e. within CPU-only docker instances)

0

Hi,

One big difference I've noticed between the inf1 and inf2 development experience is that with inf2, we are not able to compile models outside of inf2 instances (unlike inf1 where we were able to compile on a separate CPU-only instance). This definitely dampens the development experience as we're able to develop most of the model compilation toolchain locally without having to provision the remote inf instances for compilation purposes.

Am I missing something here, or are folks going through a similar challenge of developing the compilation workflows only on their inf2 instances?

質問済み 9ヶ月前291ビュー
2回答
1

With inf1 instances, you were indeed able to compile models outside of inf1 instances, which made the development and deployment process more flexible. This allowed developers to test and refine their models on CPU-based machines before deploying on the inf1 instances.

However, as AWS iterates and improves its services, there may be changes in workflow or capabilities. The restrictions on inf2 instances you mentioned could be a part of these changes, which could be due to various reasons ranging from technical requirements, performance optimization, security considerations, or other factors.

profile picture
回答済み 9ヶ月前
0

Check out https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/inference/api-torch-neuronx-trace.html#torch-neuronx-trace-api - this should work on a CPU instance, though you will need to install all the components (including the runtime-lib) on your instance for the installation to work correctly.

AWS
回答済み 4ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ