Best practice for using Inferentia in ECS

Question

Currently, we are using Elastic Inference for inferencing on AWS ECS. We use `inference_accelerators` in `ecs.Ec2TaskDefinition` to set up elastic inference. For scaling, we are monitoring `AcceleratorUtilization` metric to decide when to scale out or scale in.

Now that AWS recommends switching to AWS Inf instances, we plan to migrate from EI to Inf. We plan to use it in AWS ECS. How should we monitor Inf usage and scale our instances based on that? Is there any pre-defined metrics for this usage?

Answer

You can use [Neuron Monitor](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/neuron-monitor-user-guide.html) to monitor your Inf utilization. Neuron Monitor integrates with CloudWatch (see [this documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/neuron-monitor-user-guide.html#neuron-monitor-cloudwatch-py)). One metric you can use to determine scaling is NeuronCore utilization. For example, you can average the number of NeuronCores with utilization that's higher than some threshold and scale up or down based on that.

Best practice for using Inferentia in ECS

Relevant content