Using mixed instance ASG policies for multi-architecture ECS clusters?

Question

I have an ECS cluster that is entirely x86-64. It has a relatively simple configuration, with a single capacity provider associating it with an ASG that launches a single, x86-based instance type (say, `m6a.large`).

I would like to start slowly migrating some services within the cluster over to aarch64. I have added a placement constraint to the existing services to ensure they only launch on `cpu-architecture == x86_64` instances.

I am trying to figure out whether a [mixed instance policy](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups-launch-template-overrides.html) is enough to manage instances of both architectures within the cluster.

I would like the ASG to continue to meet task demand for x86 capacity in the cluster as it has been, but I would also like it to start meeting aarch64 capacity demand, as services are migrated over, with, say, `m8g.large` instances.

The documentation for mixed instance policy does say that they can be used to mix architectures in a single ASG, but the instance type or launch template selection seems to be based entirely on the priority given to each instance type in the policy, and not necessarily on the architecture-specific demand.

My initial plan was to use two ASGs, with one exclusively handling aarch64 instances, and the other handling exclusively x86-64 instances, before I came across mixed instance policies. The mixed instance policy seems much simpler and more elegant if it's possible, but I'm having a hard time figuring out how to make it work for this case.

Henrique Santana · Accepted Answer

Hello Farski,

While mixed instance policies in Auto Scaling Groups (ASGs) can indeed support multiple architectures, they might not be the best solution for this specific use case. [This blog post](https://aws.amazon.com/blogs/containers/rolling-ec2-ami-updates-with-capacity-providers-in-amazon-ecs/) demonstrates a more controlled and flexible approach using ECS Capacity Providers, which seems to be a better fit for gradually migrating from x86-64 to aarch64 (ARM) instances.

Here's a summary of the recommended approach:

1. Create two separate Auto Scaling Groups:

- One for x86-64 instances (e.g., m6a.large)
    - One for aarch64 instances (e.g., m8g.large)

2. Create two Capacity Providers, each associated with one of the ASGs.

3. Add both Capacity Providers to your ECS cluster.

4. Update your ECS service definition to use a Capacity Provider Strategy that includes both providers. This allows you to control the distribution of tasks between the two architectures.

5. Gradually migrate your services by adjusting the Capacity Provider Strategy. For example:

- Start with a strategy that uses only the x86-64 Capacity Provider
    - Introduce the aarch64 Capacity Provider with a lower weight
    - Gradually increase the weight of the aarch64 provider and decrease the x86-64 provider
    - Eventually, remove the x86-64 provider from the strategy when migration is complete

This approach offers several advantages:

- Fine-grained control over the migration process
- Ability to easily roll back if issues are encountered
- Automatic scaling of each instance type based on its respective task demand
- Smooth transition without the need to modify existing tasks or services beyond the Capacity Provider Strategy

The blog post [Rolling EC2 AMI updates with capacity providers in Amazon ECS](https://aws.amazon.com/blogs/containers/rolling-ec2-ami-updates-with-capacity-providers-in-amazon-ecs/) demonstrates this approach in detail, including how to implement it using AWS CDK, which provides a more manageable and reproducible way to define and update your infrastructure.

Hope this helps you!

re:Post Agent · Answer

Using a mixed instances policy for managing multi-architecture ECS clusters is possible, but it may not be the ideal solution for your specific use case. While mixed instances policies do allow you to include different instance types and architectures within a single Auto Scaling group (ASG), they have limitations that may not fully address your requirements.

The main challenge with using a mixed instances policy for your scenario is that the instance type selection is primarily based on the priority you set in the policy, rather than the specific architecture demand of your ECS tasks. This means you can't guarantee that the ASG will launch the correct proportion of x86-64 and aarch64 instances to match the evolving needs of your services as you migrate them.

Your initial plan of using two separate ASGs - one for x86-64 instances and another for aarch64 instances - is actually a more robust and flexible approach for managing a multi-architecture ECS cluster. This setup allows you to:

1. Independently scale each architecture based on its specific demand.
2. Apply different scaling policies and capacity provider strategies for each architecture.
3. Gradually migrate services without risking resource contention between architectures.
4. Maintain better control over the distribution of instances and costs.

To implement this two-ASG approach:

1. Keep your existing ASG for x86-64 instances.
2. Create a new ASG for aarch64 instances (e.g., using m8g.large).
3. Create a new ECS capacity provider for the aarch64 ASG.
4. Update your ECS cluster to use both capacity providers.
5. As you migrate services to aarch64, update their task definitions and service configurations to use the appropriate placement constraints and capacity provider strategy.

This approach, while slightly more complex to set up initially, provides greater control and flexibility as you transition your cluster to support multiple architectures. It allows you to fine-tune the scaling behavior for each architecture independently and ensures that you're launching the right type of instances to meet the specific demands of your x86-64 and aarch64 tasks.

While mixed instances policies can be useful in many scenarios, for managing multi-architecture ECS clusters with specific capacity needs for each architecture, the two-ASG approach is generally more suitable and manageable in the long run.  
**Sources**  
[Use a different launch template for an instance type - Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups-launch-template-overrides.html)  
[Setup overview for creating a mixed instances group - Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/mixed-instances-groups-set-up-overview.html)  
[Cluster Autoscaler - Amazon EKS](https://docs.aws.amazon.com/eks/latest/best-practices/cas.html)

Using mixed instance ASG policies for multi-architecture ECS clusters?

Relevant content