ECS Capacity Provider not working when task memory exceeds 16G

Based on the information provided, it appears that you're encountering an issue with your ECS Capacity Provider not properly scaling to accommodate tasks requiring more than 16GB of memory. This behavior is unexpected, as the Capacity Provider should indeed scale up to larger instance types when needed. However, there are a few potential reasons for this issue:

1. Instance Type Configuration: Ensure that your Auto Scaling Group is correctly configured to launch the larger instance types (g5.2xlarge, g5.4xlarge) when needed. The ASG should have a mixed instances policy that includes all the desired instance types.

2. Capacity Provider Configuration: Verify that your Capacity Provider is set up to manage the Auto Scaling Group correctly. The managed scaling should be enabled, and the target capacity should be set appropriately (usually 100%).

3. Task Definition Compatibility: Make sure that your task definition is compatible with the larger instance types. Sometimes, task placement constraints or other settings might prevent tasks from being placed on certain instance types.

4. CloudWatch Metric Issues: The fact that the CapacityProviderReservation metric is not changing for tasks requiring more than 16GB of memory suggests a potential issue with how this metric is being calculated or reported. This could be due to a misconfiguration or a limitation in how the metric is generated.

5. Service Auto Scaling: As an alternative approach, you might consider adding headroom to your service instead of relying solely on the Capacity Provider. This can be done by increasing the number of replicas in your service or modifying the target tracking scaling metric for the service auto scaling.

6. Memory Hard Limit: Ensure that your task definitions have a set memory limit for their container definitions. AWS Config has a rule (ECS_TASK_DEFINITION_MEMORY_HARD_LIMIT) that checks for this, and it's possible that missing memory parameters could cause issues with task placement.

To troubleshoot this issue:

1. Double-check your Auto Scaling Group configuration to ensure it includes all necessary instance types.
2. Verify your Capacity Provider settings, especially the managed scaling configuration.
3. Review your task definitions for any constraints that might prevent placement on larger instances.
4. Consider temporarily adding some larger instances manually to see if tasks can be placed on them.
5. Monitor CloudWatch logs and events for any error messages related to task placement or scaling.

If the issue persists after these checks, it may be worth contacting AWS support for further assistance, as there could be an underlying issue with how the Capacity Provider is interpreting the resource requirements for your tasks.
Sources
Automatically manage Amazon ECS capacity with cluster auto scaling - Amazon Elastic Container Service
ecs-task-definition-memory-hard-limit - AWS Config
Optimize Amazon ECS cluster auto scaling - Amazon Elastic Container Service
Creating an Auto Scaling group capacity provider using the classic console - Amazon ECS

Double check your Quota limit for the g5 instances also incase you've reached it

Thanks, I've done that and it is fine. The really strange thing that I've finding is that the CapacityProviderReservation is not changing.