Start up times for AWS Batch jobs with big outliers

There are a few potential factors that could be causing the occasional long start times for AWS Batch jobs:

• Instance limits - If there are insufficient On-Demand g4dn.2xlarge instances available in the region, it will take time for new ones to spin up. You can check CloudWatch metrics for instance usage and limits.

• Scheduling - If multiple jobs are queued at once, it can take time for the scheduler to place them all on instances. The scheduling policy may prioritize smaller jobs.

• AMI caching - Batch caches AMIs on hosts. If an AMI is evicted, it will need to be re-cached before launching new jobs, adding delay.

• Host initialization - Sometimes when hosts start they need time to initialize resources, security groups, VPC settings etc before accepting jobs.

Some things you could try:

• Request an On-Demand instance limit increase for that type.

• Use a ComputeEnvironment with a higher priority to get access to instances sooner.

• Switch to Spot Instances to increase available capacity. Can use a mix of Spot + On-Demand.

• Try spreading job submissions over time to reduce scheduling bursts.

• Check Batch metrics in CloudWatch for queue depth, scheduling latency, etc.

• Ensure latest AWS Batch agent is installed on AMIs.

The outliers are likely due to bursting above capacity and waits for new hosts. Increasing instance limits and supplementing with Spot should help smooth things out.