Skip to content

AWS Batch compute environments auto-revert to 0 vCPUs in eu-west-2 - Service Issue?

0

PROBLEM SUMMARY: All AWS Batch compute environments in eu-west-2 are automatically reverting desiredvCpus to 0 within minutes, making it impossible to run any jobs. This affects both GPU and CPU instance types.

REGION: eu-west-2
BUSINESS IMPACT: Production pipeline completely blocked

REPRODUCE STEPS:

  1. Create any compute environment (tested: g5.xlarge, m5.large)
  2. UpdateComputeEnvironment with desiredvCpus=4
  3. Wait 2-5 minutes
  4. DesiredvCpus automatically reverts to 0
  5. Auto Scaling Groups created with Max=0 (abnormal)

TROUBLESHOOTING COMPLETED:

  • Tested 3 different compute environments: all same behavior
  • Multiple instance types affected
  • Service quotas verified OK
  • IAM/networking validated
  • AMI functionality confirmed
  • No automation/Lambda interference found

QUESTION FOR AWS: Is there a known service issue with AWS Batch in eu-west-2? This appears to be systematic failure affecting all compute environments regardless of configuration.

SIMILAR ISSUES? Has anyone else experienced this behavior? All jobs remain RUNNABLE indefinitely because no instances can launch.

WORKAROUND NEEDED: This is blocking production workload. Are there any known workarounds or alternative regions recommended?

2 Answers
4

How about this as workaround:

  • Manually Set vCPUs – Try setting Min vCPUs to a non-zero value to prevent automatic scaling to zero.
  • Test in Another Region – If possible, deploy a test environment in eu-west-1 or us-east-1 to see if the issue persists.
EXPERT
answered 9 months ago
EXPERT
reviewed 9 months ago
1

Hi, thank you for using Batch!

All jobs remain RUNNABLE indefinitely because no instances can launch.

I recommend configuring notifications on blocked job queues (https://aws.amazon.com/blogs/hpc/introducing-new-alerts-to-help-users-detect-and-react-to-blocked-job-queues-in-aws-batch/). You will find out exactly the reason instances cannot launch.

UpdateComputeEnvironment with desiredvCpus=4

If your goal is to persist compute, you should use min vCpu instead of the desired vCpu (https://docs.aws.amazon.com/batch/latest/userguide/updating-compute-environments.html). Batch will scale down compute if it is incompatible for incoming jobs to save customer costs. Desired vCpu dropping to 0 is expected in your case.

AWS
answered 9 months ago
EXPERT
reviewed 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.