Possible quota issue

Question

I opened up an AWS support case to increase my quota based on the error below, but AWS support states this is not a quota issue.  They were unwilling to help me in support but directed me to this forum.

I'm getting the following error when running some AWS SageMaker batch transform jobs on m5.large instances.  I'm running 8 transforms in parallel against 8 different models in a step function map.  A few of the transforms succeed, but some fail with the following error:
{
  "resourceType": "sagemaker",
  "resource": "createTransformJob.sync",
  "error": "SageMaker.AmazonSageMakerException",
  "cause": "Rate exceeded (Service: AmazonSageMaker; Status Code: 400; Error Code: ThrottlingException; Request ID: 49a80dc1-df06-4b88-a462-24e517d13531; Proxy: null)"
}

What is going on here?  Am I doing something wrong?  What kind of throttling exception is this (what is being throttled?)

Answer

I have seen this with batch transform when I try to trigger too many jobs too quickly (usually like 4xjobs within 1-2 seconds).

What I did as a workaround was handle and recognize this quota error on my service, sleep for 5 seconds and retry the same job.

Possible quota issue

Contenus pertinents