AmazonSageMaker; Status Code: 400; Error Code: ThrottlingException

Question

Hi Team,

Our Sagemaker jobs are getting failed with error in Stepfunction
"Rate exceeded (Service: AmazonSageMaker; Status Code: 400; Error Code: ThrottlingException; Request ID: 079cf4e7-db4f-4278-b49c-8d28ebb5c1fe; Proxy: null)"
When checked in logs found, TypeError: can only concatenate str (not "flot") to str

Tried increasing the Service Quota,
Maximum number of concurrent AutoML Jobs and
Maximum total concurrency that can be allocated across all serverless endpoints
But these are still under Quota requested state?

I believe we need to increase throttling Quota limit, as due to multiple concurrence jobs Warnings are crossing the limit and the whole job is failing. Could you please help me how to increase this and how much to.

If not the Quota limit issue, do we need to make changes in the code, because of any upgraded at AWS,
https://repost.aws/knowledge-center/sagemaker-python-throttlingexception

Please let me know, what exactly the issue and its resolution.

Would be great help if replied sooner.

Thanks!
Sadiq

Answer

Hi Sadiq,

Re `can only concatenate str (not "flot")`, it's probably an issue in your code: your probably have somewhere a string like "this is a number" that you concatenate with a float with + directly instead of doing string + str(float).

Fixing with str() should remove the log that you see. It may also remove the throttling error that get: depending on where the str(float) is missing, it may trigger boto3 retries (as suggested by the article that you point to) and create the excessive rate that is throttled.

Best,

Didier

AmazonSageMaker; Status Code: 400; Error Code: ThrottlingException

Relevant content