ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateProcessingJob operation

Question

Hi All,
      I am facing the same challenge while running:(Note: i have already tried the steps in https://repost.aws/knowledge-center/sagemaker-resource-limit-exceeded-error)

```
clarify_processor_unbalanced = clarify.SageMakerClarifyProcessor(role=role,
                                                      instance_count=1,
                                                      instance_type='ml.m5.large',
                                                      sagemaker_session=sess)
```

```
clarify_processor_unbalanced.run_pre_training_bias(
    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes
    data_config=data_config_unbalanced, # Replace None
    data_bias_config=bias_config_unbalanced, # Replace None
    ### END SOLUTION - DO NOT delete this comment for grading purposes
    methods=["CI", "DPL", "KL", "JS", "LP", "TVD", "KS"],
    wait=False,
    logs=False
)

getting error: 
Job Name:  Clarify-Pretraining-Bias-2023-06-30-12-15-52-552
Inputs:  [{'InputName': 'dataset', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-east-1-822880695428/bias/unbalanced/womens_clothing_ecommerce_reviews_transformed.csv', 'LocalPath': '/opt/ml/processing/input/data', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'analysis_config', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-east-1-822880695428/bias/generated_bias_report/unbalanced/analysis_config.json', 'LocalPath': '/opt/ml/processing/input/config', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]
Outputs:  [{'OutputName': 'analysis_result', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-822880695428/bias/generated_bias_report/unbalanced', 'LocalPath': '/opt/ml/processing/output', 'S3UploadMode': 'EndOfJob'}}]
```

---------------------------------------------------------------------------
ResourceLimitExceeded                     Traceback (most recent call last)
 in ()
----> 1 clarify_processor_unbalanced.run_pre_training_bias(
      2     ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes
      3     data_config=data_config_unbalanced, # Replace None
      4     data_bias_config=bias_config_unbalanced, # Replace None
      5     ### END SOLUTION - DO NOT delete this comment for grading purposes

6 frames
/usr/local/lib/python3.10/dist-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    962             error_code = parsed_response.get("Error", {}).get("Code")
    963             error_class = self.exceptions.from_code(error_code)
--> 964             raise error_class(parsed_response, operation_name)
    965         else:
    966             return parsed_response

ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateProcessingJob operation: The account-level service limit 'ml.p2.xlarge for processing job usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please use AWS Service Quotas to request an increase for this quota. If AWS Service Quotas is not available, contact AWS support to request an increase for this quota.

Looking forward to hearing from your team.
Thanks,
Andy

Answer

This error usually occurs when you exceed the account level service quotas that are specified for your SageMaker resources.

To resolve this error, complete the following steps:

1.    Open the Service Quotas console.
Note: To use the Service Quotas console, you need the corresponding AWS Identity and Access Management (IAM) permission in your user or role.

2.    From the AWS Region selector on the navigation bar, select the Region where you get the error.

3.    In the navigation pane, choose AWS services.

4.    In the Search bar, enter Amazon SageMaker.

5.    Choose Amazon SageMaker.

6.    Select the quota that you want to increase. For the preceding example error message, select ml.p2.xlarge for endpoint usage.

7.    Choose Request quote increase.

8.    For Change quota value, enter the desired value.

9.    Choose Request.

This sends your request to AWS Support. Based on your use case and current usage, your request is either approved, denied, or partially approved.

Answer

Can you try increasing the **Processing job usage quota** for ml.c5.2xlarge as shown here

![Enter image description here](/media/postImages/original/IMk11J1ziIRseaasOnR5S2zw)

ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateProcessingJob operation

ERROR:

Relevant content