SageMaker Training Plans Not Working with Hyperparameter Tuning Jobs

Question

### Problem
Despite including `trainingPlanArn` in hyperparameter tuning job resource config, training jobs remain "Pending" and don't use reserved capacity. The training plans are ignored by hyperparameter tuning jobs. Despite there being a trainingPlanArn under the CloudTrail API call for CreateHyperParameterTuningJob, the service creates CreateTrainingJob calls without the trainingPlanArn.

### Evidence
1. **Training Plan Status**: Active with 2/2 instances available, 0 in use
2. **CloudTrail Logs**: Show `trainingPlanArn` correctly included in `CreateHyperParameterTuningJob`
3. **Job Status**: All training jobs stuck in "Pending" state
4. **Expected**: Should use reserved capacity from training plan
5. **Actual**: Jobs wait for on-demand capacity (which isn't available)

### CloudTrail Evidence
```json
"resourceConfig": {
    "instanceType": "ml.p4d.24xlarge",
    "trainingPlanArn": "arn:aws:sagemaker:REGION:ACCOUNTID:training-plan/training-plan-name"
}

re:Post Agent · Answer

Based on the information provided, there appears to be a misunderstanding about how SageMaker training plans work with different job types. The documentation doesn't explicitly mention support for hyperparameter tuning jobs with training plans.

SageMaker training plans are specifically designed to reserve compute capacity for SageMaker training jobs or HyperPod clusters. When a training plan has a "Scheduled" or "Active" status, you can utilize the reserved capacity for these specific workload types.

The documentation clearly states that you can use SageMaker training plans for your training jobs by specifying the plan of your choice when creating a training job. However, there's no mention of compatibility with hyperparameter tuning jobs.

This explains why your hyperparameter tuning jobs remain in "Pending" state despite including the trainingPlanArn in the configuration. The underlying training jobs created by the hyperparameter tuning job aren't inheriting the training plan configuration.

For now, you might need to use standard training jobs with your training plans rather than hyperparameter tuning jobs. If you need to perform hyperparameter optimization, you may need to implement your own hyperparameter search logic using individual training jobs that can properly utilize your reserved capacity.  
**Sources**  
[Training plans utilization for SageMaker training jobs - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/training-plan-utilization-for-training-jobs.html)  
[View training plan details - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/training-plan-details-using-api-cli-sdk.html)

SageMaker Training Plans Not Working with Hyperparameter Tuning Jobs

Problem

Evidence

CloudTrail Evidence

Add your answer

Relevant content