SageMaker Training Job stuck in Stopping

Hello,

I understand that you started a training job and there was a GPU memory error but the training job's status never changed to Failed status. You then stopped the training job but it is in stopping status for a day.

Since the job is in ‘Stopping’ state for almost a day, to find the root cause of the issue, back end details associated with the job need to be checked. Hence, it is recommended to open a support case with AWS to further troubleshoot this and terminate the job. Kindly provide the training job arn and AWS region in the case.

The internal team can look into the issue and force stop the training job. Please be assured that you will not be charged for this training job since it is in stopping state.

You can open a support case with AWS using the following link https://console.aws.amazon.com/support/home#/case/create