SageMaker training job fails with memory error on model upload after training completion

I am running into an issue where a SageMaker training job will successfully complete the training script but will fail during the model upload stage after training. The specific error given in the SageMaker portal is the following:

ClientError: Please use an instance type with more memory, or reduce the size of training data processed on an instance.

I upgraded the instance used for training but this did not resolve the issue. Furthermore, the issue seems to occur randomly for different training jobs where the only difference is the dataset. All other characteristics are the same. It is strange to me why this should affect the model upload stage while the training job itself is successful. Note that I do not store the training data in the model output directory. The CloudWatch log reports successful completion of the training job:

2023-06-14 07:50:10,625 sagemaker-training-toolkit INFO Reporting training SUCCESS

And the job status overview in the SageMaker portal shows where the issue arises:

Status History Starting 6/11/2023, 8:12:33 PM 6/11/2023, 8:13:47 PM Preparing the instances for training Downloading 6/11/2023, 8:13:47 PM 6/11/2023, 8:15:26 PM Downloading input data Training 6/11/2023, 8:15:26 PM 6/14/2023, 1:51:15 AM Training image download completed. Training in progress. Uploading 6/14/2023, 1:51:15 AM 6/14/2023, 1:51:21 AM Uploading generated training model Failed 6/14/2023, 1:51:21 AM 6/14/2023, 1:51:21 AM Training job failed

Anyone else encountered a similar issue? Any ideas on how to address this issue?