How to send own failure info in case of failed SageMaker Training Job?

Question

Good day!

**My main purpose:**

Easy way to collect information about different failure scenarios in SageMaker TrainingJob.

**What do I use currently?**

Sagemaker SKLearn Estimator(TrainingJobs are inside)

**Where will my model train?**

Different datasets. So, I need control and collect all information about all training processes and their final statuses on different datasets.

**Which failure scenarios do I have?**

There are plenty of them. I have create my own python Errors for them.  
For example:
1. There are labels only for one class.
2. Too small dataset(by my own criterions)
3. Missing data for crucial columns
4. e.t.c.

**Where am I stuck?**

After failed training I can't get own errors from training job response. All of them are "ExecuteUserScriptError"
I can't pass my own info in FailureReason or ErrorMessage(always it's empty). I see which error was raised in CloudWatchLogs and TrainingJobTraceback(from SagemakerNotebook). So, bad solution is parse all CloudWatchLogs in case of failure.

**Question:
How to provide my own ErrorMessage or FailureReason? **

May be I am digging in the wrong direction. Anyway, I need your advice.
Thank you so much for possibility to ask an advice here)

Answer

Hi, did you try writing to `/opt/ml/output/failure` as per the [doc here](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo-output.html)?

It's worth mentioning that in the past there was [a bug that overwrote this file](https://github.com/aws/sagemaker-training-toolkit/issues/111) in the base training toolkit that powers "script mode" containers. This got resolved at source per the linked issue, but I guess there's a chance older containers or frameworks which customize this tool could still be affected? So may be worth upgrading your framework version if you're using an older one.

Answer

`In: sklearn_estimator.latest_training_job.describe()['FailureReason']`

`Out: `
```AlgorithmError: framework error: 
Traceback (most recent call last):
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_trainer.py", line 84, in train
    entrypoint()
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_sklearn_container/training.py", line 39, in main
    train(environment.Environment())
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_sklearn_container/training.py", line 35, in train
    runner_type=runner.ProcessRunnerType)
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_training/entry_point.py", line 100, in run
    wait, capture_error
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_training/process.py", line 291, in run
    cwd=environment.code_dir,
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_training/process.py", line 208, in check_error
    info=extra_info,
sagemaker_training.errors.ExecuteUserScriptError: ExecuteUserScriptError:
ExitCode 1
ErrorMessage ""
Command "/miniconda3/bin/python train.py"

ExecuteUserScriptErro```

Answer

This issue looks like related to my problem)
There are 2 sagemaker-scikit-learn versions: 0.23-1, 0.20-0.  
So, I use:
`683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3`
sagemaker version: 2.86.0 (tried to upgrade with pip in terminal to 2.90.0 , but there is the previous version 2.86 in notebooks)
Python - 3.7

**How can I understand which versions I should use to get away of ErrorMessage problem?**

Answer

Hello, is the reported problem similar to this issue reported on SageMaker Python SDK project ?

https://github.com/aws/sagemaker-python-sdk/issues/1952

How to send own failure info in case of failed SageMaker Training Job?

Relevanter Inhalt