Sagemaker training Logs in CloudWatch

Question

Hi everyone,

I've been using AWS for training models for a week now. I'm able to submit and run serverless training jobs but unfortunately I'm unable to see the logs of these jobs. From what I've read in https://docs.aws.amazon.com/sagemaker/latest/dg/logging-cloudwatch.html, everything logged in stderr and stdout should be automatically logged in CloudWhatch under the log group "/aws/sagemaker/TrainingJobs". I've made sure to create this log group beforehand but I don't see any log stream under this log group. Also, when I try to access logs from Sagemaker UI by directly clicking on "show logs" for a training job, I correctly get redirected to ClouldWatch under the log group "/aws/sagemaker/TrainingJobs" but there is no log. Am I missing something ?

Many thanks in advance for your help

Answer

Hello.

What are the settings of the IAM role used by SageMaker?  
Try setting the following IAM policy for the IAM role used by the notebook instance.  
```
{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Action": [   
                 "logs:CreateLogDelivery",      
                 "logs:CreateLogGroup",
                 "logs:CreateLogStream",
                 "logs:DeleteLogDelivery",
                 "logs:Describe*",
                 "logs:GetLogEvents",
                 "logs:GetLogDelivery",
                 "logs:ListLogDeliveries",
                 "logs:PutLogEvents",
                 "logs:PutResourcePolicy",
                 "logs:UpdateLogDelivery"
             ],
             "Resource": "*",
             "Effect": "Allow"
         } 
     ]
 }
```

Sagemaker training Logs in CloudWatch

Relevant content