Parsing logs in Sagemaker Endpoints

Question

I have deployed a sagemaker endpoint and I'm trying to get my logs logged to the cloudwatch logstream. I did so by plainly printing the dictionary to stdout. While the logs get logged, I'm unable to parse them as sagemaker internally appends it's own prefix, making doing any kind of  downstream analytics difficult

Here's an eg of what gets logged to the logstream

```

@message	
2023-10-10T14:20:30,885 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - {'level': 'info', 'timestamp': '2023-10-10 14:20:30 GMT', 'service': 'lorem ipsum', 'description': 'some desc', 'log_event': 'lorem', 'environment': 'env', 'model_version': 'v0', 'call_id': 'some uuid'}
```

Is there a good way to bypass this prefix(2023-10-10T14:20:30,885 [INFO ] W-9000-model_1.0-stdout MODEL_LOG), so I can make the log parsable?

Answer

Hello.

Parsing Logs in CloudWatch: Since the prefix remains consistent, you can use a combination of CloudWatch insights query language or other log processing tools to split the log lines into prefix and the actual log message, and then parse the JSON content.

For instance, using CloudWatch Insights, you can structure your query like:

```
fields @timestamp, @message
| parse @message as prefix, json_content
| filter @message like "MODEL_LOG"
```
This will give you the raw JSON logs in the json_content variable which you can then further parse or manipulate as needed.

Best regards, Andrii

Parsing logs in Sagemaker Endpoints

Relevant content