Amazon CloudWatch Agent File Encodings Not Working

Question

We have been using Amazon CloudWatch Agent to ingest log files for a customer. 
Earlier yesterday we tried to configure it to collect from a file with a UTF-16 LE encoding. When it first got streamed into logs it had the standard UTF-8 to UTF-16 error with an extra space `E R R O R : . . .`.

In the CloudWatch documentation there is an encoding parameter to put into each file in collect_list. 
We first tried the suggested `utf-16le` and logs stopped streaming, we switched to see if it would take regular `utf-16` encoding but still no luck.

To determine if it was an issue on our end we specified `utf-8` since that is the default for CloudWatch but even still it won't stream unless the encoding isn't specified. The last ditch effort was to try and specify `unicode`. This formatting change allowed logs to stream again since it fits in UTF-16 byte range but of course the characters were all bogus. Unicode support is not mentioned in the documentation so I'm wondering if these values are coming from somewhere else, perhaps .NET classes?

From what I've found online this seems to be an issue unique to us, is this a niche bug on the version we are using?

Agent version: 1.300057.0
We are on Windows Server 2019.

Nick K · Accepted Answer

I wanted to report that I corrected this by (also Windows Server 2019):

- Stop AmazonCloudWatchAgent service
- Delete State files under "{Install_Directory}\AmazonCloudWatchAgent\Logs\state\*"
- Edit "{Install_Directory}\AmazonCloudWatchAgent\amazon-cloudwatch-agent.json" as follows with the uppercase UTF-16 (not sure if that is all that will work but I am so done with trying stuff):
	{
		"file_path": ...,
		"log_group_name": ...,
		"log_stream_name": ...,
		"timestamp_format": ...,
		"multi_line_start_pattern": "{timestamp_format}",
		"encoding": "UTF-16"
	}
- Start AmazonCloudWatchAgent service.

Hope this resolves it for you too.

re:Post Agent · Answer

The issue you're experiencing with CloudWatch Agent file encodings is related to how the agent handles different character encodings.

According to the CloudWatch Agent documentation, there is indeed an `encoding` parameter that can be specified in the configuration file for log collection. This parameter allows you to specify the encoding of the log file so that it can be read correctly. The default value is `utf-8`.

For UTF-16LE encoding, the correct parameter value should be `utf-16le` (which you tried). The documentation lists both `utf-16` and `utf-16le` as valid encoding options. When specifying an incorrect encoding, there might be data loss because characters that cannot be decoded are replaced with other characters, which explains the garbled output you saw.

The fact that logs stopped streaming when you specified the encoding suggests there might be an issue with how the agent is processing the encoding parameter on your Windows Server 2019 environment. This could potentially be a bug in the version you're using (1.300057.0).

To troubleshoot this further:

1. Check the CloudWatch agent logs located at `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log` for any error messages related to encoding.

2. Enable debug logging by setting the `debug` field to `true` in the `agent` section of your configuration file, then reconfigure and restart the agent. This will provide more detailed information in the logs.

3. When making configuration changes, ensure you're using the `fetch-config` option when restarting the agent to properly apply the new configuration.

The encoding values should be coming from Python's `codecs.decode()` function, not from .NET classes, as the CloudWatch agent uses these encoding standards.  
**Sources**  
[Manually create or edit the CloudWatch agent configuration file - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html)  
[CloudWatchLogsLogStream - OpsWorks](https://docs.aws.amazon.com/opsworks/latest/APIReference/API_CloudWatchLogsLogStream.html)  
[Troubleshooting the CloudWatch agent - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/troubleshooting-CloudWatch-Agent.html)  
[Issue with CloudWatch agent configuration | AWS re:Post](https://repost.aws/questions/QUjf1n7uHcRu6Tfoo8-KwMWQ/issue-with-cloudwatch-agent-configuration)  
[CloudWatch Logs agent reference - Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html)

Amazon CloudWatch Agent File Encodings Not Working

Add your answer

Relevant content