Clean Amazon AMI - CloudWatch agent not pushing logs up and error on CPU

Question

So I have a few machines running older OS's and CW is working fine.   I have a new clean Amazon AMI, easy CW install and all I want is 2 logs going up and neither are showing up.

Agent is running fine;

```
{
  "status": "running",
  "starttime": "2024-02-02T00:57:19+00:00",
  "configstatus": "configured",
  "version": "1.300028.1"
}

```
I ran the wizard, all the defaults except storing the config in SSM, and the 2 files are readable, that section of the config looks fine as it was auto built;

```
"logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/myservice.log",
                                                "log_group_name": "MyService",
                                                "log_stream_name": "{instance_id}",
                                                "retention_in_days": -1
                                        },
                                        {
                                                "file_path": "/var/log/audit/audit.log",
                                                "log_group_name": "audit",
                                                "log_stream_name": "{instance_id}",
                                                "retention_in_days": -1
                                        }
                                ]
                        }
                }
        },

```

That box has the CloudWatchAgentServerPolicy role so should have access, but nothing appears.  The only thing that shows in the amazon-cloudwatch-log is this;

```
2024-02-02T00:57:21.613Z        info    service/service.go:148  Everything is ready. Begin running and processing data.
2024-02-02T00:57:21.768Z        info    ec2tagger/ec2tagger.go:500      ec2tagger: Initial retrieval of tags succeeded  {"kind": "processor", "name": "ec2tagger", "pipeline": "metrics/host"}
2024-02-02T00:57:21.768Z        info    ec2tagger/ec2tagger.go:411      ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes       {"kind": "processor", "name": "ec2tagger", "pipeline": "metrics/host"}
2024-02-02T00:57:22Z E! [inputs.disk] [SystemPS] => error getting disk usage ("/sys/kernel/debug/tracing"): permission denied
2024-02-02T00:58:22Z E! [inputs.disk] [SystemPS] => error getting disk usage ("/sys/kernel/debug/tracing"): permission denied
2024-02-02T00:59:22Z E! [inputs.disk] [SystemPS] => error getting disk usage ("/sys/kernel/debug/tracing"): permission denied

```

So I did see that permission denied on other machines that CW is working fine, but not sure what to look at next as there were no other obvious things that would prohibit this to work.

Any help or suggestions is appreciated.

Answer

The log snippet you provided shows the error related to disk input plugin and "/sys/kernel/debug/tracing" being denied.Ensure that the user running the CloudWatch agent has the necessary permissions to read the log files. Check the file permissions for /var/log/myservice.log and /var/log/audit/audit.log to make sure the agent has the required access and also verify that the EC2 instance has the necessary outbound network access to communicate with CloudWatch. Check the security group and network ACL settings

Answer

Thanks,

So I did re-run the wizard and made sure I selected run as root, and watched a bit more the results.  So when the wizard finished, I saw the confirm file along with /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully.   When I looked at that file, I see the 2 key items, the runas root and my log information.

I manually stop the agent, `/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop` and then start `/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start` and do see it running as root now (good), that error is gone, however I don't see the log getting written to the log group.

The root user can naturally see the file, and the CW agent should have access to cloudwatch as I mentioned that server has the CloudWatchAgentServerPolicy	role so think that is all that is needed and there is no outbound network rules preventing.

Clean Amazon AMI - CloudWatch agent not pushing logs up and error on CPU

Relevant content