CloudWatch cannot access the shared credentials file

0

I am using SSM, CLI and CloudWatch on onPremise Linux systems. SSM has installed the credentials in the /root/.aws folder, which of course needs root permission to access. SSM and CLI work, but I am accessing the remote machine to configure cloudwatch, but get the following error.

*025-02-04T23:04:09Z E! [outputs.cloudwatchlogs] Aws error received when sending logs to scanner-logs/bodd-2420002: SharedCredsLoad: failed to load shared credentials file caused by: FailedRead: unable to open file caused by: open /root/.aws/credentials: permission denied 2025-02-04T23:04:09Z W! [outputs.cloudwatchlogs] Retried 4 time, going to sleep 2.027958371s before retrying. *

I have looked to see if I could 'ask' SSM to store the credentials somewhere else, but that does not seem possible. I cannot use any ~[HOME] path here, because they are agents. I have changed the permissions on the /root/.aws folder, but of course that doesn't work as root is still restricted.

7 Answers
0

Hello.

Why not try specifying "shared_credential_file" directly in the configuration file below?
From the error log, it seems that the credentials in "/root/.aws/credentials" cannot be opened, so creating and configuring a file in another directory may solve the problem.

/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

I think the settings in the following document will be helpful.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html#install-CloudWatch-Agent-iam_user-first

profile picture
EXPERT
answered 2 days ago
  • Did you make sure to use root user when starting CloudWatch Agent as below? I think that if I add "sudo" to the command, the root user will be used to run the CloudWatch Agent, so I can use "/root/.aws/credentials".

    sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path
    
  • What are the permission settings for "/root/.aws/credentials"? Can you confirm that you have read permission with the command below?

    ls -la /root/.aws/credentials
    

    Which of course points to the /root/.aws/credentials file. I would really like to move that file to some place more friendly, but SSM and CLI need it too, and they do not seem to have a mechanism to define a new path.

    How about copying ".aws/credentials" to a different Linux user's home directory and changing the "common-config.toml" settings without moving the "/root/.aws/credentials" location?

  • How about separating the IAM user access key for CloudWatch Agent and the access key for SSM Agent? In other words, how to use "/root/.aws/credentials" as is for SSM Agent, create a new access key from another IAM user, and set the access key in the other Linux user's home directory for CloudWatch Agent.

0

This is the problem. I already have the shared credentials information in the toml file... Which of course points to the /root/.aws/credentials file. I would really like to move that file to some place more friendly, but SSM and CLI need it too, and they do not seem to have a mechanism to define a new path.

answered 2 days ago
0

Hi Riku.

yeap.

sudo amazon-cloudwatch-agent-ctl -a start -m onPremise -c ssm:AmazonCloudWatch-linux-123456-v1

where 123456 is the config in the store

answered 2 days ago
0

The problem with just copying the files is that SSM will rotate the credentials at some stage, and I lose connectivity for cloudwatch.

I would like to be able to tell SSM and CLI where to go for the credentials, but I can't find that option. In the meantime, I have change CloudWatch config to have run_as_user root and as cwagent, but it always gets overwritten back to cwagent. I have also added cwagent to the sudoers file, and that did not do anything either. It is almost like the reported error is not the actual error.

answered 2 days ago
0

I also did a chmod 777 on the /root/.aws/credentials file to see if that would fix, but it did not.

answered 2 days ago
0

We have quite a lot of remote devices, so creating and managing additional IAM User accounts would be prohibitive, and more complex than SHOULD be necessary.

answered 2 days ago
0

Turns out, editing all of the json and toml files was pointless. The file that did the trick was found in

/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d

and the file was called

default

This was where editing the "run_as_user" = "root" solved all problems.

answered 2 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions