How should the CloudWatch agent be configured with credentials when it sends logs to CloudWatch in other regions?

I tested this out in a different partition cn-north-1, and I was able to publish logs from us-east-1 this partition without any issues. I used the credentials of an IAM user in the destination region, and configured it on the root user of my instance with the same command that you had shared:

$ sudo aws configure --profile AmazonCloudWatchAgent

In my common-config.toml, I specified this user profile, and provided the path for the credentials file (my path includes root as the profile was configured as a root user):

$ sudo cat /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

 [credentials]
    shared_credential_profile = "AmazonCloudWatchAgent"
    shared_credential_file = "/root/.aws/credentials"

For the configuration file, I provided the region attribute under the agent section, and specified the region as cn-north-1. Here is a sample configuration file that I used:

$ sudo cat /opt/aws/amazon-cloudwatch-agent/bin/config.json

{
        "agent": {
                "run_as_user": "root",
                "aws_sdk_log_level": "LogDebug | LogDebugWithEventStreamBody",
                "region": "cn-north-1"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/home/ec2-user/test.log",
                                                "log_group_name": "test_log_group",
                                                "log_stream_name": "test_log_stream"
                                        }
                                ]
                        }
                }
        }
}

NOTE: I did not use any instance profile during testing. But, in case you make use of it, the EC2 user credentials takes precedence over the instance profile role associated with the instance[1].

After restarting the agent, the logs were being pushed to the log group in a different partition from an instance in the us-east-1 region

Can you verify the following information to check this out further?

• Verify if the common-config.toml is correctly configured with the profile name and path; and ensure that the lines are not commented.
• Make sure that the credentials that you are using are correct while configuring the profile on the instance, and check if they have the PutLogEvents permissions, required to publish log data to CloudWatch [2].
• Check and ensure that the configuration file has the region mentioned in it under the agent section.

You can share the configuration file, the common-config.toml, and the path of your credentials; by removing any sensitive data (like AWS access keys/secrets, instance IDs, account IDs etc.,), so that I can have a look at what configuration you are making use of.

References:

[1] https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#configure-precedence

[2] https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html

The CloudWatch agent does support sending of logs from one region to another. Just by looking at the error message that you have receieved while running the agent:

2024/01/09 08:05:15 unable to scan config dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d with error: unable to parse json, error: unexpected end of JSON input

It indicates that the configuration JSON file has an issue.

Additionally, the following error:

2024/01/09 08:05:15 E! Credentials path is not set while runasuser is not root

states that the credentials may not be set up in the correct path in the common-config.toml file [1].

To check this out, can you share the agent version, the configuration file, the entire path where you have stored the credentials in and the common-config.toml file? Make sure that any sensitive data is redacted while you share them here.

The agent version can be retrieved using the status command [2]:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

The configuration file is stored in the location:

/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

The common-config.toml file is usually stored in the location [3]:

/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

References:

[1] Running the CloudWatch agent as a different user - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-common-scenarios.html#CloudWatch-Agent-run-as-user

[2] Finding information about CloudWatch agent version - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/troubleshooting-CloudWatch-Agent.html#CloudWatch-Agent-troubleshooting-agent-version

[3] CloudWatch agent files and locations - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/troubleshooting-CloudWatch-Agent.html#CloudWatch-Agent-files-and-locations

/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Every time the agent is started using the configuration file /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json, /opt/aws/amazon-cloudwatch-agent/bin/config.json will not be automatically generated. There was one before. After the Taiwan instance starts the agent, it seems that /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json will be deleted and the /opt/aws/amazon-cloudwatch-agent/bin/config.json file will be generated.

{
  "agent": {
    "metrics_collection_interval": 10,
	"region": "us-gov-west-1",
    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
	
  },
  "metrics": {
    "namespace": "CICDM_Test",
    "metrics_collected": {
      "cpu": {
        "resources": ["*"],
        "measurement": [
          { "name": "cpu_usage_idle", "rename": "CPU_USAGE_IDLE", "unit": "Percent" },
          { "name": "cpu_usage_nice", "unit": "Percent" },
          "cpu_usage_guest"
        ],
        "totalcpu": false,
        "metrics_collection_interval": 10,
        "append_dimensions": {
          "customized_dimension_key_1": "customized_dimension_value_1",
          "customized_dimension_key_2": "customized_dimension_value_2"
        }
      },
      "disk": {
        "resources": ["/"],
        "measurement": [
          { "name": "free", "rename": "DISK_FREE", "unit": "Gigabytes" },
          "total",
          "used"
        ],
        "ignore_file_system_types": ["sysfs", "devtmpfs"],
        "metrics_collection_interval": 60,
        "append_dimensions": {
          "customized_dimension_key_3": "customized_dimension_value_3",
          "customized_dimension_key_4": "customized_dimension_value_4"
        }
      },
      "diskio": {
        "resources": ["*"],
        "measurement": ["reads", "writes", "read_time", "write_time", "io_time"],
        "metrics_collection_interval": 60
      },
      "swap": {
        "measurement": ["swap_used", "swap_free", "swap_used_percent"]
      },
      "mem": {
        "measurement": ["mem_used", "mem_cached", "mem_total"],
        "metrics_collection_interval": 1
      },
      "net": {
        "resources": ["eth0"],
        "measurement": ["bytes_sent", "bytes_recv", "drop_in", "drop_out"]
      },
      "netstat": {
        "measurement": ["tcp_established", "tcp_syn_sent", "tcp_close"],
        "metrics_collection_interval": 60
      },
      "processes": {
        "measurement": ["running", "sleeping", "dead"]
      }
    },
    "append_dimensions": {
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}",
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}"
    },
    "aggregation_dimensions": [["ImageId"], ["InstanceId", "InstanceType"], ["d1"], []],
    "force_flush_interval": 30
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/home/watch/logs/antivirus/antivirus.log",
            "log_group_name": "antivirus",
            "log_stream_name": "CICDM_Test",
            "timestamp_format": "%Y-%m-%d %H:%M:%S"
          },
          {
            "file_path": "/home/watch/logs/system/system.log",
            "log_group_name": "ubuntu_ec2_status",
            "log_stream_name": "CICDM_Test",
            "timestamp_format": "%Y-%m-%d %H:%M:%S"
          },
          {
            "file_path": "/home/watch/logs/authCount.log",
            "log_group_name": "ubuntu_ec2_auth",
            "log_stream_name": "auth",
            "timestamp_format": "%Y-%m-%d %H:%M:%S"
          }
        ]
      }
    },
      "log_stream_name": "CICDM_Test_log_stream",
      "force_flush_interval": 15
  }
} 

cat /root/.aws/config
[profile AmazonCloudWatchAgent]
region = us-gov-west-1
output = json
[default]
region = us-east-1

cat /root/.aws/credentials
[AmazonCloudWatchAgent]
aws_access_key_id = *********************************
aws_secret_access_key =********************************************

` There is no configuration in common-config.toml. If the file is configured, the agent cannot be started. Common-config.toml file and regional errors will appear.

# [credentials]
#    shared_credential_profile = "/root/.aws/credentials"
#    shared_credential_file = "{file_name}"

There is another issue: configuring root/.aws/credentials and /root/.aws/config affects the connection of the CodeDeploy agent. I have been using the CloudWatch Logs Agent because it requires configuring credentials in root/.aws/credentials to upload logs to the GovCloud region. However, this affects the connection of the CodeDeploy agent. That's why I wanted to switch to CloudWatch Agent for log uploads, but it seems that it's not working well now, encountering various issues.