CloudWatch Unified Agent custom namespace not showing up on CloudWatch Metrics

We have previously succeeded in running CWUA on our Auto Scaling Group Ubuntu EC2 servers with custom configuration from SSM Parameter store. The parameter value looks like

{
	"agent": {
		"metrics_collection_interval": 60,
		"run_as_user": "root"
	},
	"metrics": {
		"namespace": "cdk-sample-asg-ASG1-asg-exp",
		"append_dimensions": {
			"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
			"InstanceId": "${aws:InstanceId}",
			"InstanceType": "${aws:InstanceType}"
		},
		"aggregation_dimensions": [ [ "AutoScalingGroupName" ] ],
		"metrics_collected": {
			"mem": {
				"measurement": [
					{
						"name": "mem_used_percent",
						"unit": "Percent"
					}
				],
				"metrics_collection_interval": 60
			}
		}
	}
}

and the user data script in the launch template to pick it up looks like

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb -O /tmp/amazon-cloudwatch-agent.deb
dpkg -i /tmp/amazon-cloudwatch-agent.deb
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:AmazonCloudWatch-cdk-sample-asg-ASG1-asg-exp

All these configuration and deployment were achieved with CDK. Now we're generalising into more reusable constructs with the same setup, but the test deployment for some reason does not show the custom ASG namespace in CloudWatch Metrics.

Initially I thought it might be because Detailed monitoring wasn't enabled but changing that did not improve the situation. What other obstacles can prevent the custom namespace from showing up?

The system log shows CWUA successfully installed and configured.

[   54.811101] cloud-init[1274]: 2022-08-12 09:59:17 (6.15 MB/s) - ‘/tmp/amazon-cloudwatch-agent.deb’ saved [57412840/57412840]
[   54.836742] cloud-init[1274]: Selecting previously unselected package amazon-cloudwatch-agent.
[   54.864478] cloud-init[1274]: (Reading database ... 65339 files and directories currently installed.)
[   54.867059] cloud-init[1274]: Preparing to unpack .../amazon-cloudwatch-agent.deb ...
[   54.896405] cloud-init[1274]: create group cwagent, result: 0
[   54.922558] cloud-init[1274]: create user cwagent, result: 0
[   54.940742] cloud-init[1274]: create group aoc, result: 0
[   54.960169] cloud-init[1274]: create user aoc, result: 0
[   54.961249] cloud-init[1274]: Unpacking amazon-cloudwatch-agent (1.247354.0b251981-1) ...
[   56.464911] cloud-init[1274]: Setting up amazon-cloudwatch-agent (1.247354.0b251981-1) ...
[   56.534388] cloud-init[1274]: ****** processing amazon-cloudwatch-agent ******
[   56.534654] cloud-init[1274]: /opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:AmazonCloudWatch-cdk-sample-asg-ASG1-asg-exp --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
[   56.540078] cloud-init[1274]: I! Trying to detect region from ec2
[   56.540893] cloud-init[1274]: D! [EC2] Found active network interface
[   56.544437] cloud-init[1274]: Region: ap-southeast-1
[   56.544580] cloud-init[1274]: credsConfig: map[]
[   56.605107] cloud-init[1274]: Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ssm_AmazonCloudWatch-cdk-sample-asg-ASG1-asg-exp.tmp
[   56.608867] cloud-init[1274]: Start configuration validation...
[   56.609027] cloud-init[1274]: /opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
[   56.618549] cloud-init[1274]: 2022/08/12 09:59:19 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ssm_AmazonCloudWatch-cdk-sample-asg-ASG1-asg-exp.tmp ...
[   56.621567] cloud-init[1274]: 2022/08/12 09:59:19 I! Valid Json input schema.
[   56.622296] cloud-init[1274]: I! Detecting run_as_user...
[   56.622471] cloud-init[1274]: I! Trying to detect region from ec2
[   56.622597] cloud-init[1274]: D! [EC2] Found active network interface
[   56.626058] cloud-init[1274]: No csm configuration found.
[   56.626169] cloud-init[1274]: No log configuration found.
[   56.626318] cloud-init[1274]: Configuration validation first phase succeeded
[   56.628994] cloud-init[1274]: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
[   56.681402] cloud-init[1274]: Configuration validation second phase succeeded
[   56.681576] cloud-init[1274]: Configuration validation succeeded
[   56.695308] cloud-init[1274]: amazon-cloudwatch-agent has already been stopped
[   56.950624] cloud-init[1274]: Created symlink /etc/systemd/system/multi-user.target.wants/amazon-cloudwatch-agent.service → /etc/systemd/system/amazon-cloudwatch-agent.service.
[  OK  ] Started Amazon CloudWatch Agent.

UPDATE

Comparing to the instances that have worked, I notice some extra actions missing in the older version (circa April 2022).

[   30.714094] cloud-init[851]: 2022-04-27 17:02:49 (4.98 MB/s) - ‘/tmp/amazon-cloudwatch-agent.deb’ saved [54613346/54613346]


[   32.203984] cloud-init[851]: (Reading database ... 110768 files and directories currently installed.)
[   32.208150] cloud-init[851]: Preparing to unpack .../amazon-cloudwatch-agent.deb ...
[   32.256865] cloud-init[851]: ****** processing cwagent-otel-collector ******
[   32.261143] cloud-init[851]: cwagent-otel-collector has already been stopped
[   32.600172] cloud-init[851]: ****** processing amazon-cloudwatch-agent ******
         Stopping Amazon CloudWatch Agent...
[  OK  ] Stopped Amazon CloudWatch Agent.
[   33.518263] cloud-init[851]: Unpacking amazon-cloudwatch-agent (1.247350.0b251780-1) over (1.247350.0b251780-1) ...
[   35.821820] cloud-init[851]: Setting up amazon-cloudwatch-agent (1.247350.0b251780-1) ...
[   35.892578] cloud-init[851]: ****** processing amazon-cloudwatch-agent ******
[   35.893649] cloud-init[851]: /opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:AmazonCloudWatch-Original-app-asg-dev --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
[   35.902414] cloud-init[851]: Region: ap-southeast-1
[   35.903189] cloud-init[851]: credsConfig: map[]
[   36.010690] cloud-init[851]: Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ssm_AmazonCloudWatch-Original-app-asg-dev.tmp
[   36.015002] cloud-init[851]: Start configuration validation...
[   36.016156] cloud-init[851]: /opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
[   36.022624] cloud-init[851]: 2022/04/27 17:02:54 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ssm_AmazonCloudWatch-Original-app-asg-dev.tmp ...
[   36.025894] cloud-init[851]: Valid Json input schema.
[   36.027002] cloud-init[851]: I! Detecting run_as_user...
[   36.030131] cloud-init[851]: No csm configuration found.
[   36.031144] cloud-init[851]: No log configuration found.
[   36.032190] cloud-init[851]: Configuration validation first phase succeeded
[   36.033524] cloud-init[851]: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
[   36.065638] cloud-init[851]: Configuration validation second phase succeeded
[   36.066681] cloud-init[851]: Configuration validation succeeded
[   36.076898] cloud-init[851]: amazon-cloudwatch-agent has already been stopped
[  OK  ] Started Amazon CloudWatch Agent.

There are no additional users and groups created, and no final symlink created between the service files.

However I am not experienced enough with Linux to properly grasp the significance of these differences and how they might cause report failure.