Greengrass aws.greengrass.LogManager Invalid token while uploading logs

Question

Hello All,

I am having and issue with aws.greengrass.LogManager on Greengrass, I have tried to redeploy, uninstall and deploy once again, but nothing helps. In greengrass.log I see that there was 3 attempts to run aws.greengrass.LogManager and all of them returned the same error:

2022-08-30T15:12:54.619Z [INFO] (pool-2-thread-83) com.aws.greengrass.logmanager.CloudWatchLogsUploader: Invalid token while uploading logs to /aws/greengrass/UserComponent/eu-central-1/com.example.Project-/2022/08/30/thing/Thing123. Retrying with the expected sequence token in CloudWatch response.. {}
2022-08-30T15:12:54.620Z [ERROR] (pool-2-thread-83) com.aws.greengrass.logmanager.LogManagerService: service-errored. {serviceNam=aws.greengrass.LogManager, currentState=RUNNING}
java.lang.NullPointerException
        at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
        at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
        at com.aws.greengrass.logmanager.CloudWatchLogsUploader.addNextSequenceToken(CloudWatchLogsUploader.java:228)
        at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:156)
        at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
        at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597)
        at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
        at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:623)
        at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:634)
        at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:530)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Configuration I am using:
{
	"logsUploaderConfiguration": {
		"systemLogsConfiguration": {
			"uploadToCloudWatch": "true",
			"minimumLogLevel": "INFO",
			"diskSpaceLimit": "10",
			"diskSpaceLimitUnit": "MB",
			"deleteLogFileAfterCloudUpload": "false"
		},
		"componentLogsConfigurationMap": {
			"com.example.Project": {
				"minimumLogLevel": "INFO",
				"diskSpaceLimit": "10",
				"diskSpaceLimitUnit": "MB",
				"deleteLogFileAfterCloudUpload": "false"
			}
		}
	},
	"periodicUploadIntervalSec": "60"
}

Would appreciate your help.

Answer

Hi,
This exception is related to the PUT in concurrentHashMap, when either the key or value is null. Please check the logGroupName and logStreamName? Those information should also printed as logger message just before this error.  The potential reason is logStreamName is null.

Answer

Hello,

Can you confirm that your Greengrass device role has the correct permissions set? I am wondering if this is a permission issue with Cloudwatch which results in an invalid token being used: https://docs.aws.amazon.com/greengrass/v2/developerguide/log-manager-component.html#log-manager-component-requirements

It may also be due to having multiple sources sending to the same log stream - see https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-invalid-sequence-token/

If that does not resolve the issue, would you be able to provide the full log?

Thanks!

Greengrass aws.greengrass.LogManager Invalid token while uploading logs

Relevant content