Greengrass V2 - Logmanager doesn't upload components log

Question

Hello,  
  
i try to upload my component logs via the logmanager component.  
  
The Component i wish to upload is: GreengrassLambdaVS  
The Configuration of the logmanager is:   
  
{  
  "reset": \[],  
  "merge": {  
    "logsUploaderConfiguration": {  
      "systemLogsConfiguration": {  
        "uploadToCloudWatch": "true",  
        "minimumLogLevel": "INFO",  
        "deleteLogFileAfterCloudUpload": "false"  
      },  
      "componentLogsConfiguration": \[  
        {  
          "componentName": "GreengrassLambdaVS",  
          "minimumLogLevel": "DEBUG",  
          "logFileDirectoryPath": "/greengrass/v2/logs",  
          "logFileRegex": "GreengrassLambdaVS.log",  
          "deleteLogFileAfterCloudUpload": "false"  
        }  
      ]  
    }  
  }  
}

The Log Manager is deployed sucessfully.   
But in cloudwatch i just get system logs:   
  
The following LogGroup  
/aws/greengrass/GreengrassSystemComponent/eu-west-1/System  
  
Which have a logstream like:   
/2021/03/10/thinggroup/GreengrassCoreGroupVS/GreengrassCoreVS  
  
Is my configuration not correct to upload component logs?  
  
After some minutes i have also an error in greengras.log for the LogManager.  
  
2021-03-10T11:19:03.551Z \[INFO] (pool-2-thread-59) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-update-finish. {serviceName=UpdateSystemPolicyService, currentState=RUNNING}  
2021-03-10T11:19:03.551Z \[INFO] (pool-2-thread-58) com.aws.greengrass.deployment.DeploymentService: deployment-task-execution. Finished deployment task. {deploymentId=arn:aws:greengrass:eu-west-1:153272823058:configuration:thinggroup/GreengrassCoreGroupVS:45, serviceN$  
2021-03-10T11:19:03.553Z \[INFO] (pool-2-thread-58) com.aws.greengrass.componentmanager.ComponentStore: delete-component-start. {componentIdentifier=GreengrassLambdaVS-v11.0.0}  
2021-03-10T11:19:03.557Z \[INFO] (pool-2-thread-58) com.aws.greengrass.componentmanager.ComponentStore: delete-component-finish. {componentIdentifier=GreengrassLambdaVS-v11.0.0}  
2021-03-10T11:19:04.544Z \[INFO] (pool-2-thread-62) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2021-03-10T12:14:04Z. {iotCredentialsPath=/role-aliases/MyGreengrassCoreTokenExchangeRoleAliasVS/credentials}  
2021-03-10T11:19:05.346Z \[ERROR] (pool-2-thread-62) com.aws.greengrass.logmanager.CloudWatchLogsUploader: Invalid token while uploading logs to /aws/greengrass/GreengrassSystemComponent/eu-west-1/System-/2021/03/10/thinggroup/GreengrassCoreGroupVS/GreengrassCoreVS. Ge$  
software.amazon.awssdk.services.cloudwatchlogs.model.InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49616050700184817829627502906959281003958256411131366930 (Service: CloudWatchLogs, Status Code: 400, Request ID:$  
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123)  
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79)  
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59)  
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40)  
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)  
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)  
  
After this error, i also don't get system logs any longer.  
  
Kind Regards  
Stefan

Answer

Maybe you could make that more obvious in the documentation that just rotated log files are uploaded. Thanks.  
  
Any updates on the roadmap for seeing the latest logs? This would be very helpful. Thanks a lot.

Answer

Hi aws-hui,  
Do you mean that Logmanager uploads only rotated files to CloudWatch? Will all ComponentName.log files will be ignored until they are rotated and renamed to ComponentName_SOME_DATE.log?  
We use  this Logmanager component and we can see only events from rotated files for components and greengrass itself. So it appears that it's not possible to have close to real time messages in CloudWatch. It takes hours or days to see them in there.  
But the documentation says that default regex matches log files with and without timestamps in their name and we assumed that events from active logs are also uploaded to CloudWatch. Is there any workaround for this? Is it possible to upload events from active log as soon as they appear?

Answer

Is there a timeline as to when real time logs what the logfileregex configuration setting description implies would happen?

Answer

Hi SwordMaster,  
  
You can find example configuration for logFileRegex here https://docs.aws.amazon.com/greengrass/v2/developerguide/log-manager-component.html#log-manager-component-configuration  
I believe you should use `GreengrassLambdaVSw*.log`. Also only the rotated files like `GreengrassLambdaVS_2020_12_15_17_0.log` will be uploaded to CloudWatch. If you currently have one active log file `GreengrassLambdaVS.log`, it will not be uploaded until rotation.  
  
Errors of `InvalidSequenceTokenException` can happen. As long as log manager retries later and continues uploading logs, I'm not concerned about this error. Can you double check if the log upload resumes with new rotated files?  
  
Thanks,  
Hui  
  
Edited by: aws-hui on Mar 23, 2021 12:24 PM

Answer

Hi,  
Correct, only the already-rotated files will be uploaded. No, there are no workarounds for this as it is intended behavior.  
  
It is on our roadmap to support uploading from the active file.  
  
Cheers,  
Michael

Greengrass V2 - Logmanager doesn't upload components log

Relevant content