Stream Manager Broken during initialization

0

We have been using stream manager for a while now but have not encountered this error before. We would like to know how to prevent this issue form happening and how to best recover form it. Full error outout inclded below, but the main log lines apear to indicate that an expected record does not exist: StreamManagerService: Error initializing and org.mapdb.DBException$GetVoid: Record does not exist, recid=1

Version Info: aws.greengrass.Nucleus: 2.10.1 StreamManager: 2.1.5 Platform: linux/arm32

Full log output:

{"exitCode":"0","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702523875,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"2023 Oct 30 21:49:32,939 \u001B[1;31m[ERROR]\u001B[m (main) com.amazonaws.iot.greengrass.streammanager.StreamManagerService: StreamManagerService: Error initializing","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573290,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"org.mapdb.DBException$GetVoid: Record does not exist, recid=1","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573292,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at org.mapdb.StoreWAL.getIndexVal(StoreWAL.kt:169) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573293,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at org.mapdb.StoreWAL.get(StoreWAL.kt:499) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573301,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at org.mapdb.DB.nameCatalogLoadLocked(DB.kt:350) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573304,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at org.mapdb.DB.nameCatalogLoad(DB.kt:343) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573305,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at org.mapdb.DB.nameCatalogVerifyGetMessages(DB.kt:1841) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573305,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at org.mapdb.DB.<init>(DB.kt:272) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573306,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at org.mapdb.DB.<init>(DB.kt:40) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573361,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at org.mapdb.DBMaker$Maker.make(DBMaker.kt:477) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573362,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.DataStoreModule.mapDb(DataStoreModule.java:92) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573374,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.DataStoreModule_MapDbFactory.get(DataStoreModule_MapDbFactory.java:29) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573380,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.DataStoreModule_MapDbFactory.get(DataStoreModule_MapDbFactory.java:10) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573381,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at dagger.internal.DoubleCheck.get(DoubleCheck.java:47) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573382,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.DataStoreModule_MessageStreamDefDaoFactory.get(DataStoreModule_MessageStreamDefDaoFactory.java:32) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573382,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.DataStoreModule_MessageStreamDefDaoFactory.get(DataStoreModule_MessageStreamDefDaoFactory.java:11) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573399,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at dagger.internal.DoubleCheck.get(DoubleCheck.java:47) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573400,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.DataStoreModule_LogStoreFactory.get(DataStoreModule_LogStoreFactory.java:59) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573401,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.DataStoreModule_LogStoreFactory.get(DataStoreModule_LogStoreFactory.java:15) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573412,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at dagger.internal.DoubleCheck.get(DoubleCheck.java:47) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573431,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.StreamServerModule_StreamServerFactory.get(StreamServerModule_StreamServerFactory.java:69) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573432,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.StreamServerModule_StreamServerFactory.get(StreamServerModule_StreamServerFactory.java:16) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573433,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at dagger.internal.DoubleCheck.get(DoubleCheck.java:47) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573434,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.dagger.DaggerStreamManagerServiceComponent.streamServer(DaggerStreamManagerServiceComponent.java:284) ~[AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573435,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"at com.amazonaws.iot.greengrass.streammanager.StreamManagerService.main(StreamManagerService.java:44) [AWSGreengrassGreenlake-1.0-super.jar:?]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702573435,"cause":null}
{"thread":"Copier","level":"WARN","eventType":"stderr","message":"Oct 30, 2023 9:49:37 PM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onConnectionSetup","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702577201,"cause":null}
{"thread":"Copier","level":"WARN","eventType":"stderr","message":"INFO: Socket connection /greengrass/v2/ipc.socket:8033 to server result [AWS_ERROR_SUCCESS]","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702577202,"cause":null}
{"thread":"Copier","level":"WARN","eventType":"stderr","message":"Oct 30, 2023 9:49:39 PM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onProtocolMessage","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702579659,"cause":null}
{"thread":"Copier","level":"WARN","eventType":"stderr","message":"INFO: Connection established with event stream RPC server","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING"},"loggerName":"aws.greengrass.StreamManager","timestamp":1698702579660,"cause":null}
{"thread":"pool-2-thread-56","level":"INFO","eventType":"shell-runner-start","message":"","contexts":{"scriptName":"services.aws.greengrass.StreamManager.lifecycle.startup.script","serviceName":"aws.greengrass.StreamManager","currentState":"STARTING","command":"[\"java -Xmx64m -cp /greengrass/v2/packages/artifacts/aws.greengrass.StreamManage...\"]"},"loggerN
posta 9 mesi fa222 visualizzazioni
1 Risposta
0

Hello @ashprengel

It looks like you are hitting a very similar issue that you had in the past which is due to disk corruption.

https://repost.aws/questions/QUySOn9sdoSWCDAuMiV9P9AA/streammanager-gets-into-broken-state-due-to-dbexception-volumeioerror

I would very highly suggest that you take the recommendations in that previous answer if you have not already done that. If stream manager loses power while writing, it is possible for this metadata store corruption to occur. If you can safely shutdown the device instead of cutting power from it, that would prevent this issue.

Cheers,

Michael

AWS
ESPERTO
con risposta 8 mesi fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande