shadow manager configuration

0

I create a named shadow on each of my greengrass devices on the server, for the devices to use (call it config1)

i would like to use these named shadows locally on the greengrass device so i am trying to install the shadow manager
in the documentation regarding the configuration it shows an example configuration like:

{
"synchronize":{
"coreThing":{
"classic":true,
"namedShadows":[
"MyCoreShadowA",
"MyCoreShadowB"
]
},
"shadowDocuments":[
{
"thingName":"MyDevice1",
"classic":false,
"namedShadows":[
"MyShadowA",
"MyShadowB"
]
},
{
"thingName":"MyDevice2",
"classic":true,
"namedShadows":[ ]
}
]
},
"rateLimits": {
"maxOutboundSyncUpdatesPerSecond":100,
"maxTotalLocalRequestsRate":200,
"maxLocalRequestsPerSecondPerThing":20
},
"shadowDocumentSizeLimitBytes":8192
}

if i would like to use shadow manager to syncronise every config1 shadow for all my devices, do i need to list all my devices in the shadowDocuments ? this doesn't seem right as we may add and remove devices from a deployment at any stage.

I've tried to only add the config1 named shadow in 'coreThing' but it immediately deletes the server created named shadow as soon as the device connects:

even if i don't specify a configuration, the server created shadows get deleted by the shadow manager:

2021-10-07T00:53:29.666Z [INFO] (pool-2-thread-26) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Executing sync request. {Type=LocalUpdateSyncRequest, thing name=smartdvr-1423019132001, shadow name=config1}
2021-10-07T00:53:29.682Z [WARN] (pool-2-thread-26) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Received conflict when processing request. Retrying as a full sync. {thing name=smartdvr-1423019132001, shadow name=config1}
software.amazon.awssdk.aws.greengrass.model.ConflictError: Missed update(s) from the cloud
at com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest.execute(LocalUpdateSyncRequest.java:142)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$0(SyncHandler.java:136)
at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:49)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$1(SyncHandler.java:134)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.syncLoop(SyncHandler.java:270)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

2021-10-07T00:53:29.683Z [INFO] (pool-2-thread-26) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Executing sync request. {Type=FullShadowSyncRequest, thing name=smartdvr-1423019132001, shadow name=config1}
2021-10-07T00:53:30Z [INFO] (pool-2-thread-26) com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest: Deleting cloud shadow document. {thing name=smartdvr-1423019132001, shadow name=config1}
2021-10-07T00:53:30.211Z [INFO] (pool-2-thread-26) com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl: Updating sync info. {thing name=smartdvr-1423019132001, shadow name=config1, cloud-version=121, local-version=0}
2021-10-07T00:53:30.245Z [INFO] (pool-2-thread-26) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Executing sync request. {Type=LocalDeleteSyncRequest, thing name=smartdvr-1423019132001, shadow name=config1}
2021-10-07T00:53:30.259Z [WARN] (pool-2-thread-26) com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler: handle-delete-thing-shadow. Unable to process delete shadow since shadow does not exist. {thing name=smartdvr-1423019132001, shadow name=config1}
software.amazon.awssdk.aws.greengrass.model.ResourceNotFoundError: No shadow found
at com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler.lambda$handleRequest$0(DeleteThingShadowRequestHandler.java:106)
at com.aws.greengrass.ipc.common.ExceptionUtil.translateExceptions(ExceptionUtil.java:33)
at com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler.handleRequest(DeleteThingShadowRequestHandler.java:87)
at com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest.execute(LocalDeleteSyncRequest.java:74)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$0(SyncHandler.java:136)
at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:49)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$1(SyncHandler.java:134)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.syncLoop(SyncHandler.java:270)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

2021-10-07T00:53:30.260Z [INFO] (pool-2-thread-26) com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest: Attempted to delete shadow that was already deleted. {thing name=smartdvr-1423019132001, shadow name=config1}
software.amazon.awssdk.aws.greengrass.model.ResourceNotFoundError: No shadow found
at com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler.lambda$handleRequest$0(DeleteThingShadowRequestHandler.java:106)
at com.aws.greengrass.ipc.common.ExceptionUtil.translateExceptions(ExceptionUtil.java:33)
at com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler.handleRequest(DeleteThingShadowRequestHandler.java:87)
at com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest.execute(LocalDeleteSyncRequest.java:74)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$0(SyncHandler.java:136)
at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:49)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$1(SyncHandler.java:134)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.syncLoop(SyncHandler.java:270)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

Edited by: clogwog on Oct 6, 2021 5:55 PM

clogwog
asked 2 years ago425 views
5 Answers
0

Hello,

Thank you for your interested in Greengrass V2. For the first question of whether "you would need to use shadow manager to syncronise every config1 shadow for all my devices, do i need to list all my devices in the shadowDocuments?", the answer is yes. For the current version of Shadow manager you would need to provide the complete list of shadows that need to be synced in the synchronize configuration. That said, we are working on adding support for wild cards in the synchronize configuration so alleviate the issue you mentioned.

For the issue you mentioned where the shadow is getting deleted, we are already aware of this issue and are working actively to resolve it. We plan to release have a patch release with that fix.

Cheers,
Nikkhil

AWS
answered 2 years ago
0

Thank you for that reply,

will the wildcards be on the 'namedShadows' level or the 'thingName' level

Is there a bug ticket that i can follow for the deleting of the named shadow ?

Also, once i have installed the shadow manager with the configuration, and i add the right configuration for the topics, can i list the named topics directly in my lambda using

import greengrasssdk
client = greengrasssdk.client("iot-data")
device = os.environ\['AWS_IOT_THING_NAME']
the_shadow = client.get_thing_shadow(thingName = device, shadowName = "config1")

or do i also need to add

"accessControl": {
"aws.greengrass.ShadowManager": {
"mtdshadowconfiglambda-dev-sync_remote_config::1": {
"policyDescription": "allow access to config# shadow",
"operations": [
"aws.greengrass#GetThingShadow",
"aws.greengrass#UpdateThingShadow",
"aws.greengrass#ListNamedShadowsForThing"
],
"resources": [
"$aws/things/thingName/shadow/name/config1"
]
}
}
}

to the lambda confiuration ?

I'm asking because even after all this i still get errors.:

2021-10-07T06:20:14.502Z [ERROR] (nioEventLoopGroup-5-4) com.aws.greengrass.lambdamanager.LambdaRequestHandler: {}
com.aws.greengrass.lambdamanager.LambdaNotFoundException: Lambda arn:aws:lambda:::function:GGShadowService does not exist
at com.aws.greengrass.lambdamanager.WorkManager.appendWorkToLambda(WorkManager.java:297)
at com.aws.greengrass.lambdamanager.LambdaRequestHandler.handlePostWorkItem(LambdaRequestHandler.java:268)
at com.aws.greengrass.lambdamanager.LambdaRequestHandler.channelRead(LambdaRequestHandler.java:113)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at com.aws.greengrass.lambdamanager.lib.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at com.aws.greengrass.lambdamanager.lib.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at com.aws.greengrass.lambdamanager.lib.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at com.aws.greengrass.lambdamanager.lib.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at com.aws.greengrass.lambdamanager.lib.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at com.aws.greengrass.lambdamanager.lib.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at com.aws.greengrass.lambdamanager.lib.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)

==> mtdshadowconfiglambda-dev-sync_remote_config.log <==
2021-10-07T06:20:14.508Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: ipc_client.py:64,HTTP Error 404:Not Found, b'Lambda arn:aws:lambda:::function:GGShadowService does not exist'. {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.510Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: Lambda.py:149,HTTP Error 404: Not Found. {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.510Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: Traceback (most recent call last):. {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.510Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: File "/runtime/python/greengrass_ipc_python_sdk/ipc_client.py", line 59, in wrapped. {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.510Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: return func(*args, **kwargs). {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.510Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: File "/runtime/python/greengrass_ipc_python_sdk/ipc_client.py", line 192, in post_work. {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.510Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: response = urlopen(request). {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.510Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen. {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.510Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: return opener.open(url, data, timeout). {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.511Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: File "/usr/lib/python3.6/urllib/request.py", line 532, in open. {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.511Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: response = meth(req, response). {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.511Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response. {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}
2021-10-07T06:20:14.511Z [ERROR] (pool-2-thread-91) mtdshadowconfiglambda-dev-sync_remote_config: 'http', request, response, code, msg, hdrs). {serviceInstance=0, serviceName=mtdshadowconfiglambda-dev-sync_remote_config, currentState=RUNNING}

Is there a complete walkthrough for accessing the named shadows in greengrass v2 ?

Edited by: clogwog on Oct 7, 2021 5:21 PM

Edited by: clogwog on Oct 7, 2021 5:22 PM

clogwog
answered 2 years ago
0

The wildcard design has not been completely etched out yet. But we will keep this use case in mind when finalizing it.

There is no ticket for the deleting shadow issue. But I have a PR out for that fix. Once that is merged in, we will plan to release a patch release for Shadow Manager. PR link -> https://github.com/aws-greengrass/aws-greengrass-shadow-manager/pull/96

For the other question regarding, you seem to be using the GG v1 SDK. Currently, we do not support using the GG v1 SDK for shadow operation in GG v2. I would suggest using the GG v2 SDK to perform shadow operations in GG v2. We have a roadmap item to support GG v1 SDK for shadow in the future.

Example code for the GG v2 SDK can be found here -> https://docs.aws.amazon.com/greengrass/v2/developerguide/ipc-local-shadows.html

Also, is there any reason you are using a lambda instead of a python script component in GG v2? We recommend using the native GG v2 component instead of a lambda.

Cheers,
Nikkhil

AWS
answered 2 years ago
0

Hello Nokkhill,

thank you for that.

I didn't realise i was using the V1 api instead of the v2. I've switched now and am reading up on the install lifecycle step to install python3 -m pip install awsiotsdk

Using a lambda because that is what i am used to, but since you challenged it i will go to using a straight python script. not sure if it makes any difference , but it may be a bit easier.

thank you for your help !! Also thank you for jumping on the deletion issue, much appreciated !!

clogwog
answered 2 years ago
0

Hi, I am also getting this error : Lambda arn:aws:lambda:::function:GGShadowService does not exist.

Now, I have 2 questions :

i) How can I perform Shadow operation in GGV2 ? ii) This document https://docs.aws.amazon.com/greengrass/v2/developerguide/ipc-local-shadows.html does not contain SDK for Nodejs and my entire code (of Lambda Component which runs on Thin Client on which GGV2 is installed) is in Nodejs. How can I proceed ?

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions