Lambda not functioning during the process of Greengrass redeployment


I have a long-lived Lambda function that publishes MQTT messages once every second to the IoT Core. And I can see the messages coming using the MQTT test client (under AWS IoT) in the AWS Console.

However, I noticed that messages are not showing up (at least for quite a while) when I redeploy the Greengrass group. The messages showed up again once the redeployment had been finished. I can reproduce this simply by (re)deploying the current Greengrass group version in the AWS console, without changing anything.

It seems that the Lambda function is frozen during the redeployment process. And this can last half a minute, or even longer than a minute, depending on how long the redeployment takes.

This can cause big problems, especially when it is critical to respond immediately in some production scenarios.

By the way, we are using Greengrass version 1.

Thank you for your help.

asked 3 years ago247 views
3 Answers

I think this is the expected behavior during deployments. We have also realized that Greengrass-aware devices got disconnect once a deployment starts (the core MQTT broker closes all connections).

In general, I would say your solution cannot rely on 100% connectivity. Devices and cores can be offline or unreachable in some periods of time. If you have critical processes that cannot live with the nature of distributed systems, then you'd better handle them at the edge instead of on the cloud.

profile picture
answered 3 years ago

Hi rodmaz, thanks for your insights.

As you mentioned, we do intend to handle them at the edge. The issue is that when we have a new version to redeploy to Greengrass Core at the edge, the Lambda functions running at the edge stops functioning during the redeployment process, which can be as long as a couple of minutes, depending on how long the redeployment process takes.

Just would like to know what would be the best practice if this is a general issue with Greengrass (we are using Greengrass v1 at the moment.)

Edited by: Yuci on Aug 11, 2021 7:03 AM

answered 3 years ago

In Greengrass V1 any deployment starts by shutting everything down, then doing the deployment, and then starting everything back up again. There is no way around this.
If you use Greengrass V2 most deployment do not take anything down unless you are changing versions of your software; even then it is only down for the minimal time between shutting down the current version and starting the newer version.

Hope that helps a bit,

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions