AWS Greengrass errors setting jvmOptions

Question

I'm having memory issues with a greengrass core device. Every two days he starts having memory heap space problems. 
The device has these characteristics (4GB RAM and Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux):

```
Arquitectura:                        x86_64
modo(s) de operación de las CPUs:    32-bit, 64-bit
Orden de los bytes:                  Little Endian
Tamaños de las direcciones:          39 bits physical, 48 bits virtual
CPU(s):                              2
Lista de la(s) CPU(s) en línea:      0,1
Hilo(s) de procesamiento por núcleo: 1
Núcleo(s) por «socket»:              2
«Socket(s)»                          1
Modo(s) NUMA:                        1
ID de fabricante:                    GenuineIntel
Familia de CPU:                      6
Modelo:                              92
Nombre del modelo:                   Intel(R) Celeron(R) CPU N3350 @ 1.10GHz
Revisión:                            9
CPU MHz:                             1591.854
CPU MHz máx.:                        2400,0000
CPU MHz mín.:                        800,0000
BogoMIPS:                            2188.80
Virtualización:                      VT-x
Caché L1d:                           48 KiB
Caché L1i:                           64 KiB
Caché L2:                            2 MiB
CPU(s) del nodo NUMA 0:              0,1

```

It has the following components installed:
eu.bluece.SubscribeParsePublishIoT	1.0.1 (custom component)
aws.greengrass.Cli	2.9.1
aws.greengrass.Cloudwatch	3.1.0
aws.greengrass.DeviceDefender	3.1.1
aws.greengrass.LogManager	2.3.0
aws.greengrass.Nucleus	2.9.1
aws.greengrass.SystemsManagerAgent	1.0.0
aws.greengrass.clientdevices.Auth	2.3.1
aws.greengrass.clientdevices.IPDetector	2.1.5
aws.greengrass.clientdevices.mqtt.Bridge	2.2.4
aws.greengrass.clientdevices.mqtt.Moquette	2.3.0

Looking for information about, I found this recomendation:  https://docs.amazonaws.cn/en_us/greengrass/v2/developerguide/configure-greengrass-core-v2.html#jvm-tuning but I'm having problems to apply them.

In my nucleus config I'm trying the following configuration:

```
{
	"jvmOptions": "-Xmx32m -XX:+UseSerialGC -Xint",
	"logging": {
		"level": "INFO",
		"format": "TEXT"
	}
}
```
When the deployment restarts I find the following errors:

```
2023-01-16T23:50:30.243Z [ERROR] (main) com.aws.greengrass.lifecyclemanager.Kernel: Cannot load service. {}
com.aws.greengrass.lifecyclemanager.exceptions.ServiceLoadException: Unable to load aws.greengrass.clientdevices.Auth as a plugin
        at com.aws.greengrass.lifecyclemanager.Kernel.locateExternalPlugin(Kernel.java:568)
        at com.aws.greengrass.lifecyclemanager.Kernel.locateGreengrassService(Kernel.java:440)
        at com.aws.greengrass.lifecyclemanager.Kernel.lambda$locateIgnoreError$3(Kernel.java:390)
        at com.aws.greengrass.dependency.Context$Value.computeObjectIfEmpty(Context.java:562)
        at com.aws.greengrass.lifecyclemanager.Kernel.locateIgnoreError(Kernel.java:388)
        at com.aws.greengrass.lifecyclemanager.Kernel.locateGreengrassService(Kernel.java:417)
        at com.aws.greengrass.lifecyclemanager.Kernel.lambda$locateIgnoreError$3(Kernel.java:390)
        at com.aws.greengrass.dependency.Context$Value.computeObjectIfEmpty(Context.java:562)
        at com.aws.greengrass.lifecyclemanager.Kernel.locateIgnoreError(Kernel.java:388)
        at com.aws.greengrass.lifecyclemanager.Kernel.locateGreengrassService(Kernel.java:417)
        at com.aws.greengrass.lifecyclemanager.Kernel.lambda$locateIgnoreError$3(Kernel.java:390)
        at com.aws.greengrass.dependency.Context$Value.computeObjectIfEmpty(Context.java:562)
        at com.aws.greengrass.lifecyclemanager.Kernel.locateIgnoreError(Kernel.java:388)
        at com.aws.greengrass.lifecyclemanager.KernelLifecycle.launch(KernelLifecycle.java:167)
        at com.aws.greengrass.lifecyclemanager.Kernel.launch(Kernel.java:244)
        at com.aws.greengrass.easysetup.GreengrassSetup.performSetup(GreengrassSetup.java:349)
        at com.aws.greengrass.easysetup.GreengrassSetup.main(GreengrassSetup.java:274)
Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
        at io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1596)
        at com.aws.greengrass.dependency.EZPlugins.lambda$loadPlugin$1(EZPlugins.java:124)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at com.aws.greengrass.dependency.EZPlugins.loadPlugin(EZPlugins.java:116)
        at com.aws.greengrass.lifecyclemanager.Kernel.locateExternalPlugin(Kernel.java:553)
        ... 16 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
        at io.github.lukehutch.fastclasspathscanner.scanner.ClassfileBinaryParser.readMore(ClassfileBinaryParser.java:110)
        at io.github.lukehutch.fastclasspathscanner.scanner.ClassfileBinaryParser.skip(ClassfileBinaryParser.java:203)
        at io.github.lukehutch.fastclasspathscanner.scanner.ClassfileBinaryParser.readClassInfoFromClassfileHeader(ClassfileBinaryParser.java:858)
        at io.github.lukehutch.fastclasspathscanner.scanner.ClasspathElement.parseClassfiles(ClasspathElement.java:272)
        at io.github.lukehutch.fastclasspathscanner.scanner.Scanner$6.processWorkUnit(Scanner.java:417)
        at io.github.lukehutch.fastclasspathscanner.scanner.Scanner$6.processWorkUnit(Scanner.java:411)
        at io.github.lukehutch.fastclasspathscanner.utils.WorkQueue.runWorkLoop(WorkQueue.java:187)
        at io.github.lukehutch.fastclasspathscanner.utils.WorkQueue.runWorkQueue(WorkQueue.java:108)
        at io.github.lukehutch.fastclasspathscanner.utils.WorkQueue.runWorkQueue(WorkQueue.java:122)
        at io.github.lukehutch.fastclasspathscanner.scanner.Scanner.call(Scanner.java:410)
        at io.github.lukehutch.fastclasspathscanner.scanner.Scanner.call(Scanner.java:55)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

```

```
2023-01-16T23:50:41.340Z [ERROR] (main) com.aws.greengrass.lifecyclemanager.Kernel: Cannot load service. {}
com.aws.greengrass.lifecyclemanager.exceptions.ServiceLoadException: Unable to load aws.greengrass.clientdevices.mqtt.Moquette as a plugin

```

```
2023-01-16T23:50:51.478Z [ERROR] (main) com.aws.greengrass.lifecyclemanager.Kernel: Cannot load service. {}
com.aws.greengrass.lifecyclemanager.exceptions.ServiceLoadException: Unable to load aws.greengrass.Cli as a plugin

```
All of them don't load by Java heap space. Also this exception:
```
2023-01-16T23:51:25.839Z [ERROR] (pool-2-thread-28) com.aws.greengrass.deployment.DeploymentService: deployment-errored. {deploymentId=e2b6289d-023c-46c3-9c6e-a800de0e0294, serviceName=DeploymentService, currentState=RUNNING}
com.aws.greengrass.deployment.exceptions.ServiceUpdateException: Service aws.greengrass.clientdevices.Auth in broken state after deployment
```
If I restart greengrass.service it seems to start without exceptions. But If I make a new revision of deployment fails again. The same thing happens on another device with 8 gigabytes of memory if I configure it in the kernel jvmoptions.

¿Is there any way to set up jvmoptions cleaner? without exceptions?

cespar · Answer

I still have memory problems.
As I said in the first post, I'm testing on two devices, in the 4 GB I have problems at 10/12 hours, in the 8 GB it lasts a few days.
I have tried to change the parameters of the jvm but the system becomes very slow. Can you help me adjust the java parameters? I have debian 11 installed.

Below I leave the error output when the problems start:
```
2023-01-22T13:01:48.699Z [ERROR] (MQTT Con: mqtt-bridge-bc6y1qg0dg1) com.aws.greengrass.lifecyclemanager.KernelExceptionHandler: uncaught-exception. {thread=Thread[MQTT Con: mqtt-bridge-bc6y1qg0dg1,5,main]}java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.base/java.lang.Thread.start0(Native Method)
        at java.base/java.lang.Thread.start(Thread.java:798)
        at org.eclipse.paho.client.mqttv3.internal.CommsSender.start(CommsSender.java:72)
        at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:728)
        at java.base/java.lang.Thread.run(Thread.java:829)
```
Regards,
César.

MichaelDombrowski-AWS · Answer

You have configured Greengrass to allow it to use only 32MB of heap space. This is not enough space for Greengrass to operate, particularly when you're adding more plugins which will use additional heap space.

You must increase this value to avoid the Java out of memory error. I would suggest a bare minimum of 64MB, though higher would be safer if you're going to use many plugins.

If you have enough memory on the device, which seems to be the case with 4GB or 8GB, then do not set these JVM options, they are not needed and will cause Greengrass to run slower.

cespar · Answer

Thanks for your quick response.

I think I remember that with 64mb it also gave me those exceptions. I just put 128mb and it seems to work fine.

Regarding your last sentence, I agree that 4gb should be enough, but every 2/3 days I run out of memory. At the level of the operating system, can the management of memory by the jvm be improved?

With command free -h I have this output, I don't undesrstand the difference between free and aviable memory:

```
$ free -h
            total     used   free      shared  buff/cache   available
Mem:   3,7Gi   1,3Gi   839Mi  10Mi       1,6Gi       2,1Gi
Swap:  976Mi  1,0Mi  975Mi
```
What usually happens is that the free memory tends to 0 and the problems begin

Regards.

AWS Greengrass errors setting jvmOptions

Add your answer

Relevant content