Greengrass v2 - moquette - Empty server certificate chain

It looks the Moquette is using
/greengrass/v2/work/aws.greengrass.clientdevices.mqtt.Moquette/keystore.jks
for SSL certificates

I was trying to verify what certificates are imported there, however password is required.
Listing certs in keystore shows:

keytool -list -v -keystore /greengrass/v2/work/aws.greengrass.clientdevices.mqtt.Moquette/keystore.jks
Enter keystore password:

***************** WARNING WARNING WARNING *****************
** The integrity of the information stored in your keystore **
** has NOT been verified! In order to verify its integrity, **
** you must provide your keystore password. **
***************** WARNING WARNING WARNING *****************

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: pk
Creation date: 3 Sep 2021
Entry type: PrivateKeyEntry
Certificate chain length: 0

Can anyone check /greengrass/v2/work/aws.greengrass.clientdevices.mqtt.Moquette/keystore.jks of his Greengrass deployment?

Hi,
Thanks for using Greengrass.
Which address and port are you connection to from your ESP32?
The supported way to get the CA and the address is not by copying /greengrass/v2/work/aws.greengrass.clientdevices.Auth/ca.pem. The client device is expected to use the discovery API to get the broker information.

See this tutorial to learn about discovery and broker-client device interaction: https://docs.aws.amazon.com/greengrass/v2/developerguide/greengrass-discover-api.html
For the discovery to work, client device has to be associated to the core beforehand. https://docs.aws.amazon.com/greengrass/v2/developerguide/associate-client-devices.html

After following this workflow, if you still get the same issue, please reply with greengrass.log attached.

In the meanwhile, for the logs snippet you provided, can you provide the complete greengrass.log file?

To check the server certificate being used by the broker, can you run the following command:

openssl s_client -showcerts -connect localhost:8883

This assumes, you have not configured the server port to be something different than 8883.

Hello
My device is associated with Core Device and I'm able to use discovery API to get CA certificate and IP address of Greengrass device.
I'm using Arduino as follows to connect to IOT Core endpoint

#include "WiFiClientSecure.h"
#include <MQTTClient.h> -> https://github.com/256dpi/lwmqtt

WiFiClientSecure net = WiFiClientSecure();
MQTTClient client = MQTTClient(256);

net.setCACert(AWS_CERT_CA); <- for IoT Core this is AmazonRootCA1.pem, for Greengrass it's CA cert from discovery API
net.setCertificate(AWS_CERT_CRT); <- Device cert
net.setPrivateKey(AWS_CERT_PRIVATE); <- Device private key
net.setInsecure(); <- temporary enabled for troubleshooting

while (!client.connect(THINGNAME)) {
Serial.print(".");
delay(500);
Serial.println(client.lastError());
Serial.println(client.returnCode());
}

For AWS IoT core endpoint this code is working.
For Greengrass it doesn't with error showed in first post.

I also tested Greengrass core with Python test scripts - test passed - I was able to connect to Greengrass core with client cert and key

greengrass.log for ESP32:
2021-09-03T19:23:03.178Z [DEBUG] (nioEventLoopGroup-3-3) io.moquette.broker.AutoFlushHandler: Initializing autoflush handler on channel [id: 0xbc5f1c8d, L:/192.168.0.148:8883 - R:/192.168.0.143:53004] Cid: null. {}
2021-09-03T19:23:03.625Z [DEBUG] (nioEventLoopGroup-3-3) io.moquette.broker.MQTTConnection: readCompleted client CId: null. {}
2021-09-03T19:23:03.739Z [ERROR] (nioEventLoopGroup-3-3) io.moquette.broker.NewNettyMQTTHandler: Unexpected exception while processing MQTT message. Closing Netty channel. CId=null. {}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain

greengrass.log for Python test:
2021-09-03T19:23:33.380Z [DEBUG] (nioEventLoopGroup-3-1) io.moquette.broker.AutoFlushHandler: Initializing autoflush handler on channel [id: 0xddaefe2f, L:/192.168.0.148:8883 - R:/192.168.0.150:55917] Cid: null. {}
2021-09-03T19:23:33.601Z [DEBUG] (nioEventLoopGroup-3-1) io.moquette.broker.MQTTConnection: readCompleted client CId: null. {}
2021-09-03T19:23:33.608Z [INFO] (nioEventLoopGroup-3-1) com.aws.greengrass.device.DeviceAuthClient: Creating new session. {}
2021-09-03T19:23:33.611Z [DEBUG] (nioEventLoopGroup-3-1) com.aws.greengrass.device.DeviceAuthClient: Certificate was not issued by local CA. {}
2021-09-03T19:23:33.613Z [DEBUG] (nioEventLoopGroup-3-1) software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain: Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumRequiredInterceptor@9a16d6, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@152af19]. {}
2021-09-03T19:23:33.619Z [DEBUG] (nioEventLoopGroup-3-1) software.amazon.awssdk.request: Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=greengrass-ats.iot.eu-west-1.amazonaws.com, port=8443, encodedPath=/greengrass/v2/verifyClientDeviceIdentity, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[]). {}

According to IoT Core and MQTT broker (Moquette) documentation, MQTT versions on both endpoints are the same MQTT v3.1.1
I also checked TLS of IoT Core and MQTT broker (Moquette) - except certificates, SSL sessions are same

I have no idea why with IoT Core endpoint ESP32 can connect and exchange messages, but with Greengrass it's failing.

I'll check this library for greengrass communication
https://github.com/aws-samples/arduino-aws-greengrass-iot

I was able to connect ESP32 to Greengrass with https://github.com/aws-samples/arduino-aws-greengrass-iot

I'm reopening the thread since I'm not able to connect my zigbee2mqtt service to the Moquette broker. I'm getting the same error as mentioned above with the following configurations. I'm running zigbee2mqtt on the same raspberry pi that is functioning as both core and client device. Initially I had not created a client device but now I have and it hasn't changed the status of the connections, that keeps getting stuck. I deployed all the required components (IpDetector, Auth, Moquette and so on). How does the Moquette keystore update?

I followed the articles mentioned by @sushantAtAWS above but once I get the response from the cloud discovery API I don't know what to do with the provided CA

2024-02-13T14:15:34.437Z [ERROR] (nioEventLoopGroup-7-4) io.moquette.broker.NewNettyMQTTHandler: Unexpected exception while processing MQTT message. Closing Netty channel. CId=null. {} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.moquette.broker.metrics.BytesMetricsHandler.channelRead(BytesMetricsHandler.java:51) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:347) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:303) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:294) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:390) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1076) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1063) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1010) at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ... 25 more

This is the z2m config mqtt: base_topic: Z2M@SHD server: mqtts://localhost:8883 rejectUnauthorized: false # inserted this to perform some tests, I know in production it has to go certificate: /opt/data/revo/greengrass/v2/clientCert.crt # I've also tried with the certificates that the greengrass sdk installs on the core device but nothing. These are the client device certificates. Note that both are running on the same machine. key: /opt/data/revo/greengrass/v2/clientPrivKey.pem.key ca: /opt/data/revo/greengrass/v2/chain2.pem