mqttAgent RX fails on large messages

Question

I am using the coreMQTT-Agent lib on an ESP32 to receive large MQTT messages via AWS LAMBDA.
Smaller messages work perfectly, but as the messages grow in size, they fail with
```
D (47173) coreMQTT: Encoded size for length 9657 is 2 bytes.
E (47173) coreMQTT: MQTT operation failed with status Invalid MQTT Status code
I (47183) sub_pub_demo: coreMQTT-Agent disconnected. Preventing coreMQTT-Agent commands from being enqueued.
```
And then the MQTT connection resets.
I have tried increasing stack and buffer sizes via menuconfig for all the relevant tasks, but I am unable to locate a config setting that fixes the error.
I have tried to use these error messages to trace the issue back to its source but am unable to see the issue, as 9657 is well within the limits of MQTT messages, and the buffers and stacks that I can find, and no helpful error messages are presented.

Is anyone able to point me in the direction of a fix?

Thank you!

Answer

Sorry for not posting the solution sooner, it was just that `OTA_MAX_BLOCK_BITMAP_SIZE` was overflowing, I increased it to 512U and the issue disappeared.

It is defined in `components/esp-aws-iot/libraries/ota-for-aws-iot-embedded-sdk/ota-for-aws-iot-embedded-sdk/source/include/ota_private.h` on line#66

Answer

For my problem, I found that partially received packet is treated as an error if the next loop didn't received any new packets.

I changed while( pMqttAgentContext->packetReceivedInLoop) to while( pMqttAgentContext->packetReceivedInLoop || operationStatus == MQTTNeedMoreBytes)

in core_mqtt_agent.c and that fixed my problem. https://github.com/FreeRTOS/coreMQTT-Agent/blob/main/source/core_mqtt_agent.c#L613

But this might not be a permanent solution since we are not supposed to wait forever if the status remains as MQTTNeedMoreBytes. Some sort of timeout would make this change more safe.

We can also consider adding a delay before the calling the MQTT_ProcessLoop() instead of calling immediately.

mqttAgent RX fails on large messages

Relevant content