Skip to content

Can´t use a MQTT message payload greater than 63KB with AWS IoT Device SDK for Python

0

Hello everybody,

I've followed the official docs tutorial for connecting a client device to a core device equipped with AWS IoT Greengrass (available here ), so right now I have my local client device in a Raspberry Pi 3 and my core device is in my laptop. I'm using AWS IoT Device SDK for Python and I've modified the basic_discovery.py to be able to send some messages during a limited period of time from my client device. If I use a message payload that exceeds 63KB, it just gets stuck at the first .result() after invoking .publish() with QoS.AT_LEAST_ONCE (value 1). The QoS doesn't seem to be the cause of the problem.

I know that the limit for IoT Core is 128KB, so I don't know what could be happening, because what it is allowing me is half that theoretical limit. I also went inside /awscrt and /awsiot dependencies inside the Python venv and I haven't seen any kind of limit expressed there. I have also searched info in the official docs and I have not found anything about it, just the quota of 128KB. If it helps, the AWS region in which my core device is deployed is eu-west-2 (London).

By the way, my very basic program works perfectly when payload is 63KB or less, and the original basic_discovery.py also works fine even through quite longer periods of time (after a little correction on the obsolete puback property).

I don't want to necessarily reach the limit with my payload, but I'm very interested in knowing the reason behind this behavior since I'm trying to finish my thesis. I don't have much experience with AWS, barely just what I have obtained until now while building this system, so I'm by no means an AWS expert.

Any kind of help is welcome :)

3 Answers
1

When working with MQTT messages in AWS IoT Greengrass, there are several limitations to be aware of regarding message payload size.

While AWS IoT Core has a message size limit of 128 KB, AWS IoT MQTT-based file delivery specifically limits the size of a block to 128 KB. If you make a request for a block that exceeds this size, the request will fail.

In the Greengrass environment, there are additional considerations. When publishing MQTT messages from Greengrass devices to AWS IoT Core, there are quotas of 100 transactions per second and 512 KB of data per second. Messages that exceed these quotas are queued in a spooler in memory.

The behavior you're experiencing where messages larger than 63 KB get stuck might be related to how the Greengrass MQTT bridge handles message payloads. The bridge component has its own limitations that may differ from the direct AWS IoT Core limits.

For applications that need to send larger messages or messages at a higher rate, AWS recommends using the Stream manager component instead of direct MQTT publishing. Stream manager is specifically designed to transfer high-volume data to the AWS Cloud, such as through Kinesis Data Streams.

If you absolutely need to send larger payloads, you might need to implement a chunking mechanism to break your data into smaller pieces, or consider using the Stream manager component as an alternative approach.
Sources
MQTT message payload - AWS IoT Core
Publish/subscribe AWS IoT Core MQTT messages - AWS IoT Greengrass
Using AWS IoT MQTT-based file delivery in devices - AWS IoT Core

answered a year ago
EXPERT
reviewed a year ago
0

Hi, the mqtt.maxMessageSizeInBytes configuration can be used to set the max limit for MQTT messages, but the default value is 128 KB so that should not be the issue here unless your core has it configured to a lower value. I would recommend opening an issue on the https://github.com/aws/aws-iot-device-sdk-python-v2/ repo since it could be a bug there.

If you could grab the logs for your component, that would be helpful (they are likely in /greengrass/v2/logs).

AWS
answered a year ago
0

Sure. This is the content extracted with sudo tail -f /greengrass/v2/logs/greengrass.log:

2025-06-25T17:38:44.784Z [INFO] (pool-3-thread-17) com.aws.greengrass.mqtt.bridge.clients.MQTTClient: Connected to broker. {clientId=mqtt-bridge-2puf2f3ahsk, brokerUri=ssl://localhost:8883}
2025-06-25T17:38:44.789Z [INFO] (nioEventLoopGroup-3-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <mqtt-bridge-2puf2f3ahsk> to topics [MqttTopicSubscription[topicFilter=clients/+/hello/world, option=SubscriptionOption[qos=AT_LEAST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
2025-06-25T17:45:12.799Z [INFO] (nioEventLoopGroup-3-2) io.moquette.broker.metrics.MQTTMessageLogger: C->B CONNECT <null>. {}
2025-06-25T17:45:13.067Z [INFO] (nioEventLoopGroup-3-2) com.aws.greengrass.mqtt.moquette.ClientDeviceAuthorizer: Successfully authenticated client device. {clientId=Local_IoT_device, sessionId=93ff2b84-06dc-4588-aeed-bd1ca59da079}
2025-06-25T17:45:13.077Z [INFO] (nioEventLoopGroup-3-2) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <Local_IoT_device> to topics [MqttTopicSubscription[topicFilter=clients/Local_IoT_device/hello/world, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
2025-06-25T17:45:23.639Z [INFO] (nioEventLoopGroup-3-2) io.moquette.broker.metrics.MQTTMessageLogger: C->B DISCONNECT <Local_IoT_device>. {}
2025-06-25T17:45:23.645Z [INFO] (nioEventLoopGroup-3-2) io.moquette.broker.metrics.MQTTMessageLogger: Channel Inactive. {}
2025-06-25T17:50:37.684Z [INFO] (nioEventLoopGroup-3-3) io.moquette.broker.metrics.MQTTMessageLogger: C->B CONNECT <null>. {}
2025-06-25T17:50:37.986Z [INFO] (nioEventLoopGroup-3-3) com.aws.greengrass.mqtt.moquette.ClientDeviceAuthorizer: Successfully authenticated client device. {clientId=Local_IoT_device, sessionId=7838c109-7325-46ab-8176-67cb1e750ef9}
2025-06-25T17:50:37.997Z [INFO] (nioEventLoopGroup-3-3) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <Local_IoT_device> to topics [MqttTopicSubscription[topicFilter=clients/Local_IoT_device/hello/world, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
2025-06-25T17:51:19.660Z [INFO] (nioEventLoopGroup-3-3) io.moquette.broker.metrics.MQTTMessageLogger: Channel Inactive. {}

Remember that this is the log right after a failed execution with a 64KB payload which, of course, got stucked until I canceled the execution with Ctrl + C.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.