- Newest
- Most votes
- Most comments
Hi Andrew. MQTT file delivery is primarily aimed at resource constrained devices that want to perform OTA entirely over MQTT. It can save the constrained device from having to include an HTTP client (just for OTA) in its firmware image, and also reduces the RAM footprint by having a single TLS connection. MQTT file delivery incurs additional messaging costs (as compared to HTTP file transfer). For resource-constrained devices, the firmware image is generally small, so these costs are small. If your device already supports an HTTP client, and/or your files are large, then HTTP file transfer may be a better choice. The AWS IoT Device SDK for Embedded C includes a small footprint HTTP client and demos of how to use it, including how to download a file from S3.
I'm not aware of a more general MQTT file streaming example that I can point you to. However, please be aware that the device-side implementation of the file streaming part of OTA is C and is free of any FreeRTOS dependencies: https://github.com/aws/ota-for-aws-iot-embedded-sdk. So you could potentially re-use the stream handling parts of that library to implement MQTT file streaming, outside of the OTA mechanism.
- https://github.com/aws/ota-for-aws-iot-embedded-sdk/blob/main/source/ota_mqtt.c
- https://github.com/aws/ota-for-aws-iot-embedded-sdk/blob/main/source/ota_cbor.c
The OTA operation you see in the console calls the CreateOTAUpdate API. OTA is in fact just an AWS IoT job with a special job document. That API call (depending on the parameters you pass) in turn calls StartSigningJob, CreateStream and CreateJob. You could instead call CreateStream (and maybe CreateJob too) directly to achieve file streaming outside of the OTA mechanism.
Again though, I would advise you to be sure you have a good reason for not using HTTP instead.
As a maintainer of the OTA for Embedded SDK library (part of the FreeRTOS team), I have a bit of further guidance for you.
- For the simplest approach possible - You could integrate AWS IoT ExpressLink into your solution. We made sure to specifically allow this kind of behavior through what we call the 'Host OTA'. The Host OTA (or HOTA for short) allows any file to be sent through the OTA mechanism to the ExpressLink device and be read off by the Host processor - which would be your MCU in this example. The Host processor can then use this file in any way it desires. The API to do so is simple, revolving around a couple of
AT+commands and diverges very little from the traditional flow of an OTA. Some documentation on this can be found here. - For the doeesn't-cost-you-anything-additional-but-software method, you could implement something very similar to ExpressLink OTA/HOTA mechanisms by using the OTA FileTypeId field to select how you handle files. For a traditional OTA, you can provide a certain file type value while for a generic file you can provide another. Doing this method, you'd need to modify the callback provided to the OTA_Init to select how it handles the files based off the file type ID.
And finally - the OTA library is configurable and allows the file transfer over either the MQTT and HTTP protocol when you configure the configENABLED_CONTROL_PROTOCOL macro which can be found here. Note this doesn't mean the entire OTA is handled over HTTP if configured so. The setup and negotiation of the OTA job state is always done over the MQTT protocol, however when configured the file transfer is done over HTTP. This is to reduce cost associated with an OTA file transfer.
If you have further problems or need guidance feel free to post on the FreeRTOS Forums or create an issue on the OTA library repository (if you select this library to use). Our team, and the FreeRTOS community, is very active and has many of experts who volunteer their knowledge.
Thank you @KodyAWS for thoughtful answer. I can see where I would examine the OtaJobEvent_t and be informed that the file is now in the OTA partition in flash. However whats not clear is how I would then access that data. Using esp-flash-read() to access that file seems odd, how would I determine for ex. the size of the file, which OTA partition, or what the raw flash address is?
In the AWS console, when creating an OTA job, there is a field for the device file path, where we always use "/". In an idea solution, I could pass a different path, and retrieve it with NVS.
The hardest part is determining where you want to write the block to. For OTA updates, it's often flash since it's non-volatile. For example, if you're using an ESP32 you can use the OTA data partition with it's corresponding APIs to store your file. Then you can simply read out from that.
The number of partitions is either determined by you or the vendor.
The size of the file should be tracked in the OtaFileContext_t struct as the
fileSizemember.@kodyAWS it feels strange to use this API, but I suppose why not? So I would wait for the OTA download to complete, and then call
esp_ota_get_next_update_partitionto obtain theuint32_t addressanduint32_t size, and thenesp_flash_readinto my buffers. The state machine for the OTA update is a little opaque, with the additional library layer from the freeRTOS example code for OTA, where would be the best place to cancel the OTA UPDATE, such that the system is left in a good state, and I am then still able to get my data? Thanks again for your help! I think we are close to having a good solution for future users!
I do have a good working implementation for transferring arbitrary files of any size over MQTT OTA update. There is a lot to it, but I am happy to share it, just let me know where/how, if it would be helpful.
Relevant content
- asked 14 days ago
- asked 2 years ago
AWS OFFICIALUpdated 3 years ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 7 months ago
Thank you so much for a great answer @Greg_B. MQTT has the added benefit of inbuilt security, so an http solution will incur some development cost to implement securely. It's a little embarrassing to admit that, although I know the OTA process is writing to the OTA partitions in flash on esp32, reading into the files you linked, I am unable to find the low level function which handle the buffer and do this writing. An arbitrary file handling solution would take these full MQTT file buffers and work on them before calling for the next chunk.
The HTTP demo I linked to uses the SigV4 library and the Credentials Provider to achieve a secure file transfer.
It's handled by the Platform Abstraction Layer (PAL). A now-deprecated example: https://github.com/aws/amazon-freertos/blob/main/vendors/espressif/boards/ports/ota_pal_for_aws/ota_pal.c
OMG i see now, the pOtaInterfaces struct has all the methods, and I see I can hook into them. Thank you so much! I deeply appreciate the thoughtful, patient and informed support. It's a great community here!
Thanks for the feedback. Please don't be shy about accepting and/or upvoting answers. :-)