How to aggregate records in S3 using IOT Rules and Action?

Question

I'd like to aggregate `prediction` MQTT messages generated by the Greengrass [ML Feedback Connector](https://docs.aws.amazon.com/greengrass/latest/developerguide/ml-feedback-connector.html) over a certain period into a JSON-Lines file in S3: Which may then be used as a SageMaker Ground Truth input manifest, or for other analysis.

I'm not clear from the doc on the [IoT Rules S3 action](https://docs.aws.amazon.com/iot/latest/developerguide/iot-rule-actions.html#s3-rule) whether:

- It would be safe to have multiple (perhaps close/concurrent) messages pushing to the same S3 bucket & key via Rules
- Messages would be concatenated into one file (with a separator?), or overwrite
- There's enough flexibility in the [IoT SQL functions](https://docs.aws.amazon.com/iot/latest/developerguide/iot-sql-functions.html) to control the level of aggregation nicely (Seems like day/week/month would be tough, but probably could just do something with integer-dividing the `timestamp()` number

It seems like this is a use case that [IoT Analytics has tools to handle](https://aws.amazon.com/blogs/iot/integrating-iot-data-with-your-data-lake-with-new-aws-iot-analytics-features/), but I'm just trying to concatenate the messages together, not perform analyses/proper aggregations.

Accepted Answer

One option would be use Firehose to batch the data:

IoT Core --> Rule --> Firehose --> S3

In this flow, Firehose writes the messages in batches to objects stored in S3.

How to aggregate records in S3 using IOT Rules and Action?

Relevant content