IoT Core to ? to Timestream

Question

I'm ingesting measurements to IoT Core and am investigating the right architecture to move the data from there to Timestream. Each measurement comes into IoT Core as a ~1-2k JSON document and has a UUID as deviceId.

- I want to batch up a number of measurements for each insert and rewrite column names to control cost

- I want to query an RDS instance to retrieve additional dimensions for the given deviceId, e.g. accountId, organizationId and such and enhance the records with that

- I want to do inserts using multi-measure records because each JSON document contains ~20 values for different kinds of measurements

The requirements above rule out using an IoT Core Rule for direct inserts. What's the best bet? IoT Core to SQS and then have a long running job in a Fargate container to tie things together? Or is there a good pattern to use Lambda across multiple ingested records? Something entirely different perhaps?

Any insights and gotchas much appreciated

Answer

Hi - Thanks for reaching and great questions. Have you got a chance to review this blog which provides [patterns for IoT data ingestion and visualization- How to decide what works best for your use case](https://aws.amazon.com/blogs/iot/7-patterns-for-iot-data-ingestion-and-visualization-how-to-decide-what-works-best-for-your-use-case/)

You can also move data to S3 (IoT Analytics) and as a result, you can now automatically create an AWS Glue catalog table containing the schema of your AWS IoT Analytics dataset content results and run queries with Amazon Athena. Because the dataset content results are saved in your S3 bucket, you can apply your own S3 permissions and manage them according to your governance policies.

There is also a good reference architecture listed here https://iotatlas.net/en/implementations/aws/telemetry_archiving/iot_analytics1/

Hope this helps in your decision making.

IoT Core to ? to Timestream

Relevant content