Consultation of possible IoT data collection and visualization solution for nested JSON data

What do you mean by "via a manifest file"?, you mean a manifest table?. You could have the job writing to a catalog table Quicksight could use

Based on the description you provided, the solution you outlined should work for processing and analyzing your IoT data. Here's a breakdown of the flow and considerations regarding permissions:

IoT Core: Your IoT device is connected to IoT Core, which allows you to receive and route messages from the device.

Message Routing: You have configured message routing in the rule settings to send JSON nested data to Amazon Kinesis Data Streams. This allows you to stream the data to be further processed.

AWS Glue Action: You have set up an AWS Glue action triggered by the Amazon Kinesis Data Stream. This enables you to perform data transformations and editing using the Glue service.

Flatten Node: By using the Flatten node in AWS Glue, you can flatten the nested JSON data structure into a tabular format suitable for analysis.

Saving to S3: You mentioned saving the flattened data to an S3 bucket in CSV format. This step allows you to store the processed data in a convenient and scalable storage solution.

QuickSight Integration: To visualize and analyze the data, you plan to send it to QuickSight via a manifest file. This enables you to create visualizations and dashboards based on the processed data.

Permissions Considerations:

Ensure that the IoT device has the necessary permissions to publish messages to IoT Core. Configure the appropriate IAM roles and policies for AWS Glue to access and process the data in Kinesis Data Streams, S3, and QuickSight.