Process ftp files with sensor readings

Question

For an IoT project I need to ingest data (files with sensor data) coming from an (on-prem) FTP server. I have a Lambda function with a Cloudwatch cronjob that on an interval basis get the content of these files. I would like (a cold path) storing the raw data to DynamoDB and/or an S3 bucket and (a warm path) by using AWS IoT Analytics.  
   
My question:  
-        Is it best to process the contents of the files and interface directly with DynamoDB and/or S3 and IoT Analytics (BatchPutMessage) or  
-        Process the incoming files sensor readings publish them first to AWS IoT, and from there use rules for the cold and warm path?  
   
Thanks.  
Guy

Answer

Of the two options you listed, I recommend the first one. You should just interface directly with S3 (or DynamoDB) and IoT Analytics if your data is not already being sent to the AWS IoT Core Message Broker.  
  
You could alternatively use IoT Analytics for both the cold path and the warm path. You can use an IoT Analytics Datastore for cold storage in your S3 Bucket. \[0] \[1] You could then enrich the data in your IoT Analytics Dataset's query. This option is preferable if a SQL query is sufficient for your data enrichment needs.  
  
Note that in both options I listed, you would ingest data into IoT Analytics through the BatchPutMessage API. What sets the second option apart is that you would not need to upload the data to S3 yourself; IoT Analytics would handle uploading the data to your bucket.  
  
\[0] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-iotanalytics-datastore-customermanageds3.html  
\[1] https://docs.aws.amazon.com/iotanalytics/latest/userguide/getting-started.html#aws-iot-analytics-step-create-datastore  
  
Edited by: DavidWataws on Feb 27, 2020 7:52 PM  
  
Edited by: DavidWataws on Feb 27, 2020 7:53 PM

Answer

Many thanks.

Process ftp files with sensor readings

Relevant content