DB Log Processing through Kinesis Data streams and Time Series DB
Hi Team, I have an architecture based question, How Postgre SQL DB log processing can be captured through AWS lambda , aws Kinisis Data streams and finally Data should loads into Time Stream Database. Providing High level scenario:
Draft Data flow :
Aurora Postgre DB ----DB Logs Processing to ----> Lambda --->Ingestion to ----> Kinesis Data Streams ---Process and Join context data and insert --- Insert to --------> Time Stream Database
I believe , we can process / loads the AWS IoT (sensors , device data) to Time Stream Database through Lambda , Kinesis Streams , Kinesis Data analytics and finally Time series Database and we can do analytics on time series data . But i am not sure How the postgre SQL db logs (write ahead logs) process through Lambda and ingest through Kinesis streams and finally load into Time Stream Database . and above flow also required to Joins some tables like Event driven tables with associated Account , Customer tables and then it will load into Time Series Database . would like to know if above flow would be accurate , as we are not processing any sensors / devices data ( where sensors data captures all measures and dimensions data from device and loads into Time Stream DB ) so Time Series database always a primary database . if anyone can through some lights , how postgre sql db logs can be integrated with Time Stream database through Kinesis Data streams , Lambda . Need your help Thanks
Hi, many thanks for your question.
I'd suggest taking a look at AWS Database Migration Service that can stream changes from PostgreSQL (https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html) database to a number of destinations, including Amazon Kinesis Data Stream (https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html) . Then you can use Amazon Kinesis Data Analytics for Apache Flink to enrich your data and send it to Amazon TimeStream DB.
Hope it helps.
Thanks for your suggestion and really appreciated . Both the links make sense to us . Here , Team want to try to load postgre db logs into Timestream DB (through write access logs) using Lambda and then Kinesis data streams , But they donot want to use Kinesis Data Analytics for apache flink , as per my understanding Kinesis Data analytics for apache flink is required here and even you have also suggested above . and we are not sure about How Postgre database - write access logs can configured in Postgre and any mandatory postgre db version is required to enabled write access logs. As per my understanding , Kinesis data streams , kinesis fire hose , kinesis data analytics for apache flink services are ideal aws services to read real time/IoT Data (sensor, device , web logs) data through some protocol and it loads data into Time Series DB and after that we can some analytics query on time series database based on timestamp partition. this is my understanding .
this is my thought , Just reading postgre DB logs through Lambda and pushing it to Time Series Database won't be a ideal scenario for Real time Data reading / Event driven scenario. here we are creating many data hubs .
one more thought , Time Series DB data always good to capture event data as it captures Dimension and Measures of a device / sensor in one table so analytical would be easy . i think , Time Series DB table won't support Joins with another time series db table.
Error when trying to add db instance to VS server explorerasked 3 years ago
How can I time restoring from a DB snapshotasked a year ago
Question on TimeStream DB
Source Data for TimeStream Database
DB Log Processing through Kinesis Data streams and Time Series DBasked 5 months ago
Dynamo DB Kinesis Steams Best PracticeAccepted Answerasked a year ago
Using AWS Lambda to run Python script, how can I save data?Accepted Answerasked 3 years ago
What's the best way to send real-time data to Amazon Redshift?Accepted Answerasked a year ago
How to stream CloudFront real time logs to cloudwatchasked 3 months ago
Invoke specified CloudWatch Rule on MediaConvert Eventasked 3 years ago