Durch die Nutzung von AWS re:Post stimmt du den AWS re:Post Nutzungsbedingungen
AWS re:Postre:Post

DB Log Processing through Kinesis Data streams and Time Series DB

0

Hi Team, I have an architecture based question, How Postgre SQL DB log processing can be captured through AWS lambda , aws Kinisis Data streams and finally Data should loads into Time Stream Database. Providing High level scenario: Draft Data flow : Aurora Postgre DB ----DB Logs Processing to ---->** Lambda** --->Ingestion to ----> **Kinesis Data Streams ** ---Process and Join context data and insert --- Insert to --------> Time Stream Database
I believe , we can process / loads the AWS IoT (sensors , device data) to Time Stream Database through Lambda , Kinesis Streams , Kinesis Data analytics and finally Time series Database and we can do analytics on time series data . But i am not sure How the postgre SQL db logs (write ahead logs) process through Lambda and ingest through Kinesis streams and finally load into Time Stream Database . and above flow also required to Joins some tables like Event driven tables with associated Account , Customer tables and then it will load into Time Series Database . would like to know if above flow would be accurate , as we are not processing any sensors / devices data ( where sensors data captures all measures and dimensions data from device and loads into Time Stream DB ) so Time Series database always a primary database . if anyone can through some lights , how postgre sql db logs can be integrated with Time Stream database through Kinesis Data streams , Lambda . Need your help Thanks

Themen
ServerlosKalkulationDatenbankAnalysen
Tags
AWS LambdaAurora PostgreSQLAmazon Kinesis DatenströmeAmazon Timestream
Sprache
English
AWS-User-8897895
gefragt vor 3 Jahren1761 Aufrufe
1 Antwort
0

Hi, many thanks for your question.

I'd suggest taking a look at AWS Database Migration Service that can stream changes from PostgreSQL (https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html) database to a number of destinations, including Amazon Kinesis Data Stream (https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html) . Then you can use Amazon Kinesis Data Analytics for Apache Flink to enrich your data and send it to Amazon TimeStream DB.

Hope it helps.

AWS
Alex_T
beantwortet vor 3 Jahren
  • AWS-User-8897895
    vor 3 Jahren

    Thanks for your suggestion and really appreciated . Both the links make sense to us . Here , Team want to try to load postgre db logs into Timestream DB (through write access logs) using Lambda and then Kinesis data streams , But they donot want to use Kinesis Data Analytics for apache flink , as per my understanding Kinesis Data analytics for apache flink is required here and even you have also suggested above . and we are not sure about How Postgre database - write access logs can configured in Postgre and any mandatory postgre db version is required to enabled write access logs. As per my understanding , Kinesis data streams , kinesis fire hose , kinesis data analytics for apache flink services are ideal aws services to read real time/IoT Data (sensor, device , web logs) data through some protocol and it loads data into Time Series DB and after that we can some analytics query on time series database based on timestamp partition. this is my understanding .

    this is my thought , Just reading postgre DB logs through Lambda and pushing it to Time Series Database won't be a ideal scenario for Real time Data reading / Event driven scenario. here we are creating many data hubs .

    one more thought , Time Series DB data always good to capture event data as it captures Dimension and Measures of a device / sensor in one table so analytical would be easy . i think , Time Series DB table won't support Joins with another time series db table.

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen

Relevanter Inhalt