By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Question on TimeStream DB

0

Hi team, would like to know , Is there a possibility / use case, where Postgre SQL Write Ahead Logs (WAL) (which is mainly used for database backup recovery functionality) can be used / considered as a Event data or Source data and this WAL logs further can be processed , transform through Kinesis data streams and Load into destination db i.e. AWS Time stream database . just try to show in diagram

---- source --- ----Data processing----- Target DB Postgre SQL (WAL Logs) -----> Kinesis Data Streams (with Lambda) -------> Time Stream DB

Is this architecture is feasible ? as per my understanding , Time Stream DB stores the device / sensor /web logs data and time stream can be considered as Primary DB always . please add your thoughts Regards, Mangesh

Topics
Database
Tags
Amazon Timestream
Language
English
AWS-User-8897895
asked 2 years ago382 views
1 Answer
0

PostgreSQL WAL logs are in binary format mainly used for backup and streaming replication and cannot be parsed by any other application or services. To capture the steam of changes to the PostgreSQL database , you can implement logical replication as mentioned in https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication as the source for kinesis data stream.

AWS
AWS-User-Raj
answered 2 years ago
  • AWS-User-8897895
    2 years ago

    Hi Raj, thanks for clarifying , so what i understood here , i) these WAL logs has only useful in backup and recovery , streaming replication etc. but same WAL logs (which is binary in nature) cannot parse by any application like python , i mean it can not be converted into a typical database record . please correct me. 2) here we wants to use this WAL logs as a source and want to transform such logs into a database record. after that , delta capture needs to do from WAL logs which is different for each table. but i believe WAL logs capture all the logs in database irrespective of changes in all the database tables ( for e.g. 10 tables are getting insert and updated) 3) As we can implement logical replication , so what is purpose of replication ? i believe , WAL logs are not meant for processing Delta records or any ETL transformation work , please correct me . Thanks in Advance

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content