By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

[Question] Firehose CDC postgres RDS to iceberg.

0

Two questions related to Firehose from Postgres RDS to Iceberg Tables.

  1. Can it support partitioned tables? for example if I have postgres partitioned table of users (users_1,users_2,users_3....) can I configure Firehose to write those to only one "users" iceberg table?

  2. Regarding upserts and deletions, if I want Firehose to be a CDC for DDL such as creating new tables created in postgres seamlessly but also to update / delete events for existing rows, can it be made without redefining the iceberg tables?

Thanks

Topics
AnalyticsDatabase
Tags
Amazon Data FirehosePostgreSQLAmazon RDS Database Preview Environment
Language
English
Amit
asked 14 days ago49 views
1 Answer
0

Hello Amit, Briefly:

  1. Partitioned Tables: Firehose doesn't natively consolidate partitioned Postgres tables into a single Iceberg table. You'd need a custom ETL pipeline.
  2. Upserts/Deletions: Native support for CDC in Firehose is limited; external tools like Debezium with Apache Iceberg are required for seamless DDL and CDC handling.
profile picture
EXPERT
Giovanni Lauria
answered 14 days ago
  • Amit
    14 days ago

    Thanks Giovanni!

    1. It seems like there is a method where you can customize the target iceberg table through JsonQuery/ Lambda as it seems https://docs.aws.amazon.com/firehose/latest/dev/apache-iceberg-format-input-record.html do you know if this layer is also supported with the CDC approach? if not it would be kinda deal breaker for me so just so I understand if there is ~realtime mechanism where I can have all my partitioned tables from postgres to Iceberg.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content