Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

[Question] Firehose CDC postgres RDS to iceberg.

0

Two questions related to Firehose from Postgres RDS to Iceberg Tables.

  1. Can it support partitioned tables? for example if I have postgres partitioned table of users (users_1,users_2,users_3....) can I configure Firehose to write those to only one "users" iceberg table?

  2. Regarding upserts and deletions, if I want Firehose to be a CDC for DDL such as creating new tables created in postgres seamlessly but also to update / delete events for existing rows, can it be made without redefining the iceberg tables?

Thanks

Topics
AnalyticsDatabase
Tags
Amazon RDS Database Preview EnvironmentPostgreSQLAmazon Data Firehose
Language
English
asked a year ago523 views
2 Answers
0

Hello Amit, Briefly:

  1. Partitioned Tables: Firehose doesn't natively consolidate partitioned Postgres tables into a single Iceberg table. You'd need a custom ETL pipeline.
  2. Upserts/Deletions: Native support for CDC in Firehose is limited; external tools like Debezium with Apache Iceberg are required for seamless DDL and CDC handling.
EXPERT
answered a year ago
  • Amit
    a year ago

    Thanks Giovanni!

    1. It seems like there is a method where you can customize the target iceberg table through JsonQuery/ Lambda as it seems https://docs.aws.amazon.com/firehose/latest/dev/apache-iceberg-format-input-record.html do you know if this layer is also supported with the CDC approach? if not it would be kinda deal breaker for me so just so I understand if there is ~realtime mechanism where I can have all my partitioned tables from postgres to Iceberg.
0

I'd also love to know if support for Postgres partitioning is on the roadmap as well. It would be great if Firehose CDC can be used as the single ingestion tool from RDS to Iceberg, without having to build out other ingestion solutions to handle the partitioned table use case.

answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content