Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Support Apache Iceberg in Redshift Managed Storage: dbt Mesh compatibility

0

We use Amazon Redshift as our main data warehouse and want to adopt dbt Mesh to enable cross-platform data sharing between Redshift and Snowflake.

Currently, dbt Mesh depends on Apache Iceberg as the open table format for sharing data without duplication. Redshift can register tables in AWS Glue, but Redshift Managed Storage (RMS) doesn’t support native Iceberg tables, so we can’t publish dbt models for downstream platforms to consume directly.

Please consider adding native Apache Iceberg support in Redshift Managed Storage, or allowing Redshift to read/write Iceberg tables through Glue. This would unlock true cross-platform data mesh capabilities while keeping data securely in AWS.

Topics
Analytics
Tags
AWS GlueAmazon Redshift
Language
English

asked a year ago141 views

1 Answer
0

My understanding is that supporting Iceberg on RMS would need fundamental change to Redshift storage architecture.

Supporting Iceberg would require either:

  1. Native integration - Modifying RMS to natively read/write Iceberg table formats

  2. Glue-mediated access - Allowing Redshift to read/write Iceberg tables through AWS Glue as an intermediary layer Both approaches would likely require significant architectural changes to how Redshift handles:

  • Table metadata management
  • File format compatibility
  • Query planning and execution
  • Storage layer interactions
AWS

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content