- Newest
- Most votes
- Most comments
Yes, you can send data from Amazon Data Firehose to Amazon S3 Tables. Here's a step-by-step guide to set up this integration:
-
Create an S3 table bucket and integrate it with AWS analytics services through AWS Glue Data Catalog and AWS Lake Formation.
-
Create a namespace in the table bucket using the AWS CLI.
-
Create a table within that namespace.
-
Create a resource link in the default catalog to point to the namespace in the table bucket.
-
Create an IAM role for Firehose with the necessary permissions to access AWS Glue tables and write data to S3 tables.
-
Grant AWS Lake Formation permissions to the Firehose role to allow it to describe the resource link and read/write to the underlying table.
-
Set up a Firehose stream:
- Open Firehose in the AWS Console and choose "Create Firehose Stream"
- Select "Direct PUT" as the source
- Choose "Apache Iceberg Tables" as the destination
- Name your Firehose stream
-
Configure the destination settings:
- For the database name, choose the resource link you created earlier
- For the table name, select the table you created in step 3
- Specify an Amazon S3 general purpose bucket to store records that fail to be delivered to your S3 table bucket
-
Complete the Firehose stream setup by configuring any additional settings like data transformation, buffer hints, or error logging as needed.
Once set up, Firehose will automatically deliver the incoming streaming data to your specified S3 table. You can then use services like Amazon Athena to query and analyze the data in your S3 tables.
This integration allows you to stream data from various sources supported by Firehose (such as Kinesis Data Streams, Direct PUT API, or AWS services like VPC Flow Logs) directly into Amazon S3 Tables without requiring any code development or complex multi-step processes.
Sources
Build a data lake for streaming data with Amazon S3 Tables and Amazon Data Firehose | AWS Storage Blog
Amazon Data Firehose now delivers real-time streaming data into Amazon S3 Tables - AWS
Relevant content
asked 2 years ago

Step 4 is optional. Firehose can directly stream to S3 tables using DestinationDatabaseName = namespace after selecting correct catelog (same as s3 table bucket name) while creating destination stream e.g. [ { "DestinationTableName": "<table name inside table bucket>", "DestinationDatabaseName": "<namespace>", "S3ErrorOutputPrefix": "error" } ]