Firehose Returns Iceberg.NoSuchIcebergTable When Writing to Shared Iceberg Table via Table Resource Link

Question

I'm using **Amazon Kinesis Data Firehose** to deliver streaming data into an **Apache Iceberg table** registered in the **AWS Glue Data Catalog**, which is part of a non-default catalog (`s3tablescatalog/project-data-bucket`). I created a **table-level resource link** in my account using AWS Lake Formation.

However, when sending data from Kinesis Data Generator through Firehose, I receive the following error:

```json
"Iceberg.NoSuchIcebergTable: Input Glue table is not an iceberg table."
```

---

### 🔧 Setup Summary

#### 🔹 Original Iceberg Table 
- **Catalog**: `123456789012:s3tablescatalog/project-data-bucket`
- **Database**: `marketing_data`
- **Table**: `ad_campaign_metrics`
- **Format**: Iceberg (Parquet)
- **Partitioned By**: `tenant_id`, `day(event_timestamp)`

#### 🔹 Resource Link
- **Resource Link Type**: **Table-level only** (no resource-linked database)
- **Linked Table Name**: Appears in my Glue catalog as `marketing_linked_data.ad_campaign_metrics_linked`
- **Created in**: Lake Formation → *Table Resource Link* section

#### 🔹 Firehose Configuration
- **Delivery Stream Name**: `FIREHOSE_TO_ICEBERG`
- **Target Glue Table**: `marketing_linked_data.ad_campaign_metrics_linked` *(resource link)*

#### 🔹 IAM & Lake Formation Permissions
- IAM role used by Firehose has:
  - **Lake Formation permissions**: `ALL` on both the **resource link** and the **original table**
  - Permissions include **grant option**
- Table is accessible through Athena via the resource link

#### 🔹 Sample Input Payload
```json
{
  "event_id": "ev001",
  "tenant_id": "t001",
  "ad_id": "a001",
  "clicks": 150,
  "cost": 75.25,
  "event_timestamp": "2025-01-01T00:00:00Z"
}
```

---

### ❗ Observed Behavior

Despite the correct setup and permissions, Firehose fails to deliver the records and returns:

```
"Iceberg.NoSuchIcebergTable: Input Glue table is not an iceberg table. Table: marketing_linked_data.ad_campaign_metrics_linked"
```

---

### 💬 Questions

1. **Does Amazon Kinesis Firehose support writing to Apache Iceberg tables via a table-level resource link (without a resource-linked database)?**
2. Is it mandatory to have a **resource-linked database** for Firehose to resolve the underlying Iceberg table correctly?
3. Are there additional configurations or prerequisites required for Firehose to recognize the resource-linked table as an Iceberg table?

Any guidance or clarification on Firehose's compatibility with table-level resource links in Lake Formation would be greatly appreciated.

re:Post Agent · Answer

Based on the information provided and the current capabilities of Amazon Kinesis Data Firehose, it appears that there are some limitations when working with Apache Iceberg tables via table-level resource links. Let's address your questions:

1. Amazon Kinesis Data Firehose does not currently support writing to Apache Iceberg tables using only a table-level resource link without a resource-linked database. This limitation is likely the root cause of the error you're experiencing.

2. Yes, it seems that having a resource-linked database is mandatory for Firehose to correctly resolve the underlying Iceberg table. Firehose expects to work with a complete database structure, which includes both the database and table information.

3. To get Firehose to recognize the resource-linked table as an Iceberg table, you'll need to make some changes to your setup:

- Instead of creating just a table-level resource link, you should create a resource link for the entire database that contains your Iceberg table.
   - The resource link should point to the namespace (database) in your table bucket.
   - When configuring Firehose, you would then use the name of this resource-linked database as the database name, and the original table name as the table name.

To resolve your issue, follow these steps:

1. Create a database-level resource link in your AWS Glue Data Catalog that points to the `marketing_data` database in the `s3tablescatalog/project-data-bucket` catalog.

2. Grant the necessary Lake Formation permissions on both the resource link and the target (linked) namespace or table to your Firehose service role.

3. In your Firehose configuration, use the name of the resource-linked database as the database name, and `ad_campaign_metrics` as the table name.

4. Ensure that the column names and data types in your input JSON match exactly with those defined in your Iceberg table schema. Firehose is particular about this matching, including capitalization.

5. If you have nested JSON structures, be aware that Firehose only processes the first level of nodes in a multi-level nested JSON. Ensure your `event_timestamp` field is at the top level of your JSON structure.

By making these adjustments, you should be able to successfully stream data from Firehose to your shared Iceberg table. If you continue to experience issues, double-check your IAM permissions and ensure that the Firehose role has all necessary permissions to access the resource link, the underlying table, and write to the S3 bucket where the Iceberg data is stored.
      
**Sources**  
[Streaming data to tables with Amazon Data Firehose - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-firehose.html)  
[Consideration and limitations - Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/apache-iceberg-considerations.html)  
[How to send data from Firehose to S3 tables ? | AWS re:Post](https://repost.aws/questions/QUB7sYQp79TaShD9m5YKaU8w/how-to-send-data-from-firehose-to-s3-tables)

Firehose Returns Iceberg.NoSuchIcebergTable When Writing to Shared Iceberg Table via Table Resource Link

🔧 Setup Summary

🔹 Original Iceberg Table

🔹 Resource Link

🔹 Firehose Configuration

🔹 IAM & Lake Formation Permissions

🔹 Sample Input Payload

❗ Observed Behavior

💬 Questions

Relevant content