Move data from kafka topics to S3 table

Yes, it is possible to move data from Kafka topics to Amazon S3 tables. However, the configuration you provided is for moving data to a regular S3 bucket using the Apache Iceberg sink connector, not specifically to S3 tables.

To move data from Kafka topics to S3 tables, you would need to use a different approach. One option is to use the Amazon S3 Tables Catalog for Apache Iceberg in combination with Apache Spark or AWS Glue.

Here's a high-level overview of how you could achieve this:

1. Set up a Spark session configured to use the Amazon S3 Tables Catalog for Apache Iceberg. This involves specifying the correct catalog implementation and warehouse location.

2. Create a Spark streaming job that reads from your Kafka topic and writes to the S3 table using the Iceberg format.

3. Configure the Spark job to connect to your Kafka cluster and S3 table bucket.

4. Run the Spark job, which will continuously consume data from the Kafka topic and write it to the S3 table.

Alternatively, you could use AWS Glue to create an ETL job that reads from your Kafka topic and writes to the S3 table. This would involve creating a PySpark script that configures the Spark session for Apache Iceberg and connects to your S3 table bucket.

It's important to note that the direct use of the Apache Iceberg sink connector you mentioned may not be compatible with S3 tables as they are a specific AWS implementation. You would need to adapt your approach to use AWS-specific tools and services that are designed to work with S3 tables.

Remember to ensure that you have the necessary permissions set up for your IAM role to access both the Kafka cluster and the S3 table bucket.

Sources
Accessing Amazon S3 tables with the Amazon S3 Tables Catalog for Apache Iceberg - Amazon Simple Storage Service
Running ETL jobs on Amazon S3 tables with AWS Glue - Amazon Simple Storage Service

Hello, Thank you for reaching out to us.

Unfortunately, it is not possible to use the S3 sink connector for the S3Table bucket. As of now it can only be used with normal general purpose S3 buckets. However, there is an option to integrate the S3 Table with Kinesis firehose. [+] Streaming data to tables with Amazon Data Firehose - https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-firehose.html

For this, you have to integrate Kafka to kinesis streams to kinesis firehose and then to S3Tables. [+] https://docs.aws.amazon.com/msk/latest/developerguide/integrations-kinesis-data-firehose.html

Feel free to reach out for further queries or concerns. We will be happy to assist you.

Thank you again, please have a nice day!

this is working ->

{
  "name": "aws-S3table",
  "config": {
    "connector.class": "org.apache.iceberg.connect.IcebergSinkConnector",
    "tasks.max": "1",
    "topics": "aaaaaa",
    "iceberg.tables.auto-create-enabled": "true",
    "iceberg.tables": "default.table_A",
    "iceberg.catalog.catalog-impl": "software.amazon.s3tables.iceberg.S3TablesCatalog",
    "iceberg.catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
    "iceberg.catalog.warehouse": "arn:aws:s3tables:us-east-1:ABCABCABC:bucket/test-S3bucket",
    "iceberg.catalog.s3.path-style-access": "true",
    "iceberg.control.commit.interval-ms": "18000"
  }
}```