How to use Apache Iceberg REST API on redshift

Hello,

Generally, when using Iceberg tables on Redshift, you will go through the following process.

[+] Using Apache Iceberg tables with Amazon Redshift https://docs.aws.amazon.com/redshift/latest/dg/querying-iceberg.html

To get started using Iceberg tables with Amazon Redshift:

1. Create an Apache Iceberg table on an AWS Glue Data Catalog database using a compatible service such as Amazon Athena or Amazon EMR. To create an Iceberg table using Athena, see Using Apache Iceberg tables in the Amazon Athena User Guide.

2. Create an Amazon Redshift cluster or Redshift Serverless workgroup with an associated IAM role that allows access to your data lake. For information on how to create clusters or workgroups, see Get started with Amazon Redshift provisioned data warehouses and Get started with Redshift Serverless data warehouses in the Amazon Redshift Getting Started Guide.

3. Connect to your cluster or workgroup using query editor v2 or a third-party SQL client. For information about how to connect using query editor v2, see Connecting to an Amazon Redshift data warehouse using SQL client tools in the Amazon Redshift Management Guide.

4. Create an external schema in your Amazon Redshift database for a specific Data Catalog database that includes your Iceberg tables. For information about creating an external schema, see External schemas in Amazon Redshift Spectrum.

5. Run SQL queries to access the Iceberg tables in the external schema you created.

In addition, we will deliver the following additional documents regarding how to use Iceberg as REST API method.

[+] Read and write S3 Iceberg table using AWS Glue Iceberg Rest Catalog from Open Source Apache Spark

https://aws.amazon.com/ko/blogs/big-data/read-and-write-s3-iceberg-table-using-aws-glue-iceberg-rest-catalog-from-open-source-apache-spark/

If an error occurs even after connecting through the above method, it is recommended that you create an AWS Support Case.

https://console.aws.amazon.com/support/home#/case/create

To use the Apache Iceberg REST API with Amazon Redshift, you need to follow these steps:

1. Ensure your Amazon Redshift cluster or serverless workgroup is registered with the AWS Glue Data Catalog.

2. Set up the necessary permissions using AWS Lake Formation for the catalog you want to query.

3. Make sure you have an IAM role with the 'AmazonRedshiftServiceLinkedRolePolicy' managed policy attached.

4. Use the AWS Glue Iceberg REST extension endpoint to access the catalog. The endpoint URL should be in the format: https://glue.[region].amazonaws.com/extensions

5. When querying tables, use the three-part syntax: database@namespace.schema.table

6. To execute queries using the REST API, you'll need to provide the catalog's Amazon Resource Name (ARN) when making API calls.

It's important to note that while you can query Iceberg tables in Athena directly, using the REST API with Redshift requires additional configuration. The REST API is primarily used for programmatic access and integration with other services that support the Apache Iceberg REST Open API specification.

If you're having trouble using the REST API, double-check your endpoint configuration, ensure you have the correct permissions set up in Lake Formation, and verify that your IAM role has the necessary policies to access the Glue Data Catalog and Redshift resources.

Also, keep in mind that Redshift supports read-only access to Iceberg tables, so you can perform SELECT queries, but not data modification operations through this interface.

If you continue to experience issues, you may want to review your specific API calls and ensure they align with the Apache Iceberg REST API specification and AWS Glue's implementation of it.

Sources
Querying catalogs registered in the AWS Glue Data Catalog - Amazon Redshift
Accessing the Data Catalog - AWS Glue
Apache Iceberg compatibility for Amazon Redshift - Amazon Redshift

Use the Apache Spark configuration mentioned in the link: Accessing tables using the Amazon S3 Tables Iceberg REST endpoint. I was able to use the REST API to create Iceberg tables in S3 table bucket.