Accessing S3 tables from AWS Glue ( Spark + Python )

When working with AWS Glue and PySpark to access S3 tables, you don't need to explicitly include the package "software.amazon.s3tables:s3-tables-catalog-for-iceberg-runtime:0.1.3" as you would in a standalone Spark environment. AWS Glue provides built-in support for working with Iceberg tables and S3.

To access S3 tables from an AWS Glue PySpark job, you can use the following approach:

1. Ensure your Glue job has the necessary IAM permissions to access the relevant S3 buckets.

2. In your Glue job, use the GlueContext to create a DataFrame from the AWS Glue Data Catalog. Here's an example:

from awsglue.context import GlueContext
from pyspark.context import SparkContext

sc = SparkContext()
glueContext = GlueContext(sc)

# Read an Iceberg table from Glue Data Catalog
df = glueContext.create_data_frame.from_catalog(
    database="your_database_name",
    table_name="your_table_name"
)

3. If you need to configure Iceberg-specific options, you can do so by setting job parameters when creating your Glue job. For example:

   • Set "--datalake-formats" to "iceberg"
   • Add necessary Iceberg configurations using "--conf" parameters

4. If you need to write data to an Iceberg table, you can use the Spark DataFrame API or Spark SQL, and AWS Glue will handle the underlying Iceberg operations.

Remember that AWS Glue 3.0 and later versions support the Iceberg framework, so make sure you're using a compatible Glue version for your job.

By using these approaches, you can work with S3 tables in your AWS Glue PySpark job without needing to explicitly include the S3 tables catalog package as you would in a standalone Spark environment.
Sources
Using the Iceberg framework in AWS Glue - AWS Glue
Amazon S3 connections - AWS Glue