- Newest
- Most votes
- Most comments
I apologize for the confusion regarding the package dependency. The issue you're experiencing is likely due to the package not being available in the default Maven or Spark repositories.
To resolve this, you need to add the AWS Labs repository to your Spark configuration. The Amazon S3 Tables Catalog for Apache Iceberg client catalog is an open-source library hosted by AWS Labs, and it's not in the standard Maven Central repository.
Here's an updated version of the command that should resolve the dependency issue:
spark-shell
--packages software.amazon.s3:s3-tables-catalog-for-iceberg:0.1.0
--repositories https://aws.github.io/aws-sdk-java-v2/dev/repo
--conf spark.sql.catalog.s3tablesbucket=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.s3tablesbucket.catalog-impl=software.amazon.s3tables.iceberg.S3TablesCatalog
--conf spark.sql.catalog.s3tablesbucket.warehouse=your-table-bucket-ARN
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
The key addition here is the --repositories option, which adds the AWS Labs repository where the required package is hosted.
Also, please ensure that you have the necessary permissions to access the S3 bucket and the metadata table. The IAM role or user you're using should have the appropriate permissions to read from the S3 bucket and query the metadata table.
Remember that the S3 Metadata feature is currently in preview release and subject to change. If you continue to experience issues, you may want to check the latest AWS documentation or contact AWS support for the most up-to-date information on using this feature with Apache Spark.
Sources
Querying metadata tables with open-source query engines - Amazon Simple Storage Service
Configuring metadata tables - Amazon Simple Storage Service
answered 2 years ago
Hello,
After the adjustments of the updates from the AWS documentation, the queries that looks like below work on localhost: connect to AWS S3 and list data from configured table-bucket: my_namespace.my_table.
spark-shell
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,software.amazon.s3tables:s3-tables-catalog-for-iceberg-runtime:0.1.3,software.amazon.awssdk:bundle:2.29.43
--conf spark.sql.catalog.s3tablesbucket=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.s3tablesbucket.catalog-impl=software.amazon.s3tables.iceberg.S3TablesCatalog
--conf spark.sql.catalog.s3tablesbucket.warehouse=<arn:aws:s3tables:us-east-1:111122223333:bucket/amzn-s3-demo-bucket1>
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
scala> spark.sql(" SELECT * FROM s3tablesbucket.my_namespace.`my_table` ").show()
Have a good day,
Mihai ADAM
answered 2 years ago
Relevant content
asked 2 years ago
asked 2 years ago
asked 10 months ago
- AWS OFFICIALUpdated 9 months ago

Thank you for the answer, The package does not exists in that repo (https://aws.github.io/aws-sdk-java-v2/dev/repo) either, and I still have the error: UNRESOLVED DEPENDENCIES