Skip to content

S3 metadata configuration, UNRESOLVED DEPENDENCIES error

0

Hello,

I've used Management Console, S3 Metadata - Preview -> Create a metadata configuration option to create a metadata table in a table bucket. The table was successfully created.

I tried to initialize Spark, on localhost, to use Amazon S3 Tables Catalog for Apache Iceberg, using the following command , like specified here

spark-shell 
--packages software.amazon.s3:s3-tables-catalog-for-iceberg:0.1.0
--conf spark.sql.catalog.s3tablesbucket=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.s3tablesbucket.catalog-impl=software.amazon.s3tables.iceberg.S3TablesCatalog
--conf spark.sql.catalog.s3tablesbucket.warehouse=your-table-bucket-ARN
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

but I received an error about UNRESOLVED DEPENDENCIES: software.amazon.s3:s3-tables-catalog-for-iceberg:0.1.0, because this package does not exists in Maven or Spark repos.
Could you please specify an update for the command so that the package to be found ?

Thank you,
Mihai ADAM

asked 2 years ago518 views

2 Answers
0

I apologize for the confusion regarding the package dependency. The issue you're experiencing is likely due to the package not being available in the default Maven or Spark repositories.

To resolve this, you need to add the AWS Labs repository to your Spark configuration. The Amazon S3 Tables Catalog for Apache Iceberg client catalog is an open-source library hosted by AWS Labs, and it's not in the standard Maven Central repository.

Here's an updated version of the command that should resolve the dependency issue:

spark-shell 
--packages software.amazon.s3:s3-tables-catalog-for-iceberg:0.1.0
--repositories https://aws.github.io/aws-sdk-java-v2/dev/repo
--conf spark.sql.catalog.s3tablesbucket=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.s3tablesbucket.catalog-impl=software.amazon.s3tables.iceberg.S3TablesCatalog
--conf spark.sql.catalog.s3tablesbucket.warehouse=your-table-bucket-ARN
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

The key addition here is the --repositories option, which adds the AWS Labs repository where the required package is hosted.

Also, please ensure that you have the necessary permissions to access the S3 bucket and the metadata table. The IAM role or user you're using should have the appropriate permissions to read from the S3 bucket and query the metadata table.

Remember that the S3 Metadata feature is currently in preview release and subject to change. If you continue to experience issues, you may want to check the latest AWS documentation or contact AWS support for the most up-to-date information on using this feature with Apache Spark.
Sources
Querying metadata tables with open-source query engines - Amazon Simple Storage Service
Configuring metadata tables - Amazon Simple Storage Service

answered 2 years ago

0

Hello,

After the adjustments of the updates from the AWS documentation, the queries that looks like below work on localhost: connect to AWS S3 and list data from configured table-bucket: my_namespace.my_table.

spark-shell 
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,software.amazon.s3tables:s3-tables-catalog-for-iceberg-runtime:0.1.3,software.amazon.awssdk:bundle:2.29.43 
--conf spark.sql.catalog.s3tablesbucket=org.apache.iceberg.spark.SparkCatalog 
--conf spark.sql.catalog.s3tablesbucket.catalog-impl=software.amazon.s3tables.iceberg.S3TablesCatalog 
--conf spark.sql.catalog.s3tablesbucket.warehouse=<arn:aws:s3tables:us-east-1:111122223333:bucket/amzn-s3-demo-bucket1>
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

scala> spark.sql(" SELECT * FROM s3tablesbucket.my_namespace.`my_table` ").show()

Have a good day,
Mihai ADAM

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.