How can I change default s3 storage class of Hive connector of EMR Trino?

0

I want to change the default s3 storage class to INTELLIGENT_TIERING of Hive connector of EMR Trino 426 (EMR 6.15.0). I found the hive.s3.storage-class option in the Trino 426 official manual but it doesn't work as below.

> 1) Error: Configuration property 'hive.s3.storage-class' was not used

1 error
io.airlift.bootstrap.ApplicationConfigurationException: Configuration errors:

1) Error: Configuration property 'hive.s3.storage-class' was not used

1 error
	at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:232)
	at io.trino.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:148)
	at io.trino.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:90)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at io.trino.plugin.hive.HiveConnectorFactory.create(HiveConnectorFactory.java:61)
	at io.trino.connector.DefaultCatalogFactory.createConnector(DefaultCatalogFactory.java:224)
	at io.trino.connector.DefaultCatalogFactory.createCatalog(DefaultCatalogFactory.java:133)
	at io.trino.connector.LazyCatalogFactory.createCatalog(LazyCatalogFactory.java:45)
	at io.trino.connector.StaticCatalogManager.lambda$loadInitialCatalogs$1(StaticCatalogManager.java:157)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
	at io.trino.util.Executors.executeUntilFailure(Executors.java:41)
	at io.trino.connector.StaticCatalogManager.loadInitialCatalogs(StaticCatalogManager.java:151)
	at io.trino.server.Server.doStart(Server.java:144)
	at io.trino.server.Server.lambda$start$0(Server.java:91)
	at io.trino.$gen.Trino_426_amzn_0____20240427_030723_1.run(Unknown Source)
	at io.trino.server.Server.start(Server.java:91)
	at io.trino.server.TrinoServer.main(TrinoServer.java:38)
asked 12 days ago96 views
2 Answers
0
Accepted Answer

Found the anwser by myself. EMR Trino uses EMRFS as its storage backend. It means we have to change the storage class setting of EMRFS.

    {
      "Classification": "emrfs-site",
      "Properties": {
        "fs.s3.storageClass": "INTELLIGENT_TIERING"
      }
    }
answered 12 days ago
AWS
SUPPORT ENGINEER
reviewed 10 days ago
0

I found hive.insert-existing-partitions-behavior=STORAGE_CODE:INTELLIGENT_TIERING is the solution in claude 3. Is it right?

answered 12 days ago
  • I tested but it's not.

    1) Error: Invalid value 'STORAGE_CODE:INTELLIGENT_TIERING' for type HiveSessionProperties$InsertExistingPartitionsBehavior (property 'hive.insert-existing-partitions-behavior') in order to call [public HiveConfig HiveConfig.setInsertExistingPartitionsBehavior(HiveSessionProperties$InsertExistingPartitionsBehavior)]
    
    2) Error: Invalid value 'STORAGE_CODE:INTELLIGENT_TIERING' for type HiveSessionProperties$InsertExistingPartitionsBehavior (property 'hive.insert-existing-partitions-behavior') in order to call [public HiveConfig HiveConfig.setInsertExistingPartitionsBehavior(HiveSessionProperties$InsertExistingPartitionsBehavior)]
    
    3) Error: Invalid value 'STORAGE_CODE:INTELLIGENT_TIERING' for type HiveSessionProperties$InsertExistingPartitionsBehavior (property 'hive.insert-existing-partitions-behavior') in order to call [public HiveConfig HiveConfig.setInsertExistingPartitionsBehavior(HiveSessionProperties$InsertExistingPartitionsBehavior)]
    
    3 errors
    

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions