How can I change default s3 storage class of Hive connector of EMR Trino?

0

I want to change the default s3 storage class to INTELLIGENT_TIERING of Hive connector of EMR Trino 426 (EMR 6.15.0). I found the hive.s3.storage-class option in the Trino 426 official manual but it doesn't work as below.

> 1) Error: Configuration property 'hive.s3.storage-class' was not used

1 error
io.airlift.bootstrap.ApplicationConfigurationException: Configuration errors:

1) Error: Configuration property 'hive.s3.storage-class' was not used

1 error
	at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:232)
	at io.trino.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:148)
	at io.trino.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:90)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at io.trino.plugin.hive.HiveConnectorFactory.create(HiveConnectorFactory.java:61)
	at io.trino.connector.DefaultCatalogFactory.createConnector(DefaultCatalogFactory.java:224)
	at io.trino.connector.DefaultCatalogFactory.createCatalog(DefaultCatalogFactory.java:133)
	at io.trino.connector.LazyCatalogFactory.createCatalog(LazyCatalogFactory.java:45)
	at io.trino.connector.StaticCatalogManager.lambda$loadInitialCatalogs$1(StaticCatalogManager.java:157)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
	at io.trino.util.Executors.executeUntilFailure(Executors.java:41)
	at io.trino.connector.StaticCatalogManager.loadInitialCatalogs(StaticCatalogManager.java:151)
	at io.trino.server.Server.doStart(Server.java:144)
	at io.trino.server.Server.lambda$start$0(Server.java:91)
	at io.trino.$gen.Trino_426_amzn_0____20240427_030723_1.run(Unknown Source)
	at io.trino.server.Server.start(Server.java:91)
	at io.trino.server.TrinoServer.main(TrinoServer.java:38)
gefragt vor einem Monat165 Aufrufe
2 Antworten
0
Akzeptierte Antwort

Found the anwser by myself. EMR Trino uses EMRFS as its storage backend. It means we have to change the storage class setting of EMRFS.

    {
      "Classification": "emrfs-site",
      "Properties": {
        "fs.s3.storageClass": "INTELLIGENT_TIERING"
      }
    }
beantwortet vor einem Monat
AWS
SUPPORT-TECHNIKER
überprüft vor 25 Tagen
0

I found hive.insert-existing-partitions-behavior=STORAGE_CODE:INTELLIGENT_TIERING is the solution in claude 3. Is it right?

beantwortet vor einem Monat
  • I tested but it's not.

    1) Error: Invalid value 'STORAGE_CODE:INTELLIGENT_TIERING' for type HiveSessionProperties$InsertExistingPartitionsBehavior (property 'hive.insert-existing-partitions-behavior') in order to call [public HiveConfig HiveConfig.setInsertExistingPartitionsBehavior(HiveSessionProperties$InsertExistingPartitionsBehavior)]
    
    2) Error: Invalid value 'STORAGE_CODE:INTELLIGENT_TIERING' for type HiveSessionProperties$InsertExistingPartitionsBehavior (property 'hive.insert-existing-partitions-behavior') in order to call [public HiveConfig HiveConfig.setInsertExistingPartitionsBehavior(HiveSessionProperties$InsertExistingPartitionsBehavior)]
    
    3) Error: Invalid value 'STORAGE_CODE:INTELLIGENT_TIERING' for type HiveSessionProperties$InsertExistingPartitionsBehavior (property 'hive.insert-existing-partitions-behavior') in order to call [public HiveConfig HiveConfig.setInsertExistingPartitionsBehavior(HiveSessionProperties$InsertExistingPartitionsBehavior)]
    
    3 errors
    

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen