When configuring an EMR Serverless application, you can choose the disk size for your preinitialized capacity and a maximum disk limit for the application: ![Disk size](/media/postImages/original/IMeTrUgzw3RciHzq-O7IJugw) However, at least for Hive jobs, EMR Serverless stores both the scratch and warehouse directories on S3. So what is this configured disk space used for? I've searched the documentation and there's no mention of how this space is used, or why it's needed.

EMR Serverless uses disk size for the following purposes. 1. Storing the log files ( Hive, Tez AM logs, Tez Task logs etc) 2. Shuffle data is stored on the local disk. 3. Resources such as aux libraries, user jar etc. Hope it answers your question.

EMR Serverless Application Disk Space?

When configuring an EMR Serverless application, you can choose the disk size for your preinitialized capacity and a maximum disk limit for the application:

Disk size

However, at least for Hive jobs, EMR Serverless stores both the scratch and warehouse directories on S3. So what is this configured disk space used for? I've searched the documentation and there's no mention of how this space is used, or why it's needed.

トピック

分析ストレージ

タグ

Amazon EMR ストレージ Amazon EMR Serverless

言語

English

Patrick

質問済み 2年前1108ビュー

1回答

新しい順
投票が多い順
コメントが多い順

承認された回答

EMR Serverless uses disk size for the following purposes.

Storing the log files ( Hive, Tez AM logs, Tez Task logs etc)
Shuffle data is stored on the local disk.
Resources such as aux libraries, user jar etc.

Hope it answers your question.

Shailesh

回答済み 1年前

サポートエンジニア

Yokesh NK

レビュー済み 23日前

Patrick
1年前
This does answer the question, thanks. But we needed this answer 3 months ago. We've since moved our company's infrastructure to a new cloud provider.

EMR Serverless Application Disk Space?

関連するコンテンツ