When configuring an EMR Serverless application, you can choose the disk size for your preinitialized capacity and a maximum disk limit for the application: ![Disk size](/media/postImages/original/IMeTrUgzw3RciHzq-O7IJugw) However, at least for Hive jobs, EMR Serverless stores both the scratch and warehouse directories on S3. So what is this configured disk space used for? I've searched the documentation and there's no mention of how this space is used, or why it's needed.

EMR Serverless uses disk size for the following purposes. 1. Storing the log files ( Hive, Tez AM logs, Tez Task logs etc) 2. Shuffle data is stored on the local disk. 3. Resources such as aux libraries, user jar etc. Hope it answers your question.

EMR Serverless Application Disk Space?

When configuring an EMR Serverless application, you can choose the disk size for your preinitialized capacity and a maximum disk limit for the application:

Disk size

However, at least for Hive jobs, EMR Serverless stores both the scratch and warehouse directories on S3. So what is this configured disk space used for? I've searched the documentation and there's no mention of how this space is used, or why it's needed.

주제

분석 스토리지

태그

아마존 EMR 스토리지 아마존 EMR 서버리스

언어

English

Patrick

질문됨 2년 전1107회 조회

1개 답변

최신
최다 투표
가장 많은 댓글

수락된 답변

EMR Serverless uses disk size for the following purposes.

Storing the log files ( Hive, Tez AM logs, Tez Task logs etc)
Shuffle data is stored on the local disk.
Resources such as aux libraries, user jar etc.

Hope it answers your question.

Shailesh

답변함 일 년 전

지원 엔지니어

Yokesh NK

검토됨 23일 전

Patrick
일 년 전
This does answer the question, thanks. But we needed this answer 3 months ago. We've since moved our company's infrastructure to a new cloud provider.

EMR Serverless Application Disk Space?

관련 콘텐츠