Spark shuffle huge amount of data even read data is not huge

Reading few gb say 15gb of parquet skewed data , after few transformation such as data type change for some columns and then doing repartitions (dataframe.repartition(120)) before writing it to s3 in csv gzip format results in huge amount of shuffle writes as can be seen in spark UI though Input data size is 15 gb , shuffle write is 600gbs

Interested to know why its happening ?

主題

分析資料庫儲存空間

標籤

AWS Glue 提取轉換與載入資料 Amazon GameSparks S3 Select

語言

English

Bibhu

已提問 1 個月前檢視次數 284 次

1 個回答

最新
最多得票
最多評論

這些答案是否有幫助？支持正確答案，以幫助社區從您的知識中受益。

That number is normally larger (e.g. 2x) because it compresses rows while parquet columnar compression is much more efficient.
Must mean in your data there are many columns with repeated values.

專家

Gonzalo Herreros

已回答 1 個月前

Bibhu
1 個月前
Can you please explain a bit more. I dont have any repeated values as such but few values are nulls.

How we can optimise .Because without repartition I tried writing to s3 in csv format ,its 500 GB of data
Gonzalo Herreros 專家
1 個月前
Avoid the shuffle if you can, otherwise don't worry too much about the amount, the transfer is quite fast
Bibhu
1 個月前
Data is skewed, so using repartition to distribute the data evenly which is resulting in huge shuffle writes. Even without repartition it is taking around 1 hr to complete with G2.x and 60 DPUs
Bibhu
1 個月前
These parquet data is being read from Glue Catalog Tables directly.

Spark shuffle huge amount of data even read data is not huge

相關內容