Al usar AWS re:Post, aceptas las AWS re:Post Términos de uso
AWS re:Postre:Post

Spark shuffle huge amount of data even read data is not huge

0

Reading few gb say 15gb of parquet skewed data , after few transformation such as data type change for some columns and then doing repartitions (dataframe.repartition(120)) before writing it to s3 in csv gzip format results in huge amount of shuffle writes as can be seen in spark UI though Input data size is 15 gb , shuffle write is 600gbs

Interested to know why its happening ?

Temas
AnálisisBase de datosAlmacenamiento
Etiquetas
AWS GlueExtraer datos de transformación y cargaAmazon GameSparksS3 Select
Idioma
English
Bibhu
preguntada hace un mes284 visualizaciones
1 Respuesta
0

That number is normally larger (e.g. 2x) because it compresses rows while parquet columnar compression is much more efficient.
Must mean in your data there are many columns with repeated values.

profile pictureAWS
EXPERTO
Gonzalo Herreros
respondido hace un mes
  • Bibhu
    hace un mes

    Can you please explain a bit more. I dont have any repeated values as such but few values are nulls.

    How we can optimise .Because without repartition I tried writing to s3 in csv format ,its 500 GB of data

  • Gonzalo Herreros EXPERTO
    hace un mes

    Avoid the shuffle if you can, otherwise don't worry too much about the amount, the transfer is quite fast

  • Bibhu
    hace un mes

    Data is skewed, so using repartition to distribute the data evenly which is resulting in huge shuffle writes. Even without repartition it is taking around 1 hr to complete with G2.x and 60 DPUs

  • Bibhu
    hace un mes

    These parquet data is being read from Glue Catalog Tables directly.

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas

Contenido relevante