Al usar AWS re:Post, aceptas las AWS re:Post Términos de uso
AWS re:Postre:Post

Are partitions advantageous for groupby operations in Glue jobs?

0

I have a table with columns A, B, C, D, ..., where A is a partition key. In a Glue job I want to group records of this table by column A. Is there a way to make the glue workers aware of the partitions in A, such that workers do not have to process dataframes created from different partitions in order to speed up the processing time?

Temas
AnálisisAlmacenamientoBase de datos
Etiquetas
AWS GlueLagos de datosExtraer datos de transformación y carga
Idioma
English
Thomas Mueller
preguntada hace 2 años644 visualizaciones
1 Respuesta
0

Yes, The way to do this is with a pushdown predicate. When reading a dynamic frame, you would use the field push_down_predicate.

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-partitions.html#aws-glue-programming-etl-partitions-pushdowns

https://aws.amazon.com/premiumsupport/knowledge-center/glue-job-specific-s3-partition/

jb-po
respondido hace 2 años
  • Thomas Mueller
    hace 2 años

    In my case, the table has maybe in the order of 1e9 lines which can be grouped into around 1e6 groups. Do I understand it right, that I should then start 1e6 Glue jobs for each partition/group in parallel and perform the selection push_down_predicate? This does not sound practical to me, as I assume that it would be better to effieciently use Glue's internal parallelisation.

  • jb-po
    hace 2 años

    Yeah sorry, I haven't run tests on the scaling of the number of partitions so high with low data, but my assumption is that it scales and using partitions is better since the query you would be making to s3 is using presto to optimize the data grabbed and how it is grabbed with the partitions organization.

    This might help. https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-presto-s3select.html

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas

Contenido relevante