Improve slow DMS table validation query

0

DMS is using this query to perform table validation post full load migration (in this case on a MySQL source) :

SELECT tableID FROM ( SELECT @row:=@row+1 AS rownum, tableID FROM (SELECT @row:=0) v, ( SELECT tableID FROM mySchema.bigTable ORDER BY tableID ASC ) AS sorted ) as ranked WHERE rownum % [PartitionSize] = [PartitionNumber]

It is quite slow on a table with over 10 million rows (20 seconds for each 10000 partition pass) or painfully slow on a table with over 50 million rows (around 7 minutes for each 10000 partition pass).

Why not use this much faster query until no rows are returned?

MySQL:

SELECT tableID FROM mySchema.bigTable ORDER BY tableID ASC LIMIT [PartitionNumber]*[PartitionSize],[PartitionSize]

Oracle / PostgreSQL:

SELECT tableID FROM mySchema.bigTable ORDER BY tableID ASC LIMIT [PartitionSize] OFFSET [PartitionNumber]*[PartitionSize]

SQL Server:

SELECT tableID FROM mySchema.bigTable ORDER BY tableID ASC OFFSET [PartitionNumber]*[PartitionSize] ROWS FETCH FIRST [PartitionSize] ROWS ONLY

profile picture
demandé il y a un an99 vues
Aucune réponse

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions