Improve slow DMS table validation query

0

DMS is using this query to perform table validation post full load migration (in this case on a MySQL source) :

SELECT tableID FROM ( SELECT @row:=@row+1 AS rownum, tableID FROM (SELECT @row:=0) v, ( SELECT tableID FROM mySchema.bigTable ORDER BY tableID ASC ) AS sorted ) as ranked WHERE rownum % [PartitionSize] = [PartitionNumber]

It is quite slow on a table with over 10 million rows (20 seconds for each 10000 partition pass) or painfully slow on a table with over 50 million rows (around 7 minutes for each 10000 partition pass).

Why not use this much faster query until no rows are returned?

MySQL:

SELECT tableID FROM mySchema.bigTable ORDER BY tableID ASC LIMIT [PartitionNumber]*[PartitionSize],[PartitionSize]

Oracle / PostgreSQL:

SELECT tableID FROM mySchema.bigTable ORDER BY tableID ASC LIMIT [PartitionSize] OFFSET [PartitionNumber]*[PartitionSize]

SQL Server:

SELECT tableID FROM mySchema.bigTable ORDER BY tableID ASC OFFSET [PartitionNumber]*[PartitionSize] ROWS FETCH FIRST [PartitionSize] ROWS ONLY

profile picture
질문됨 일 년 전99회 조회
답변 없음

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인