DynamoDB Pagination - LastEvaluatedKey returns null prematurely

Question

I am currently using the PHP SDK with Laravel to run a filtered scan inside of a do/while loop either until I have the requested number of items or until I get an empty value for the LastEvaluatedKey response property.

I know that the way scans work is it actually moves a "frame" sized by the given query limit and filters items within that frame to build the result set, and that shifting the frame forward means getting identifiable information about the last item in the frame in the LastEvaluatedKey property and using it as the starting key in the next query.

Unfortunately this property is returning null prematurely. I know that it's early because if I extend the limit of the scan, I get more results (of course there is a limit to the efficacy of this as eventually it eats up throughput).

Here's how I know it's exiting prematurely:
| Limit | Loop Count | Total Records |
| --- | --- | --- |
| 100 | 2 | 2  |
| 250 | 3 | 4 |
| 500 | 4 | 11  |
| 750 | 11 | 33  |

In each of these cases, I'm adding to the number of times the query has been run, watching the value of the LastEvaluatedKey in each response, confirming that it is always null in the last one.

The result sets are not large, either. Even the full 33 records from the maximum item count I've received has not resulted in half a megabyte in size in total, much less in a single response payload.

Answer

You should be very careful with this pattern - it is open-ended on throughput consumption. Note that the limit relates to items evaluated - not items returned.

Answer

Hello, 
Thank you for reaching out with DynamoDB query.

From the issue, I understand that you're receiving null prematurely in the LastEvaluatedKey property.

With pagination, the Scan results are divided into "pages" of data that are 1 MB in size (or less). An application can process the first page of results, then the second page, and so on. A single Scan only returns a result set that fits within the 1 MB size limit. 
However, in your case even though your item total 33 records from the max item count received is less than 1MB in total size, items are not retrieved in a single payload.

Since this issues needs further deep dive on the code that you're using and settings, I would suggestyou please reach out to AWS Support[1], along with your issue/use case in detail and share relevant AWS table resource names. So that we can deep dive into the issue further to gain more insights.

Thank you!

[1] https://support.console.aws.amazon.com/

DynamoDB Pagination - LastEvaluatedKey returns null prematurely

Relevant content