DynamoDB querying GSI, page.lastEvaluatedKey returns non-null/ non-empty key when there are no more items to be retrieved.

0

I have Project Table with a GSI on the userID.

For a user say TestUser, I have 10 projects in my Dynamo Table/ the GSI FilterByUserID.

I'm using awssdk QueryEnhancedRequest for my query to get these projects:

QueryEnhancedRequest.Builder requestBuilder = QueryEnhancedRequest.builder(); requestBuilder .exclusiveStartKey(<startKey> ) .queryConditional(<Query condition matching the PK of the GSI>) .limit(10) .build();

I then subscribe the pages using

SdkPublisher<Page<Project>> publisher
subscribe(page -> {})

page.lastEvaluatedKey() in this case is expected to be null as the first page returns all the available 10 items/ projects we had in the DDB. But surprisingly page.lastEvaluatedKey() return the 10th item from the GSI that matched the key.

I'm wondering why this would happen, as per documentation https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.Pagination.html

A single Query only returns a result set that fits within the 1 MB size limit. To determine whether there are more results, and to retrieve them one page at a time, applications should do the following: Examine the low-level Query result:

  1. If the result contains a LastEvaluatedKey element and it's non-null, proceed to step 2.
  2. If there is not a LastEvaluatedKey in the result, there are no more items to be retrieved.

Any help to understand this scenario would be appreciated. Thanks

asked 7 months ago453 views
1 Answer
1

This is because you didn't allow DynamoDB to evaluate that there was more than 10. You had exactly 10, you asked for 10, but DynamoDB did not look for an 11th, so it returned you a LastEvaluatedKey which would have returned no items, as only when you return for the second page that DynamoDB continues to evaluate if there is 11,12..... items.

profile pictureAWS
EXPERT
answered 7 months ago
  • Thanks Leeroy Makes sense, but shouldn't the page.lastEvaluatedKey be independent of the limit over the query.

    If there is any documentation over this, that you might have handy please feel free to share with me.

  • You use .limit(10) which informs DynamoDB to obtain only 10 items. So the read request has no idea if more than 10 existed, so it provides you a pointer to the last read item, which is item 10. When it goes back, theres no more items and it now knows that, so LastEvaluatedKey is now empty/null.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions