How do I implement pagination in Amazon DynamoDB?

Resolution

This issue occurs under the following conditions:

• Your initial Scan or Query operation contains LastEvaluatedKey in the response.
• You didn't perform a subsequent operation using this key as ExclusiveStartKey for the next request.

You can resolve this issue by implementing pagination for your DynamoDB table. Pagination is the process of sending subsequent requests to continue when a previous request is incomplete. A Query or Scan operation in DynamoDB might return results that are incomplete and require subsequent requests to attain the entire result set. This is because DynamoDB paginates the results from a Query or Scan operation and returns a maximum of 1MB of data in a single operation. This is a hard limit in DynamoDB. With pagination, the results from the Scan and Query operations are divided into pages of data that are 1 MB or less in size.

To paginate your results and retrieve them one page at a time, check the low-level result from a Scan or Query operation to see if the result contains the LastEvaluatedKey element. You can get the remaining results of the initial request using another Scan or Query operation with the same parameters, but using the LastEvaluatedKey value as the ExclusiveStartKey parameter. If the result doesn't include the LastEvaluatedKey value, then there are no items to be retrieved.

For more information, see Paginating table query results and Paginating the results.

To implement pagination in Boto3 using LastEvaluatedKey, use the following code sample:

from __future__ import print_function  # Python 2/3 compatibility
import boto3
from botocore.exceptions import ClientError

# Create Client
session = boto3.session.Session()
dynamoDbClient = session.client('dynamodb')

table_name = 'AmazonBins'

# Track number of Items read
item_count = 0

try:
    # Get the first 1MB of data
    response = dynamoDbClient.scan(
        TableName=table_name
    )

except ClientError as error:
    print("Something went wrong: ")
    print(error.response['ResponseMetadata'])

# Track number of Items read
item_count += len(response['Items'])

# Paginate returning up to 1MB of data for each iteration
while 'LastEvaluatedKey' in response:
    try:
        response = dynamoDbClient.scan(
            TableName=table_name,
            ExclusiveStartKey=response['LastEvaluatedKey']
        )
        # Track number of Items read
        item_count += len(response['Items'])

    except ClientError as error:
        print("Something went wrong: ")
        print(error.response['ResponseMetadata'])

print("Total number of items found: {}".format(item_count))