How do I implement pagination in Amazon DynamoDB?

3 minute read
0

When I use a Query or Scan operation to fetch items from an Amazon DynamoDB table, the response doesn't return the complete results.

Resolution

This issue occurs because of these two conditions:

  • Your initial Query or Scan operation contains LastEvaluatedKey in the response.
  • You didn't use this key as ExclusiveStartKey for the next request to perform a subsequent operation.

To resolve this issue, implement pagination for your DynamoDB table. Pagination is the process of sending subsequent requests to continue when a previous request is incomplete. A Query or Scan operation in DynamoDB might return results that are incomplete and require subsequent requests to get the entire result set. This is because DynamoDB paginates the results from a Query or Scan operation and returns a maximum of 1MB of data in a single operation. The data maximum is a hard limit in DynamoDB. With pagination, the results from the Scan and Query operations are divided into pages of data that are 1 MB or smaller.

To implement pagination in Amazon DynamoDB, use the built-in pagination functionality. Run a command similar to this example:

use DynamoDB.Paginator.Query.paginate

client = boto3.client("dynamodb")
paginator = client.get_paginator("scan")
params = {}

for page in paginator.paginate(params):
# do something

You can also paginate your results and retrieve them one page at a time. To see if the result contains the LastEvaluatedKey element, check the low-level result from a Scan or Query operation. To get the remaining results, use another Scan or Query operation with the same parameters. For the second operation, use the LastEvaluatedKey as the ExclusiveStartKey parameter. If the result doesn't include the LastEvaluatedKey value, then there are no items to retrieve.

For more information, see Paginating table query results and Paginating the results.

To use LastEvaluatedKey to implement pagination in Boto3, run a command similar to this example:

from __future__ import print_function  # Python 2/3 compatibility
import boto3
from botocore.exceptions import ClientError

# Create Client
session = boto3.session.Session()
dynamoDbClient = session.client('dynamodb')

table_name = 'AmazonBins'

# Track number of Items read
item_count = 0

try:    
    # Get the first 1MB of data    
    response = dynamoDbClient.scan(
        TableName=table_name
    )

except ClientError as error:
    print("Something went wrong: ")
    print(error.response['ResponseMetadata'])

# Track number of Items read
item_count += len(response['Items'])

# Paginate returning up to 1MB of data for each iteration
while 'LastEvaluatedKey' in response:
    try:
        response = dynamoDbClient.scan(
            TableName=table_name,
            ExclusiveStartKey=response['LastEvaluatedKey']
        )
        # Track number of Items read
        item_count += len(response['Items'])

    except ClientError as error:
        print("Something went wrong: ")
        print(error.response['ResponseMetadata'])

print("Total number of items found: {}".format(item_count))
AWS OFFICIAL
AWS OFFICIALUpdated a year ago