How do I troubleshoot issues with read API operations in DynamoDB?

Short description

When you use read API operations to access data from a DynamoDB table, you might encounter the following issues:

• Your scan and query operations return only partial or no data.
• The page size of your DynamoDB read results is too small.
• You're missing pagination, such as scan and query.
• You get validation exceptions because of syntax errors.
• You don't know how to query items that have maps or lists.

Resolution

The following best practices use the AWS Command Line Interface (AWS CLI) and the Amazon SDK for examples of the API requests.

Note: If you receive errors when you run AWS CLI commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

You get a ValidationException error

The ValidationException error can occur for reasons such as, a data type mismatch or a missing parameter. Review the syntax of your requests to make sure that there's no data type mismatch or missing parameter.

In the following example, a GetItem API request is used. The GetItem request can retrieve a single item from DynamoDB table. To get the item, the request requires a mandatory primary key value. For a primary key, you can use a partition key value. For a composite primary key, you can use a combination of partition key and sort key values.

Note: The GetItem operation isn't allowed on DynamoDB secondary indexes.

CLI example:

aws dynamodb get-item \
--table-name Movies \
--key '{"year": {"N": "1933"}, "title": {"S": "King Kong"}}'

SDK example:

import boto3

client = boto3.client('dynamodb')

response = client.get_item(
    TableName='Movies',
    Key={
        'year': {'N': '1933'},
        'title': {'S': 'King Kong'}
    }
)

print(response["Item"])

You get an "UnprocessedKey" parameter in the response when you invoke a BatchGetItem request

The BatchGetItem API request retrieves up to 16 MB of data, and can contain as many as 100 items from multiple tables.

BatchGetItem returns a partial response in the following scenarios:

• The response size is exceeded.
• The table's provisioned throughput is exceeded.
• More than 1 MB per partition is requested.
• An internal processing failure occurs.

If your request returns a partial result, then note the UnprocessedKeys parameter in the API response. Use this parameter to retry the operation. Start with the next item to get.

CLI example:

aws dynamodb batch-get-item \
--request-items \
'{
    "Movies": {
        "Keys": [
            {"year": {"N": "2014"}, "title": {"S": "Chef"}}, 
            {"year": {"N": "1933"}, "title": {"S": "King Kong"}}
            ]
        },
    "Product": {
        "Keys": [
            {"uuid": {"S": "734838426"}}
        ]
    }
}'

SDK example:

import boto3

client = boto3.client('dynamodb')

result = []

response = client.batch_get_item(
    RequestItems = {
        "Movies": {
            "Keys": [
                {"year": {"N": "2014"}, "title": {"S": "Chef"}}, 
                {"year": {"N": "1933"}, "title": {"S": "King Kong"}}
                ]
            },
        "Product": {
            "Keys": [
                {"uuid": {"S": "734838426"}}
            ]
        }
    }
)

result.extend(response['Responses'].values())

unProcessedKeys = response['UnprocessedKeys']

# as long as there are UnprocessedKeys in the request, it will resend the request with the remaining UnprocessedKeys
while unProcessedKeys:
    response = client.batch_get_item(RequestItems=unProcessedKeys)

    if response['Responses']:
        result.extend(response['Responses'].values())

    unProcessedKeys = response['UnprocessedKeys']

print(result)

You don't know how to query an attribute of map data-type or a list data-type in your table item

To access nested data, such as in a map or a list, use dereference operators.

In the following example, a Query API operation is used. The Query API request can query tables, local secondary indexes, and global secondary indexes. To get results, this request requires a partition key. The Query request returns all items that have the same partition key value provided in the request. To refine the results, you can include a sort key value.

To make a Query, specify a mandatory KeyConditionExpression parameter in the Query request that contains the value of the table's partition key.

Note: KeyConditionExpression accepts only a partition key or an optional sort key.

Optionally, you can provide a FilterExpression parameter to determine what items in the Query are returned.

For a single query, results are limited to a 1 MB page size that you can't increase. To iterate through the query results, use the LastEvaluatedKey parameter. LastEvaluatedKey is used in the ExclusiveStartKey parameter of subsequent query requests until either LastEvaluatedKey is null or the response no longer contains this value.

Note: FilterExpression is applied after the results are fetched from the DynamoDB table. Only items that match the filter expression are returned.

The following SDK example gets data that's stored as maps. The info attribute is the map and rank is the key for the map. To get the data, add a period between the map and the key when you run your command:

CLI example:

aws dynamodb query \
--table-name Movies \
--key-condition-expression '#year = :yearData' \
--filter-expression '#info.#rank < :rankData' \
--expression-attribute-names '{"#year": "year", "#rank": "rank", "#info": "info"}' \
--expression-attribute-values '{":yearData": {"N": "2014"}, ":rankData": {"N": "500"}}'

SDK example:

import boto3

client = boto3.client('dynamodb')

response = client.query(
        TableName='Movies',
        KeyConditionExpression='#year = :yearData',
        FilterExpression='#info.#rank < :rankData',
        ExpressionAttributeNames={
            "#year": "year", 
            "#rank": "rank", 
            "#info": "info"},
        ExpressionAttributeValues={
            ":yearData": {"N": "2014"}, 
            ":rankData": {"N": "500"}
        }
    )

print(response['Items'])

For Scan requests, you get partial or no results

The Scan API operation accesses each item in a table or secondary index and returns items from a DynamoDB table or index. Scan requests have a limit of 1MB for the page size that you can't increase. To get the entire results, you must iterate through all result pages.

Because the Scan operation scans the entire table or index, the operation doesn't require mandatory key conditions. To filter results, use the FilterExpression parameter.

Note: Because of the additional costs, it's a best practice not to use the Scan operation. Instead, use the Query operation. For more information, see Best practices for querying and scanning data.

The AWS CLI automatically manages pagination. However, in the following AWS CLI example, the page-size parameter limits the number of items per page.

In the following examples, the Scan request is paginated. You can also review all pages to see the full results.

CLI example:

aws dynamodb scan \
--table-name Product \
--filter-expression 'Country = :countryData' \
--expression-attribute-values '{":countryData": {"S": "Macedonia"}}' \
 --page-size 5

SDK example:

import boto3

client = boto3.client('dynamodb')
result = []

# fetch first resultant page
response = client.scan(
        TableName='Product',
        FilterExpression='Country = :countryData',
        ExpressionAttributeValues={
            ":countryData": {"S": "Macedonia"}
        }
    )
result.extend(response['Items'])

# loop through all the pages of the result
while("LastEvaluatedKey" in response):
    response = client.scan(
        TableName='Product',
        FilterExpression='Country = :countryData',
        ExpressionAttributeValues={
            ":countryData": {"S": "Macedonia"}
        }
    )
    result.extend(response['Items'])
    print(response['Items'])


print(result)

Related information

Error handling with DynamoDB