Query with scanIndexForward=true and limit=1

0

I have a dynamo db table that looks something like this:

RequestType (PK)CreationTime (SK)Other
A2Data
A4Data
B1Data
B5Data
B7Data

My main requirement is to fetch the oldest item in a partition (based on CreationTime). Essentially, should behave like a FIFO queue.

I'm using DynamoDbMapper to query this table using the partition key value, scanIndexForward=true and limit=1. Now when i try to get the first item in the results, it takes around 25 seconds (there were ~15,000 items in the partition). Here's the function used:

public Optional<MyDdbItem> getLastItemInPartition(@NonNull final String partitionKey) {
        final DynamoDBQueryExpression<MyDdbItem> queryExpression =
                new DynamoDBQueryExpression<MyDdbItem>()
                        .withHashKeyValues(MyDdbItem.builder()
                                .requestType(partitionKey).build())
                        // This is set true for ascending order of sort key (creationTime)
                        .withScanIndexForward(true)
                        // Fetch only one item
                        .withLimit(1);
        Optional<MyDdbItem> result = Optional.empty();
        try {
            final PaginatedQueryList<MyDdbItem> results =
                    getDynamoDBMapper().query(MyDdbItem.class, queryExpression);
            result = results.stream().findFirst();
        } catch (final Exception exception) {
            DDBExceptionHandler.handleDDBExceptions(exception);
        }
        return result;
    }

Things i verified:

  1. The table metrics indicate that the query latency has never gone beyond 2 ms.
  2. Performed a similar query from a python script, and it took around 500 ms.

Python script:

def get_last_item_in_partition(table_name, partition_key_value):
    query_params = {
    'TableName': table_name,
    'KeyConditionExpression': '#pk = :pkval',
    'ExpressionAttributeNames': {'#pk': 'RequestType'},
    'ExpressionAttributeValues': {':pkval': {'S': partition_key_value}},
    'ScanIndexForward': True,  
    'Limit': 1 
    } 

    try:
        response = dynamodb.query(**query_params)
        items = response['Items']
        if items:
            return items[0]
        else:
            return None
    except NoCredentialsError as e:
        print(f"Error: {e}")
        return None

Posting here to verify if the index-design accommodates my requirement.

asked 5 months ago403 views
1 Answer
1
Accepted Answer

The issue is the API which you are calling with Mapper:

getDynamoDBMapper().query(MyDdbItem.class, queryExpression);

You are calling the Query which auto-paginates when you use a high-level client such as Mapper. Your Limit=1 limits it to 1 item per page, so that results in you making multiple API requests to DynamoDB to retrieve all of the items, 1 item at a time.

What you need to use is the QueryPage which only fetches a single page from DynamoDB, and as a result will return exactly what you require:

getDynamoDBMapper().queryPage(MyDdbItem.class, queryExpression);

profile pictureAWS
EXPERT
answered 5 months ago
profile pictureAWS
EXPERT
reviewed 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions