Query with scanIndexForward=true and limit=1

0

I have a dynamo db table that looks something like this:

RequestType (PK)CreationTime (SK)Other
A2Data
A4Data
B1Data
B5Data
B7Data

My main requirement is to fetch the oldest item in a partition (based on CreationTime). Essentially, should behave like a FIFO queue.

I'm using DynamoDbMapper to query this table using the partition key value, scanIndexForward=true and limit=1. Now when i try to get the first item in the results, it takes around 25 seconds (there were ~15,000 items in the partition). Here's the function used:

public Optional<MyDdbItem> getLastItemInPartition(@NonNull final String partitionKey) {
        final DynamoDBQueryExpression<MyDdbItem> queryExpression =
                new DynamoDBQueryExpression<MyDdbItem>()
                        .withHashKeyValues(MyDdbItem.builder()
                                .requestType(partitionKey).build())
                        // This is set true for ascending order of sort key (creationTime)
                        .withScanIndexForward(true)
                        // Fetch only one item
                        .withLimit(1);
        Optional<MyDdbItem> result = Optional.empty();
        try {
            final PaginatedQueryList<MyDdbItem> results =
                    getDynamoDBMapper().query(MyDdbItem.class, queryExpression);
            result = results.stream().findFirst();
        } catch (final Exception exception) {
            DDBExceptionHandler.handleDDBExceptions(exception);
        }
        return result;
    }

Things i verified:

  1. The table metrics indicate that the query latency has never gone beyond 2 ms.
  2. Performed a similar query from a python script, and it took around 500 ms.

Python script:

def get_last_item_in_partition(table_name, partition_key_value):
    query_params = {
    'TableName': table_name,
    'KeyConditionExpression': '#pk = :pkval',
    'ExpressionAttributeNames': {'#pk': 'RequestType'},
    'ExpressionAttributeValues': {':pkval': {'S': partition_key_value}},
    'ScanIndexForward': True,  
    'Limit': 1 
    } 

    try:
        response = dynamodb.query(**query_params)
        items = response['Items']
        if items:
            return items[0]
        else:
            return None
    except NoCredentialsError as e:
        print(f"Error: {e}")
        return None

Posting here to verify if the index-design accommodates my requirement.

gefragt vor 5 Monaten441 Aufrufe
1 Antwort
1
Akzeptierte Antwort

The issue is the API which you are calling with Mapper:

getDynamoDBMapper().query(MyDdbItem.class, queryExpression);

You are calling the Query which auto-paginates when you use a high-level client such as Mapper. Your Limit=1 limits it to 1 item per page, so that results in you making multiple API requests to DynamoDB to retrieve all of the items, 1 item at a time.

What you need to use is the QueryPage which only fetches a single page from DynamoDB, and as a result will return exactly what you require:

getDynamoDBMapper().queryPage(MyDdbItem.class, queryExpression);

profile pictureAWS
EXPERTE
beantwortet vor 5 Monaten
profile pictureAWS
EXPERTE
überprüft vor 5 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen