Filtering / Paging gives strange results

0

I have a dataset with seven records:

Records

I want to return all records between 14 and 20 Dec, paging them two at a time. This is my first call, for the first page of results:

Dictionary<string, AttributeValue> expressionAttributeValues = new Dictionary<string, AttributeValue>();
            Dictionary<string, string> expressionAttributeNames = new Dictionary<string, string>();
            string filterExpression = string.Empty;
            AmazonDynamoDBClient client = new Amazon.DynamoDBv2.AmazonDynamoDBClient(creds);
            List<RecordingMetaData> recordings = new List<RecordingMetaData>();

            expressionAttributeValues.Add(":startTime", new AttributeValue { S = $"2022-12-14" });
            expressionAttributeValues.Add(":endTime", new AttributeValue { S = $"2022-12-20" });
            filterExpression = $"#start > :startTime AND #start <= :endTime";
            expressionAttributeNames.Add("#start", "StartTime");

            ScanRequest scanRequest = new ScanRequest()
            {
                TableName = "SCRecordingMetaData",
                ExpressionAttributeNames = expressionAttributeNames,
                FilterExpression = filterExpression.Length > 0 ? filterExpression : null,
                ExpressionAttributeValues = expressionAttributeValues,
                Limit = 2
            };

            var response = new ScanResponse();
            do
            {
                response = client.Scan(scanRequest);

                response.Items.ForEach(i => recordings.Add(new RecordingMetaData()
                {
                    RecordingID = i["RecordingID"].S,
                    Duration = Convert.ToInt32(i["Duration"].N),
                    StartTime = Convert.ToDateTime(i["StartTime"].S),
                    EndTime = Convert.ToDateTime(i["EndTime"].S),
                    ANI = i["ANI"].S,
                    DNS = i["DNS"].S,
                    CreatedTime = Convert.ToDateTime(i["CreatedTime"].S),
                    FilePath = i["FilePath"].S,
                    AgentIDs = i.ContainsKey("AgentIDs") ? i["AgentIDs"].L : null,
                    QueueIDs = i.ContainsKey("QueueIDs") ? i["QueueIDs"].L : null
                }));
            }
            while (response.LastEvaluatedKey.Count > 1);

it should, and does, return 3 records when there's no Limit set in the request. When it's set to 2, I would expect 2 results to be returned for the first page, with the LastEvaluatedKey being "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad" since this is the second key in the list that meets the filter criteria. However, only one result is returned, "2a45d769-fc43-4325-96ec-76e5cf0750fe" and this is also the LastEvaluatedKey. If I pass this value to the ExclusiveStartKey of the next call I get 0 results returned, but the LastEvaluatedKey is "8bce9df3-3921-4259-a70a-c9cf6d34f61f".

So, what seems to be happening is that it only returns records that meet the filter criteria, but it is still in some way recognising that the other records that haven't been returned still exist, i.e. the first record set with no search filter should have "8c8c7afb-e529-4cb5-adb2-ab79d51f5741" and "2a45d769-fc43-4325-96ec-76e5cf0750fe", but because of the filter it only returns one record "2a45d769-fc43-4325-96ec-76e5cf0750fe". Surely it should return "2a45d769-fc43-4325-96ec-76e5cf0750fe" and "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad".

How can you write any kind of paging with behaviour like this?

Nick
asked a year ago227 views
1 Answer
0
Accepted Answer

Hi Nick,

the behavior you experience is due to the Limit parameter in the DynamoDB Scan operation: when Limit is set, DynamoDB fetches up to that number of records and then applies the filtering criteria.

The first paragraph in the following documentation section explains the behavior: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.Limit

To my knowledge, to paginate the results as you desire using Java you'll have to implement your own client code.

The Python Boto3 SDK, on the other hand provides paginators that can be used to perform this use case:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Paginator.Scan

Cheers,

Massimiliano

AWS
EXPERT
answered a year ago
  • Thanks for your response. I was worried that was going to be the case. I'm using the .Net SDK, so Boto3 is not an option. I have already implemented pagination, allowing scrolling next/prev, so putting another layer to ensure I'm getting the same number of records each time will require a bit of a rethink.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions