Filtering / Paging gives strange results

0

I have a dataset with seven records:

Records

I want to return all records between 14 and 20 Dec, paging them two at a time. This is my first call, for the first page of results:

Dictionary<string, AttributeValue> expressionAttributeValues = new Dictionary<string, AttributeValue>();
            Dictionary<string, string> expressionAttributeNames = new Dictionary<string, string>();
            string filterExpression = string.Empty;
            AmazonDynamoDBClient client = new Amazon.DynamoDBv2.AmazonDynamoDBClient(creds);
            List<RecordingMetaData> recordings = new List<RecordingMetaData>();

            expressionAttributeValues.Add(":startTime", new AttributeValue { S = $"2022-12-14" });
            expressionAttributeValues.Add(":endTime", new AttributeValue { S = $"2022-12-20" });
            filterExpression = $"#start > :startTime AND #start <= :endTime";
            expressionAttributeNames.Add("#start", "StartTime");

            ScanRequest scanRequest = new ScanRequest()
            {
                TableName = "SCRecordingMetaData",
                ExpressionAttributeNames = expressionAttributeNames,
                FilterExpression = filterExpression.Length > 0 ? filterExpression : null,
                ExpressionAttributeValues = expressionAttributeValues,
                Limit = 2
            };

            var response = new ScanResponse();
            do
            {
                response = client.Scan(scanRequest);

                response.Items.ForEach(i => recordings.Add(new RecordingMetaData()
                {
                    RecordingID = i["RecordingID"].S,
                    Duration = Convert.ToInt32(i["Duration"].N),
                    StartTime = Convert.ToDateTime(i["StartTime"].S),
                    EndTime = Convert.ToDateTime(i["EndTime"].S),
                    ANI = i["ANI"].S,
                    DNS = i["DNS"].S,
                    CreatedTime = Convert.ToDateTime(i["CreatedTime"].S),
                    FilePath = i["FilePath"].S,
                    AgentIDs = i.ContainsKey("AgentIDs") ? i["AgentIDs"].L : null,
                    QueueIDs = i.ContainsKey("QueueIDs") ? i["QueueIDs"].L : null
                }));
            }
            while (response.LastEvaluatedKey.Count > 1);

it should, and does, return 3 records when there's no Limit set in the request. When it's set to 2, I would expect 2 results to be returned for the first page, with the LastEvaluatedKey being "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad" since this is the second key in the list that meets the filter criteria. However, only one result is returned, "2a45d769-fc43-4325-96ec-76e5cf0750fe" and this is also the LastEvaluatedKey. If I pass this value to the ExclusiveStartKey of the next call I get 0 results returned, but the LastEvaluatedKey is "8bce9df3-3921-4259-a70a-c9cf6d34f61f".

So, what seems to be happening is that it only returns records that meet the filter criteria, but it is still in some way recognising that the other records that haven't been returned still exist, i.e. the first record set with no search filter should have "8c8c7afb-e529-4cb5-adb2-ab79d51f5741" and "2a45d769-fc43-4325-96ec-76e5cf0750fe", but because of the filter it only returns one record "2a45d769-fc43-4325-96ec-76e5cf0750fe". Surely it should return "2a45d769-fc43-4325-96ec-76e5cf0750fe" and "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad".

How can you write any kind of paging with behaviour like this?

Nick
asked a year ago198 views
1 Answer
1

This is how filtering on DynamoDB works. It first reads up to the number of items defined by Limit and then applies the filter. In your case you set Limit=2 meaning you will get ~2 items, but not guaranteed to be two.

A FilterExpression is applied after the items have already been read; the process of filtering does not consume any additional read capacity units.

https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html#DDB-Scan-request-FilterExpression

profile pictureAWS
EXPERT
answered a year ago
  • I guess that's logical, but not great when trying to explain to non-technical users why there's a different number of results on each page. I've had to change the design now so that all results are displayed on one page, but there's a "Get More Results" button that add records to the bottom. It's an OK solution.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions