Filtering / Paging gives strange results

0

I have a dataset with seven records:

Records

I want to return all records between 14 and 20 Dec, paging them two at a time. This is my first call, for the first page of results:

Dictionary<string, AttributeValue> expressionAttributeValues = new Dictionary<string, AttributeValue>();
            Dictionary<string, string> expressionAttributeNames = new Dictionary<string, string>();
            string filterExpression = string.Empty;
            AmazonDynamoDBClient client = new Amazon.DynamoDBv2.AmazonDynamoDBClient(creds);
            List<RecordingMetaData> recordings = new List<RecordingMetaData>();

            expressionAttributeValues.Add(":startTime", new AttributeValue { S = $"2022-12-14" });
            expressionAttributeValues.Add(":endTime", new AttributeValue { S = $"2022-12-20" });
            filterExpression = $"#start > :startTime AND #start <= :endTime";
            expressionAttributeNames.Add("#start", "StartTime");

            ScanRequest scanRequest = new ScanRequest()
            {
                TableName = "SCRecordingMetaData",
                ExpressionAttributeNames = expressionAttributeNames,
                FilterExpression = filterExpression.Length > 0 ? filterExpression : null,
                ExpressionAttributeValues = expressionAttributeValues,
                Limit = 2
            };

            var response = new ScanResponse();
            do
            {
                response = client.Scan(scanRequest);

                response.Items.ForEach(i => recordings.Add(new RecordingMetaData()
                {
                    RecordingID = i["RecordingID"].S,
                    Duration = Convert.ToInt32(i["Duration"].N),
                    StartTime = Convert.ToDateTime(i["StartTime"].S),
                    EndTime = Convert.ToDateTime(i["EndTime"].S),
                    ANI = i["ANI"].S,
                    DNS = i["DNS"].S,
                    CreatedTime = Convert.ToDateTime(i["CreatedTime"].S),
                    FilePath = i["FilePath"].S,
                    AgentIDs = i.ContainsKey("AgentIDs") ? i["AgentIDs"].L : null,
                    QueueIDs = i.ContainsKey("QueueIDs") ? i["QueueIDs"].L : null
                }));
            }
            while (response.LastEvaluatedKey.Count > 1);

it should, and does, return 3 records when there's no Limit set in the request. When it's set to 2, I would expect 2 results to be returned for the first page, with the LastEvaluatedKey being "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad" since this is the second key in the list that meets the filter criteria. However, only one result is returned, "2a45d769-fc43-4325-96ec-76e5cf0750fe" and this is also the LastEvaluatedKey. If I pass this value to the ExclusiveStartKey of the next call I get 0 results returned, but the LastEvaluatedKey is "8bce9df3-3921-4259-a70a-c9cf6d34f61f".

So, what seems to be happening is that it only returns records that meet the filter criteria, but it is still in some way recognising that the other records that haven't been returned still exist, i.e. the first record set with no search filter should have "8c8c7afb-e529-4cb5-adb2-ab79d51f5741" and "2a45d769-fc43-4325-96ec-76e5cf0750fe", but because of the filter it only returns one record "2a45d769-fc43-4325-96ec-76e5cf0750fe". Surely it should return "2a45d769-fc43-4325-96ec-76e5cf0750fe" and "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad".

How can you write any kind of paging with behaviour like this?

Nick
已提問 1 年前檢視次數 237 次
1 個回答
0
已接受的答案

Hi Nick,

the behavior you experience is due to the Limit parameter in the DynamoDB Scan operation: when Limit is set, DynamoDB fetches up to that number of records and then applies the filtering criteria.

The first paragraph in the following documentation section explains the behavior: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.Limit

To my knowledge, to paginate the results as you desire using Java you'll have to implement your own client code.

The Python Boto3 SDK, on the other hand provides paginators that can be used to perform this use case:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Paginator.Scan

Cheers,

Massimiliano

AWS
專家
已回答 1 年前
  • Thanks for your response. I was worried that was going to be the case. I'm using the .Net SDK, so Boto3 is not an option. I have already implemented pagination, allowing scrolling next/prev, so putting another layer to ensure I'm getting the same number of records each time will require a bit of a rethink.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南