Filtering / Paging gives strange results

0

I have a dataset with seven records:

Records

I want to return all records between 14 and 20 Dec, paging them two at a time. This is my first call, for the first page of results:

Dictionary<string, AttributeValue> expressionAttributeValues = new Dictionary<string, AttributeValue>();
            Dictionary<string, string> expressionAttributeNames = new Dictionary<string, string>();
            string filterExpression = string.Empty;
            AmazonDynamoDBClient client = new Amazon.DynamoDBv2.AmazonDynamoDBClient(creds);
            List<RecordingMetaData> recordings = new List<RecordingMetaData>();

            expressionAttributeValues.Add(":startTime", new AttributeValue { S = $"2022-12-14" });
            expressionAttributeValues.Add(":endTime", new AttributeValue { S = $"2022-12-20" });
            filterExpression = $"#start > :startTime AND #start <= :endTime";
            expressionAttributeNames.Add("#start", "StartTime");

            ScanRequest scanRequest = new ScanRequest()
            {
                TableName = "SCRecordingMetaData",
                ExpressionAttributeNames = expressionAttributeNames,
                FilterExpression = filterExpression.Length > 0 ? filterExpression : null,
                ExpressionAttributeValues = expressionAttributeValues,
                Limit = 2
            };

            var response = new ScanResponse();
            do
            {
                response = client.Scan(scanRequest);

                response.Items.ForEach(i => recordings.Add(new RecordingMetaData()
                {
                    RecordingID = i["RecordingID"].S,
                    Duration = Convert.ToInt32(i["Duration"].N),
                    StartTime = Convert.ToDateTime(i["StartTime"].S),
                    EndTime = Convert.ToDateTime(i["EndTime"].S),
                    ANI = i["ANI"].S,
                    DNS = i["DNS"].S,
                    CreatedTime = Convert.ToDateTime(i["CreatedTime"].S),
                    FilePath = i["FilePath"].S,
                    AgentIDs = i.ContainsKey("AgentIDs") ? i["AgentIDs"].L : null,
                    QueueIDs = i.ContainsKey("QueueIDs") ? i["QueueIDs"].L : null
                }));
            }
            while (response.LastEvaluatedKey.Count > 1);

it should, and does, return 3 records when there's no Limit set in the request. When it's set to 2, I would expect 2 results to be returned for the first page, with the LastEvaluatedKey being "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad" since this is the second key in the list that meets the filter criteria. However, only one result is returned, "2a45d769-fc43-4325-96ec-76e5cf0750fe" and this is also the LastEvaluatedKey. If I pass this value to the ExclusiveStartKey of the next call I get 0 results returned, but the LastEvaluatedKey is "8bce9df3-3921-4259-a70a-c9cf6d34f61f".

So, what seems to be happening is that it only returns records that meet the filter criteria, but it is still in some way recognising that the other records that haven't been returned still exist, i.e. the first record set with no search filter should have "8c8c7afb-e529-4cb5-adb2-ab79d51f5741" and "2a45d769-fc43-4325-96ec-76e5cf0750fe", but because of the filter it only returns one record "2a45d769-fc43-4325-96ec-76e5cf0750fe". Surely it should return "2a45d769-fc43-4325-96ec-76e5cf0750fe" and "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad".

How can you write any kind of paging with behaviour like this?

Nick
질문됨 일 년 전237회 조회
1개 답변
0
수락된 답변

Hi Nick,

the behavior you experience is due to the Limit parameter in the DynamoDB Scan operation: when Limit is set, DynamoDB fetches up to that number of records and then applies the filtering criteria.

The first paragraph in the following documentation section explains the behavior: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.Limit

To my knowledge, to paginate the results as you desire using Java you'll have to implement your own client code.

The Python Boto3 SDK, on the other hand provides paginators that can be used to perform this use case:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Paginator.Scan

Cheers,

Massimiliano

AWS
전문가
답변함 일 년 전
  • Thanks for your response. I was worried that was going to be the case. I'm using the .Net SDK, so Boto3 is not an option. I have already implemented pagination, allowing scrolling next/prev, so putting another layer to ensure I'm getting the same number of records each time will require a bit of a rethink.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠