I have a dataset with seven records:
I want to return all records between 14 and 20 Dec, paging them two at a time. This is my first call, for the first page of results:
Dictionary<string, AttributeValue> expressionAttributeValues = new Dictionary<string, AttributeValue>();
Dictionary<string, string> expressionAttributeNames = new Dictionary<string, string>();
string filterExpression = string.Empty;
AmazonDynamoDBClient client = new Amazon.DynamoDBv2.AmazonDynamoDBClient(creds);
List<RecordingMetaData> recordings = new List<RecordingMetaData>();
expressionAttributeValues.Add(":startTime", new AttributeValue { S = $"2022-12-14" });
expressionAttributeValues.Add(":endTime", new AttributeValue { S = $"2022-12-20" });
filterExpression = $"#start > :startTime AND #start <= :endTime";
expressionAttributeNames.Add("#start", "StartTime");
ScanRequest scanRequest = new ScanRequest()
{
TableName = "SCRecordingMetaData",
ExpressionAttributeNames = expressionAttributeNames,
FilterExpression = filterExpression.Length > 0 ? filterExpression : null,
ExpressionAttributeValues = expressionAttributeValues,
Limit = 2
};
var response = new ScanResponse();
do
{
response = client.Scan(scanRequest);
response.Items.ForEach(i => recordings.Add(new RecordingMetaData()
{
RecordingID = i["RecordingID"].S,
Duration = Convert.ToInt32(i["Duration"].N),
StartTime = Convert.ToDateTime(i["StartTime"].S),
EndTime = Convert.ToDateTime(i["EndTime"].S),
ANI = i["ANI"].S,
DNS = i["DNS"].S,
CreatedTime = Convert.ToDateTime(i["CreatedTime"].S),
FilePath = i["FilePath"].S,
AgentIDs = i.ContainsKey("AgentIDs") ? i["AgentIDs"].L : null,
QueueIDs = i.ContainsKey("QueueIDs") ? i["QueueIDs"].L : null
}));
}
while (response.LastEvaluatedKey.Count > 1);
it should, and does, return 3 records when there's no Limit set in the request. When it's set to 2, I would expect 2 results to be returned for the first page, with the LastEvaluatedKey being "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad" since this is the second key in the list that meets the filter criteria. However, only one result is returned, "2a45d769-fc43-4325-96ec-76e5cf0750fe" and this is also the LastEvaluatedKey. If I pass this value to the ExclusiveStartKey of the next call I get 0 results returned, but the LastEvaluatedKey is "8bce9df3-3921-4259-a70a-c9cf6d34f61f".
So, what seems to be happening is that it only returns records that meet the filter criteria, but it is still in some way recognising that the other records that haven't been returned still exist, i.e. the first record set with no search filter should have "8c8c7afb-e529-4cb5-adb2-ab79d51f5741" and "2a45d769-fc43-4325-96ec-76e5cf0750fe", but because of the filter it only returns one record "2a45d769-fc43-4325-96ec-76e5cf0750fe". Surely it should return "2a45d769-fc43-4325-96ec-76e5cf0750fe" and "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad".
How can you write any kind of paging with behaviour like this?
Thanks for your response. I was worried that was going to be the case. I'm using the .Net SDK, so Boto3 is not an option. I have already implemented pagination, allowing scrolling next/prev, so putting another layer to ensure I'm getting the same number of records each time will require a bit of a rethink.