Dynamodb : Need sorted + filtered + paginated data

0

Hi All,

Need some help around dynamodb.

We have an app which process the a batch of 44K records and saves the data to DDB table. This has

  1. PK (run_id_random_number)
  2. SK (run_id)

We want to fetch the filtred data page by page and in a sorted way .

Lets say there 20 attibutes pre row and we have to filter the data by region,country,status for example region='Europe' & country='Luxembourg' the total number of records come down to 20K, and have to sort on status attiribte.

How can I get records in a paginated manner(page size is 500) ?

Thanks, Ashish

Ashish
asked a year ago1065 views
1 Answer
0
Accepted Answer

You must use Scan with a FilterExpression to achieve your use case.

You can set the Limit to 500, but keep in mind that the Filter is applied after the limit is reached, so you are only guaranteed ~500 items, not exactly 500.

A single Scan operation reads up to the maximum number of items set (if using the Limit parameter) or a maximum of 1 MB of data and then apply any filtering to the results using FilterExpression. If LastEvaluatedKey is present in the response, you need to paginate the result set. For more information, see Paginating the Results in the Amazon DynamoDB Developer Guide.

this will filter the data , I need the data to be ordred by one of the columns In this case , I need the data to be sorted by 'status' attribute ,, status attribute can have values (PASS,FAILED,EVALUATING,DEGRADING,<>)

If you need all your data ordered by status, then you would have to create a GSI with a static partition key and status as sort key.

GSI_PKGSI_SKData
1DEGRADINGData
1PASSData
1PASSData

This would ensure you can get the data returned sorted by the sort key as DynamoDB does not provide sorting across partition keys. Keep in mind this solution would limit your writes to 1000 WCU per second. If you require more you can shard the partition key.

profile pictureAWS
EXPERT
answered a year ago
  • this will filter the data , I need the data to be ordred by one of the columns In this case , I need the data to be sorted by 'status' attribute ,, status attribute can have values (PASS,FAILED,EVALUATING,DEGRADING,<>)

  • Updated my answer.

  • Thank you , it there a possibility to sort by multiple attibutes

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions