- Newest
- Most votes
- Most comments
When you Scan with the CLI pagination happens automatically as described here, however, when you introduce --limit
this prevents the CLI from auto-paginating resulting in the request returning the first 1MB worth of data, as described here.
This is further documented in the Scan CLI Docs:
A single Scan operation reads up to the maximum number of items set (if using the Limit parameter) or a maximum of 1 MB of data and then apply any filtering to the results using FilterExpression . If LastEvaluatedKey is present in the response, you need to paginate the result set. For more information, see Paginating the Results in the Amazon DynamoDB Developer Guide .
TLDR:
CLI auto-paginates Scan/Query requests only if you do not provide --limit
param, in which case it will return the first 1MB of data.
How much data was returned with that 4000 entries? If it exceeds 1 MB then the scan will return less than 1 MB and indicates the pagination with the LastEvaluatedKey
.
Quote from our documentation:
DynamoDB paginates the results from Scan operations. With pagination, the Scan results are divided into "pages" of data that are 1 MB in size (or less). An application can process the first page of results, then the second page, and so on.
A single Scan only returns a result set that fits within the 1 MB size limit.
Relevant content
- asked 2 years ago
- asked 5 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
Can you run the command with the
--debug
flag? Does it return all the entries when you remove the--limit
param?Why are you scanning for so many items? It is better to export the data to S3 and use Athena to query the data in a richer and more cost-effective way.