AWS scan and query not working as intended

When you Scan with the CLI pagination happens automatically as described here, however, when you introduce --limit this prevents the CLI from auto-paginating resulting in the request returning the first 1MB worth of data, as described here.

This is further documented in the Scan CLI Docs:

A single Scan operation reads up to the maximum number of items set (if using the Limit parameter) or a maximum of 1 MB of data and then apply any filtering to the results using FilterExpression . If LastEvaluatedKey is present in the response, you need to paginate the result set. For more information, see Paginating the Results in the Amazon DynamoDB Developer Guide .

TLDR:

CLI auto-paginates Scan/Query requests only if you do not provide --limit param, in which case it will return the first 1MB of data.

How much data was returned with that 4000 entries? If it exceeds 1 MB then the scan will return less than 1 MB and indicates the pagination with the LastEvaluatedKey.

Quote from our documentation:

DynamoDB paginates the results from Scan operations. With pagination, the Scan results are divided into "pages" of data that are 1 MB in size (or less). An application can process the first page of results, then the second page, and so on.

A single Scan only returns a result set that fits within the 1 MB size limit.