DynamoDB - scanning a 7.5GB table

In general, we recommend the customer perform parallel Scan with multiple threads. The following AWS documentation provides a detailed explanation on how Parallel Scan works:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan

With the Scan API call, each Scan returns 1 MB data. A single thread can perform around 20 Scan API calls in a second, assuming the round-trip latency for each API call is around 50 ms. In other words, you can Scan around 20 MB data per second per thread.

On an Amazon EC2 instance with 16 vCPU cores, run one thread per vCPU core. You will achieve something like 300 MB/s in throughput. With this first approximate, you can get things done in 7500 / 300 = 25 seconds, which is less than 1 minute.

If you want to load data from Amazon DynamoDB into memory, then you will need a data structure that can be shared by multiple threads and is also thread-safe. In Java you will need to think about things like Concurrent Collections.

If you want to load data from Amazon DynamoDB onto disk (export), you might want to refer to the DDBImportExport project on github, which I developed as a demo on how to use perform parallel Scan in DynamoDB. This implementation uses Python, but the same logic can be easily re-implemented in any other languages.