How to paginate Redshift Data API statement results using Boto3?

Hi good morning,

Thanks for sending your questions regarding pagination and Redshift Data API using boto3. The boto3 for Redshift Data API supports the following methods: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/redshift-data.html

The method get_paginator () can be used to paginate the results and specifically to the pagination of the method get_statement_result, you can set only MaxItems per page in the configuration (PageSize is not supported).

I've reproduced the scenario using Redshift sample data available in database sample_data_dev and I'm getting the same scenario when using MaxItems.

To help you with some alternatives you can do the following:

• Execute your statement using the method execute_statement (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/redshift-data/client/execute_statement.html)

• Get the results of your statement using the method get_statement_result (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/redshift-data/client/get_statement_result.html). This method will return a NextToken created automatically by Redshift Data API depending on result to be retrieved, volume, etc. Redshift Data API automatically paginate when returning the data using this method in some cases, but it's not customizable. As mentioned earlier, Redshift Data API will split in parts automatically considering some variables. So, some results can be paginated or not.

Regarding the method get_paginator and the iterator using the paginator for Redshift Data API, I will analyze and revert back as soon as I have more info. Meanwhile, I hope the alternatives I've provided helps to achieve the goals of your development.

Hi good morning,

Regarding the method get_statement_result (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/redshift-data/client/get_statement_result.html)), the Redshift Data API will automatically split the results and return a NextToken automatically based on system internal conditions (query result size, etc). In this API, you can't configure a limit of rows per page (e.g. MaxItems or PageSize). However, as mentioned, the system will split the result automatically based on internal conditions.