How should I size Amazon Redshift for my project?

Background

Amazon Redshift launched in 2013 with a Redshift cluster. A Redshift cluster is made of multiple nodes which makes it an MPP or massively parallel processing machine. The latest node family is RA3 with four node types ra3.large being the smallest , then ra3.xlplus , then ra3.4xlarge which provides 3x resources, then ra3.16xlarge which provides further 4x resources. You must choose which node type and also choose the number of nodes when you create your Redshift cluster. Amazon Redshift console provides you a Help me choose widget which can recommend a cluster configuration based on the total data volume and amount of data you expect to typically query. However, this sizing is not an easy decision to make. This widget is great for a start but it does not take into account the number of queries being executed concurrently or query complexity, which are often the most influential factors for performance. So you might need to go 2x or 4x or x/2 or x/4 from this initial sizing recommendation if you go down this route. Better option is to leverage AI-driven scaling ... read on!

Fast-foward

Amazon Redshift launched Serverless offering in 2022, with default autoscaling enabled to handle high concurrency workloads. Here you are not choosing the node type anymore, nor are you choosing the number of nodes. Instead you are just providing the number of RPU or Redshift Processing Units. Each RPU provides you 16 GB of memory and you estimate how many RPU's you might need. While this simplified the decision making to a certain extent, you still needed to pick a number for RPU. Picking a low RPU leads to not enough memory, and queries spill to disk. Disk based queries are typically 5x to 10x slower than queries that complete in-memory.
Let's take example for 32 RPU and the query is running for 60 minutes. At 128 RPU, the query finishes in 15 mins.
Let's calculate your costs for this query

RPU	Time (mins)	RPU Rate	Calculation	Cost
32	60	$0.36/RPU-Hr	32 x 60 x 0.36 / 60	$11.52
128	15	$0.36/RPU-Hr	128 x 15 x 0.36 / 60	$11.52

The query cost stayed the same $11.52 , but with a higher RPU you were able to get the query executed 4x faster , and data delivered much sooner!

Enter AI-driven scaling

Amazon Redshift Serverless came with AI-driven scaling and optimizations in 2023. Now you do not even need to pick a number for RPU. You simply provide a price-performance target, from 50/Balanced (recommended) to 100/Performance focus to 1/Cost focus. With your price-performance target specification Amazon Redshift will automatically leverage compute as required by the queries you execute. So when a query that is 5x your typical queries comes in, the AI-driven scaling will determine that you need a larger compute for this heavy task, provision the larger compute, execute the query on this secondary compute, and release the larger compute. All this while your main compute was processing your regular queries. So AI-driven scaling is able to accommodate larger query easily and without impacting the performance of your regular queries. This powerful feature makes Amazon Redshift Serverless with AI-driven scaling the de facto choice for any and every project you take up.

Scenarios

Q1) I am starting a brand new project, how should I size Redshift?

A) With a brand new project you are most likely not going to be running queries 24x7. This makes Serverless a perfect fit for you. Start with the minimum of Base 8 RPU (128 GB memory). If you experience that queries are too slow then you can bump to Base 16 RPU (256 GB memory) or even more Base 24 RPU (384 GB memory). This range 8 RPU to 24 RPU is the explorative range and should be sufficient as you are merely exploring the project.

Q2) I am done exploring and my new project has progressed, how should I size Redshift?

A) Now that your project is taking good shape, you might see that the explorative RPU range is not able to keep up with the variety of queries or the number of queries being executed. Now you should change to Base 32 RPU (512 GB memory), and after 30+ minutes change to AI-driven scaling with 50/Balanced setting. Why? Because, as of this writing, it is recommended to enable AI-driven scaling when the Base RPU is at minimum 32 RPU and between 512 RPU. In most cases 32 RPU should provide you enough resources to run queries scanning up to 100 GB data, 64 RPU for up to 250 GB data scans, 128 RPU for up to 500 GB data scans and so on. Setting an apt Base RPU before turning on AI scaling will help, as AI scaling will use that RPU as a starting point. If you end up setting a Base RPU too high then AI scaling will automatically reduce RPU too, so you are covered on both fronts. Kindly note that you could directly start with AI-driven scaling from the get-go also. You will see that Amazon Redshift uses the default RPU setting of 128 RPU and can change it as it learns your workload patterns.

Q3) These are too many increments from 8 RPU to 32 RPU to AI-driven scaling. How can I get to Redshift AI-driven scaling faster?

A) Start directly with AI-driven scaling at default 50/Performance setting. If this meets your price-performance needs then great else go higher to 75/Performance if your queries are too slow. Or go down to 25/Cost if cost is higher than your expectation. If you are switching existing workgroup from Base RPU to price-performance targets then take into account additional considerations. Firstly, the workgroup needs to be between 32 RPU and 512 RPU Base. Secondly, when switching

• from Base 128 RPU or higher, then start with price-performance slider at 50/Balanced
• from Base lower than 128 RPU, then price-performance slider at 75/Performance
• from Base between 64 and 32 RPU, then use price-performance slider at 100/Performance

Q4) But how do I know how much data my queries are scanning?

A) Use the queries and scans SQL provided here. This SQL will query SYS_QUERY_* tables and fetch for you for number of queries executed and data scanned by queries. Run it with a Superuser account as Superusers can see all rows; regular users can see only their own data.

Q5) I already have an Amazon Redshift cluster and I am taking on a new project, how should I size Redshift?

A) Your existing Amazon Redshift cluster has been sized to the projects that it is already executing. You can try to estimate how many more nodes to add for the new project and perform a resize. But this sizing is actually the same challenging part what we have been discussing above. So better to provision a new Workgroup for this new project and follow Q1 and Q2 above.

Q6) But my new project needs data from my existing Amazon Redshift cluster. How should I proceed?

A) Amazon Redshift fully supports data sharing, so you can create a data share from your existing Amazon Redshift cluster, add the tables / schemas / databases to data share, and be able to access the shared data from your new Workgroup.

Q7) But my new project needs to load data to the existing Amazon Redshift cluster. How should I proceed?

A) Amazon Redshift data sharing also supports multi-data warehouse writes through data sharing, so you can provision the new Workgroup for ingesting the new data and leverage data sharing writes to write the data to you central Amazon Redshift cluster.

Q8) But serverless will get expensive and my costs could go out of control?

A) All the protective features from Amazon Redshift cluster are available on Workgroup. You can setup the same Query Monitoring Rules to protect your Amazon Redshift from bad queries. Also you can setup RPU usage limits on your Workgroup so that costs will never exceed the quotas you have established.

Q9) But Reserved Instance (RI) pricing offers me discounts over Serverless?

A) Lets compare the pricing for equivalently sized cluster to a workgroup. Since 8 RPU is minimum let's use that for comparison. 8 RPU provides you 8 x 16 GB = 128 GB memory. Each node of ra3.xlplus provides 32 GB memory, so that gives us 4-node cluster. Lets compare price for one month in US East (Ohio) for 24 x 7 utilization

Type	Memory	Billing	Cost	Vs. Serverless
8 RPU Serverless	128 GB	On-demand	$2,108.16	Baseline
ra3.xlp.4n Cluster	128 GB	On-demand	$3,171.12	50% expensive
ra3.xlp.4n Cluster	128 GB	1 Yr No Upfront	$2,219.78	5% expensive
ra3.xlp.4n Cluster	128 GB	3 Yr. No Upfront	$1,379.70	35% cheaper

So you see RI is cheaper but only if you go for the 3 Year term. That is a long commitment to make and technology is rapidly evolving every year. And secondly, with Serverless you are only paying when queries execute. So if your utilization is 18 hours instead of 24 hours, then your Serverless cost automatically goes down by 25%. Where as with RI you are paying a fixed price even if you do not use the cluster.

Update 22-Apr-2025 , Redshift introduces Serverless Reservations which are managed at the AWS payer account level and can be shared between multiple AWS accounts. With Serverless Reservations, you can commit to a specific number of RPUs for a 1-year term, and choose between two payment options: a no-upfront option that provides a 20% discount off on-demand rates, or an all-upfront option that provides a 24% discount.

Q10) So then when should I ever consider a provisioned cluster?

A) Provisioned cluster with 3 Yr RI's is a good fit for very repeatable and static workloads, where you are not anticipating much variability in terms of data volumes or processing power needs. For example, you are receiving IOT data streaming continuously. This cluster is going to be running 24 x 7, processing more or less the same number of messages, parsing the JSON payload into staging tables, and executing transformations to populate final target tables.

Q11) With provisioned cluster I know my costs will not change. But will serverless autoscaling my costs become variable?

A) Firstly "provisioned cluster I know my costs will not change" is not a fully true statement. If you enable concurrency scaling then your costs will fluctuate depending on usage (just like Serverless). If you have turned off concurrency scaling then Serverless also provides a Max RPU setting. So in theory you could set Base RPU and Max RPU at the same value so there is no more autoscaling (like concurrency scaling turned off). However, note that this defies the example provided above where 128 RPU provided better performance than 32 RPU, and at same price point.

Q12) I want to use Serverless but I am already under Provisioned RI's, how do I proceed from here?

A) If your cluster is meeting all its expectations and there are no issues for performance, then you can pick up Serverless when you take on the next project as covered under Q3, Q4 and Q5. But if you are facing some performance challenges on your cluster and you need to reevaluate the sizing then you can spin-off one of the existing workload into Serverless. This will release resources and help stabilize your existing cluster, while the new workgroup can service the spun-off workload. You can keep isolating more workloads this way until your main cluster ends up being right sized.

Q13) If I create new workgroup every time then how do I manage them?

A) You can Connect Redshift with AWS IAM Identity Center for a single sign-on experience. This integration provides the following benefits

1. A central location for your workforce users in AWS. You can create users and groups directly in AWS IAM Identity Center or connect existing users and groups that you manage in a standards-based identity provider like Okta, PingOne, or Microsoft Entra ID (Azure AD). AWS IAM Identity Center directs authentication to your chosen source of truth for users and groups, and it maintains a directory of users and groups for access by Redshift.
2. You can share one AWS IAM Identity Center instance with multiple Redshift clusters and workgroups with a simple auto-discovery and connect capability. This makes it fast to add new Redshift instances without the extra effort of configuring the AWS IAM Identity Center connection for each, and it ensures that all clusters and workgroups have a consistent view of users, their attributes, and groups.
3. Because user identities are known and logged along with data access, it's easier for you to meet compliance regulations through auditing user access in AWS CloudTrail.

Q14) I have more questions and I need to discuss further with someone?

A) You can reach out to us and reference this article

1. Your AWS Account team
2. AWS Developers slack channel #amazon-redshift
3. Email us at redshift-specialists-amer@amazon.com