Redshift ra3.large type provides only 1TB storage even though in the docs it says it gives 8TB per node

You're right that the RA3 instances in Amazon Redshift come with managed storage, and for ra3.large, it's indeed 8TB per node. Here's an explanation of what's happening and how to address your issue:

Local vs. Managed Storage:

The 1TB you're seeing is likely the local SSD storage on the node. The remaining 7TB is part of the managed storage, which is automatically handled by Redshift. You don't need to do anything special to access the full 8TB. Redshift manages data placement automatically between local storage and S3. Your queries will work seamlessly across all 8TB of data.

Verifying Storage Capacity: Execute the following query to check your total storage capacity:

SELECT SUM(capacity) AS total_capacity_gb
FROM stv_attached_volume;

This should show you the full 8TB (8192 GB) capacity

There appears to be a misunderstanding about the storage capacity of RA3 nodes in Amazon Redshift. RA3 nodes, including ra3.large, do not have a fixed storage limit of 1TB per node. Instead, they use a concept called managed storage, which allows for virtually unlimited storage capacity.

The 8TB figure you mentioned likely refers to the maximum amount of data that can be stored locally on the high-performance SSDs of the RA3 node. However, this is not a hard limit on your total storage capacity.

With RA3 nodes, Amazon Redshift uses a tiered storage architecture:

1. Frequently accessed "hot" data is stored on fast local SSDs.
2. Less frequently accessed "cold" data is automatically offloaded to Amazon S3.

This system, called Redshift Managed Storage (RMS), allows you to store and analyze much more data than what fits on the local SSDs. You pay the same rate for RMS regardless of whether the data is on SSDs or in S3.

If your COPY jobs are getting stuck after 1TB, it's likely due to a configuration issue or a misunderstanding of how the storage works. The cluster should continue to accept data beyond 1TB, automatically managing the tiered storage in the background.

To resolve your issue and utilize the full capacity of managed storage:

1. Ensure your cluster is properly configured to use managed storage.
2. Check if there are any error messages or logs that might explain why the COPY jobs are getting stuck.
3. Verify that you have the necessary permissions to write data to the cluster.
4. Monitor your cluster's storage usage through the AWS console or CloudWatch metrics.

If the problem persists, it would be advisable to contact AWS support for further assistance, as they can provide specific guidance based on your cluster's configuration and usage patterns.
Sources
Cloud Data Warehouse – Amazon Redshift Pricing– AWS
Amazon Redshift RA3 - Amazon Web Services
Changing cluster from dc2.large 4 nodes to ra3.xlplus 1 node | AWS re:Post