Skip to content

High CPU usage of one Redshift node (not leader). How understand what is causing this imbalance?

0

Hi there, I have a problem that I can't solve yet. One node has high CPU load almost all the time, but I can't find any significant skew in data storage. What could it be? Is it possible to track which queries are loading the node so much? Enter image description here

asked a year ago385 views
1 Answer
0

To understand and address the high CPU usage on a single non-leader Redshift node, you can take several steps to investigate and identify the root cause:

  1. Monitor cluster metrics: Check the CloudWatch metrics for your Redshift cluster, focusing on CPUUtilization, DatabaseConnections, and QueryPlanning. This can help you identify any unusual spikes or patterns in resource usage.

  2. Analyze query performance: Use the Amazon Redshift console to view information about queries and loads running in your database. This can help you identify long-running queries or those creating bottlenecks.

  3. Examine data distribution: Although you mentioned not finding significant skew in data storage, it's worth double-checking the data distribution across nodes. Run a query to identify tables with data skew or unsorted rows in your Redshift cluster. This can help pinpoint if certain tables are causing uneven workload distribution.

  4. Investigate longest-running queries: Use a diagnostic query to identify the longest-running queries in your cluster. This can help you pinpoint specific queries that might be causing the high CPU usage on the affected node.

  5. Compare query execution: For queries that seem to be problematic, compare their execution plans and metrics between faster and slower instances. This can help identify if the issue is related to a specific query or data distribution problem.

  6. Check for recent changes: Review any recent patches, updates, or version changes on the cluster that could have impacted performance. Also, look at any changes in WLM settings or Concurrency Scaling configurations.

  7. Analyze query metrics: Use system tables to gather detailed execution metrics for queries. This can help you understand which queries are consuming the most resources on the affected node.

By following these steps, you should be able to track down which queries are loading the node so heavily and identify potential causes for the imbalance. If the issue persists after your investigation and attempted optimizations, it may be worth contacting AWS Support for further assistance.
Sources
How do I troubleshoot high CPU usage in Amazon Redshift?

Monitoring Amazon Redshift cluster performance - Amazon Redshift
Redshift diagnostics queries to identify the performance bottleneck | AWS re:Post
Monitoring and troubleshooting query performance using system tables on Amazon Redshift provisioned clusters | AWS re:Post

answered a year ago
AWS
SUPPORT ENGINEER
reviewed a year ago
AWS
SUPPORT ENGINEER
revised a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.