Why does my Amazon Redshift query keep exceeding the WLM timeout that I set?

Short description

A WLM timeout applies to queries only during the query running phase. If WLM doesn’t terminate a query when expected, it’s usually because the query spent time in stages other than the execution stage. For example, the query might wait to be parsed or rewritten, wait on a lock, wait for a spot in the WLM queue, hit the return stage, or hop to another queue.

Resolution

When querying STV_RECENTS, starttime is the time the query entered the cluster, not the time that the query begins to run. When the query is in the Running state in STV_RECENTS, it is live in the system. However, the query doesn't use compute node resources until it enters STV_INFLIGHT status. For more information about query planning, see Query planning and execution workflow.

To view the status of a running query, query STV_INFLIGHT instead of STV_RECENTS:

select * from STV_INFLIGHT where query = your_query_id;

Use this query for more information about query stages:

select * from SVL_QUERY_REPORT where query = your_query_id ORDER BY segment, step, slice;

Use the STV_EXEC_STATE table for the current state of any queries that are actively running on compute nodes:

select * from STV_EXEC_STATE where query = your_query_id ORDER BY segment, step, slice;

Here are some common reasons why a query might appear to run longer than the WLM timeout period:

The query is in the "return" phase

There are two "return" steps. Check STV_EXEC_STATE to see if the query has entered one of these return phases:

• The return to the leader node from the compute nodes
• The return to the client from the leader node

A rollback is in progress

If a data manipulation language (DML) operation encounters an error and rolls back, the operation doesn't appear to be stopped because it is already in the process of rolling back. You can view rollbacks by querying STV_EXEC_STATE. You can find additional information in STL_UNDONE.

The query spends time queuing prior to running

Query STV_WLM_QUERY_STATE to see queuing time:

select * from STV_WLM_QUERY_STATE where query = your_query_id;

The query is waiting on a lock

If the query is visible in STV_RECENTS, but not in STV_WLM_QUERY_STATE, the query might be waiting on a lock and hasn't entered the queue. For more information about checking for locks, see How do I detect and release locks in Amazon Redshift?

A query hopped to another queue

If a read query reaches the timeout limit for its current WLM queue, or if there's a query monitoring rule that specifies a hop action, then the query is pushed to the next WLM queue. To confirm whether the query hopped to the next queue:

• While the query is running: query STV_WLM_QUERY_STATE
• After the query runs: query STL_WLM_QUERY

To prevent queries from hopping to another queue, configure the WLM queue or WLM query monitoring rules. For more information about query hopping, see WLM query queue hopping.

A networking or firewall issue

If an Amazon Redshift server has a problem communicating with your client, then the server might get stuck in the "return to client" state. Check for conflicts with networking components, such as inbound on-premises firewall settings, outbound security group rules, or outbound network access control list (network ACL) rules. For more information, see Connecting from outside of Amazon EC2 —firewall timeout issue.

An issue with the cluster

Issues on the cluster itself, such as hardware issues, might cause the query to freeze. When this happens, the cluster is in "hardware-failure" status. To recover a single-node cluster, restore a snapshot. In multi-node clusters, failed nodes are automatically replaced.

---

Related information

Tuning query performance

Analyzing and improving queries