How is data returned from Spectrum to Redshift cluster?

0

Seeking clarification on how the results of a Spectrum query are returned to the main Redshift cluster. Presumably this would impact the sizing of the nodes in the Redshift cluster to ensure they have sufficient capacity to process the results. Namely, are all results returned to the leader node, or is there some logic that maps data into the appropriate slice? If the latter, how does the association of the data to the slice work?

AWS
Jay_M
已提問 4 年前檢視次數 436 次
1 個回答
0
已接受的答案

More detailed info is in my blog: https://aws.amazon.com/blogs/big-data/10-best-practices-for-amazon-redshift-spectrum/. Spectrum layer will do what it needs to do (push-down operations). Hopefully it will "filter" out majority of the rows off S3 before sending a small portion back to the main Redshift cluster for further processing (such as joins or DISTINCT). No, a tiny 2 x dc2.large cluster would not be able to handle 1M of 1GB Parquet files in S3 and do joins on these large external tables. Each slice of the main Redshift cluster can invoke up to a max of 10 Spectrum nodes per query. Data post-Spectrum filtering will be sent to Redshift slices depending on the next step in the execution pipeline (as generated by Redshift Optimizer) and hashing values of the join/GBY columns, etc. This is not much different from performing joins between a regular Redshift table that is using DISTSTYLE EVEN and another Redshift table that uses DISTKEY distribution.

專家
已回答 4 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南