is there a way to control Redshift external function parallelism?

Question

I have an external function in redshift that invocates a lambda function. The lambda functions hits another service that has concurrency limits, therefore I want to be able to limit the number of concurrent lambda functions. There doesn't seem to be any documentation on how Redshift decides how many lambda functions to invocate and how many records to submit to each invocation.

I see that you can set the max records/payload, but I have seen that the external function will create many invocations with less than that max number.

Is there any way to set the max number of concurrent lambda invocations for a redshift external function? If not, is there consistent logic for the number of invocations created given the number of input records?

Answer

Hello there, the concurrent lambda invocations depend upon the cluster configuration that you have in place (specifically the number of slices on the cluster). Slice is a unit of parallel computing in Redshift and hence when the query is submitted, each slice will work in parallel to make concurrent calls to the lambda.

The below documentation’s provides more details

https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_EXTERNAL_FUNCTION.html#r_CREATE_FUNCTION-usage-notes

https://aws.amazon.com/blogs/big-data/accessing-external-components-using-amazon-redshift-lambda-udfs/

There is no publicly documented configuration parameter that can set the max number of concurrent lambda invocations. I would recommend you to create a support case with Redshift service with your cluster details and Redshift team can possibly look at tweaking any internal configurations that can help with this scenario.

Answer

Hi there,

Thank you for posting the question and the detailed explanation. To answer your question, a screen-share session would be needed and will require details that are non-public information. Can you please open a support case with AWS using the following [link](https://support.console.aws.amazon.com/support/home#/case/create) and we will be happy to help you.

Thank you!

is there a way to control Redshift external function parallelism?

Relevant content