I am deploying a CloudFormation stack with a number of Lambda-backed custom resources which load data into AWS. These Lambda functions take approximately 7 minutes to execute, and a number of them are run in parallel by having multiple custom
resources backed by the same Lambda function. The total time from the first CREATE_IN_PROGRESS event for these resources approximately matches the time it takes for the Lambda function to run, and they run successfully.
There are further custom resources (using different Lambda functions) that run after the previously custom resources are complete. However, we are seeing unusual behaviour for these custom resources, as there is a significant wait from the first CREATE_IN_PROGRESS event to the Lambda function actually being invoked (as seen in the logs for the Lambda function). This was has been from 10 minutes to over an hour, sometimes causing the custom resource to time out without the Lambda function ever being invoked. It appears that the events from CloudFormation which normally trigger the custom resource Lambda are being significantly delayed.
Why might this be?
This looks to have been the problem. Thanks for your help.