[Merge Fellas Unlimited Shake] - Cost spike & throttling on Lambda. What's the right pattern?

When dealing with Lambda throttling and cost issues for heavy data processing workloads like your "Merge Fellas Unlimited Shake" pattern, there are several approaches to consider.

For the throttling issue, this is occurring because your Lambda function is hitting concurrency limits during peak hours. When Lambda functions scale up rapidly to handle incoming requests but reach your account's concurrency limit, throttling occurs. CloudWatch metrics would show the maximum concurrency being reached followed by throttling events.

Here are recommendations to address both issues:

1. Implement an event-driven architecture with SQS:

• Create a pattern using S3 → SQS → Lambda or directly use SQS as a buffer before your Lambda function
• This decouples event generation from processing, allowing better handling of traffic spikes
• SQS acts as a buffer, smoothing out traffic and preventing Lambda from being overwhelmed
• Use a Standard SQS queue (not FIFO) for higher throughput when exact ordering isn't critical
• Configure batch processing in your Lambda to process multiple messages in a single invocation

2. Consider Provisioned Concurrency:

• Provisioned concurrency keeps pre-initialized execution environments ready to respond
• This helps eliminate cold starts and provides more predictable performance
• It's particularly useful for functions that need to respond quickly during traffic spikes
• Be aware that this incurs additional charges based on the amount of concurrency you configure and the time period it's configured for

3. Implement throttle tolerance strategies:

• Use timeouts, retries, and backoff with jitter to smooth out retried invocations
• This helps ensure Lambda can scale up within seconds to minimize throttling

4. Optimize your function:

• Review your function's memory configuration and execution time
• Performance testing can help determine the optimum memory size
• Consider breaking down the "unlimited shake" process into smaller, more manageable chunks

For your specific options:

• Provisioned Concurrency would help with throttling but may increase costs if not properly sized
• Moving to Fargate might be appropriate if the workload is consistently high and long-running
• Adding an SQS queue is likely the most flexible solution as it helps manage traffic spikes while allowing you to optimize Lambda execution

The best approach may be a combination: implement the SQS buffer to manage traffic spikes, optimize your Lambda function's performance, and consider using provisioned concurrency during known peak periods if predictable response times are critical.
Sources
Best practices for working with AWS Lambda functions - AWS Lambda
Serverless Computing – AWS Lambda Pricing – Amazon Web Services
Troubleshoot event source mapping issues in Lambda - AWS Lambda
Preventing Maximum Lambda Invocation Errors and Managing Large Traffic Spikes | AWS re:Post