Execute Lambda function periodically for multiple inputs

Question

Hello, I hope you can guide me.

I have a Lambda function (let's call it lambda-1) that I need to run once a day. The function takes an identifier (ID) as input and, with that ID, extracts a set of data from a DynamoDB table to perform some analysis. Everything works fine when running it manually. As an additional note, the IDs are stored in another DynamoDB table.

However, this function must run for multiple IDs and once a day, which can make it complex in terms of resources or how to make this happen.

I understand that the normal approach would be to use EventBridge. The only difficulty is that the IDs keep growing (as devices are added to the application), so the idea is that the task that runs once a day can take all the existing IDs and run the Lambda (lambda-1) for each of them, which will perform the analysis.

I first thought about creating a Lambda (lambda-2) whose function is to take all the IDs from the DynamoDB table containing them and then execute the Lambda (lambda-1) that performs the analysis for each of the IDs. This Lambda (lambda-2) would be subject to periodic execution in EventBridge. However, the problem is that when a very large number of IDs is present, there may be an issue due to an execution time limit.

That said, I'm not sure what the correct solution architecture should be, and if what I'm proposing is feasible.

Thanks!

Jesus Bernal · Accepted Answer

I would recommend going with Step Functions as an orchestrator. You could create a workflow where you can have a initial task that gets all the IDs using a Lambda function (same as one of the steps you planned for lambda-2) from the DynamoDB table and then, iterate through them using a [Map state](https://docs.aws.amazon.com/step-functions/latest/dg/state-map.html). Each iteration will process an ID using lambda-1.

![Step Functions workflow sample](/media/postImages/original/IMlM7R75jHRb-O-I4NzadhEQ)

Some additional notes:
- Now the orchestration is moved away from Lambda and handled by a Step Functions workflow. This way 1) you don't assume cost from a Lambda function that is most of the time waiting for other functions to do their job 2) you move away from the execution time out
- The Step Function workflow can be triggered daily using EventBridge as it is a supported target.
- You can - and probably should - leverage the [optimized integration with DynamoDB](https://docs.aws.amazon.com/step-functions/latest/dg/connect-ddb.html) to get all the IDs. This will reduce execution time and save cost.
- If all IDs don't fit in a single DynamoDB operation you will have to add pagination.

re:Post Agent · Answer

Your approach to solving this problem is on the right track, but we can refine it to create a more scalable and efficient solution. Here's a suggested architecture that addresses your requirements:

1. Create an EventBridge rule that triggers once a day.

2. This rule should invoke a new Lambda function (let's call it lambda-orchestrator) that will manage the process.

3. The lambda-orchestrator function will:
   a. Retrieve all the IDs from the DynamoDB table.
   b. For each ID, invoke lambda-1 asynchronously.

4. Lambda-1 remains unchanged, performing the analysis for a single ID.

This architecture has several advantages:

1. Scalability: By invoking lambda-1 asynchronously for each ID, you avoid the execution time limit issue. Each invocation of lambda-1 runs independently, allowing for parallel processing of multiple IDs.

2. Efficiency: The lambda-orchestrator function's job is relatively quick - it just needs to fetch the IDs and trigger the other Lambda functions. This reduces the risk of hitting execution time limits.

3. Flexibility: As new IDs are added to the DynamoDB table, they'll automatically be included in the next day's processing without any changes to your code.

4. Cost-effective: You only pay for the actual execution time of each Lambda function.

To implement this:

1. Create the EventBridge rule to run daily.

2. Create the lambda-orchestrator function. This function should:
   - Scan the DynamoDB table to get all IDs.
   - Use the AWS SDK to invoke lambda-1 asynchronously for each ID.

3. Ensure that lambda-1 is set up to receive the ID as an input parameter.

4. Grant necessary permissions:
   - Allow EventBridge to invoke lambda-orchestrator.
   - Allow lambda-orchestrator to read from DynamoDB and invoke lambda-1.
   - Ensure lambda-1 has permissions to read from DynamoDB and perform its analysis tasks.

This solution should be able to handle a growing number of IDs efficiently, running the analysis for each ID once a day as required.  
**Sources**  
[Tutorial: Create an EventBridge scheduled rule for AWS Lambda functions - Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-run-lambda-schedule.html)  
[Use scheduled events to invoke a Lambda function - AWS SDK Code Examples](https://docs.aws.amazon.com/code-library/latest/ug/dynamodb_example_cross_LambdaScheduledEvents_section.html)  
[Understanding basic Lambda concepts - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/concepts-basics.html)

Execute Lambda function periodically for multiple inputs

Relevant content