- Newest
- Most votes
- Most comments
One option here is to decouple the database read from the API query. You can do this by using SQS: As you read each entry from the database, send it to SQS and then trigger Lambda functions from the SQS queue. That way, each Lambda function has a separate runtime and doesn't have a shared fate with any of the other queries. You probably want to limit the concurrency there so that you're not overloading the target API with calls. This method will also let you scale well beyond 100 URLs.
Please take a look at this workshop - https://async-messaging.workshop.aws/scatter-gather.html
It appears very similar to what you are trying to achieve - 1 front-end call branching off to multiple backend call and responses of the backend calls getting written to DynamoDB. Front-end waits for all responses or a finite amount of time, before responding back to the client.
I would go for an approach with a StepFunction.
Step1 - call lambda that returns urls to process
Step2 - parallel iterator (you can configure parallellism)
Step2.1 - call PageSpeed for 1 url + store in DDB (you can configure error handling, and gather a result)
The result of the StepFunction might be a analysis report of the whole run
Relevant content
- asked 10 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago