How to optimize Lambda to reduce iterator age?
I have a Glue Job which inserts around 1.2m register to a DDB table. This is used with DynamicFrame and so the insert mechanic is very straightforward and lasts only a few minutes. My Glue job can "full refresh" the whole table to update all registers (inserting the 1.2m of data) but usually it is in "incremental" mode, only adding new register around hundreds each hour.
This data is then capture by the DDB Streams, which triggers a lambda function to get this data and then insert it into another cross-account DDB. When doing incremental mode, everything works fine and pretty fast. However, when I need to update the whole table, my lambda takes around 7 hours to process everything. See images below for the metrics:
We can see by the iterator age that my data has been sitting
My current lambda trigger has a batch size of 1 (since I dont have any control how many data gets to me every hour) and no batch window. My concurrency has: Function concurrency: Use unreserved account concurrency Unreserved account concurrency: 5000
My question is, how do I get down iterator is too high: its taking 7,30 hours to process the last data when I need to process the "full refresh". How do I get down this to only lasts a few minutes at max?
- Newest
- Most votes
- Most comments
There are a few things you can do:
- Increase the batch size. This will reduce your cost as well as your processing time. By setting a large batch size, Lambda does not wait until all the messages arrive (unless you also set a batch window). There are fewer invocations, fewer overhead, and further more, you can use batch operations when writing to the destination database, which will reduce your duration event further. You can also create threads in your function to handle the messages in the batch in a parallel manner.
- Set the Parallelization Factor to 10. This will give you a concurrency of 10 instead of 1 for each DDB partition. By doing this alone, you will probably see a 10x decrease in processing time.
Relevant content
- Accepted Answerasked 8 months ago
- asked 4 months ago
- asked 5 years ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated a year ago
- How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?AWS OFFICIALUpdated 2 years ago
Yes. You want to quickly move a pile of rocks, right, but you're putting just 1 rock in your wheelbarrow at a time and assigning just 1 wheelbarrow per partition. Load more rocks each time and have more wheelbarrows to take turns. You may be thinking a batch size of 100 means it sits there waiting until it sees the 100th arrive (like an airport shuttle bus) but no, it starts off after a certain short time even if the 100 isn't reached.