Maintaining a single running total serverless

Question

Sorry if it's been asked,. Seems like a common issue and I did some searches but don't know what terms to use.

My set up:
I have customers (in cognito) and each customer needs a value associated with their account. It's basically an Account Balance (but it's not a currency, it's hours.)
I haven't decided where to store the canonical version of the number yet so that's open, I'm leaning towards S3.
I can generate events (probably will use an SQS queue) that can send timestamps, or deltas calculated off the timestamps.
I can't guarantee these events will be generated in order and they will be asynchronous.

My problem:
If I consume the events with a lambda, and more than one fires at the same time for the customer, setValue(getValue() - delta) there can be a race condition between get and set there.

I'm leaning towards using a FIFO (do I need a FIFO queue here?) and set max concurrency one for the lambda, probably set up batching too (possibly one queue per customer with one write at the end?) If this is the best option, is there a way to do it with one queue?

S3 Locks could also work, would I just check for a lock and return it to the Queue, sounds like that's the same as max concurrent 1? And Dynamo is on the table too.

Thanks

Didier Durand · Answer

Hi,

This article details common solutions to the race condition problem that you raise: https://medium.com/@moradiyabhavik/race-condition-understanding-and-solution-926b6d2808c

This one may also be interesting for you: https://cdg.dev/tech/2020/03/avoiding-race-conditions-in-concurrent-aws-lambda-functions/

Best,

Didier

sash · Answer

Hi Kelnor,

While leveraging SQS FIFO, you order guarantee is dependent on the the order in which the messages are published to the queue.  
In your case you have mentioned  "I can't guarantee these events will be generated in order and they will be asynchronous."  so you need to implement a mechanism to discard/process when they arrive out of order in the consumer before persisting.

The sendMessages section in the following document captures this information https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html

Also the messages in FIFO are ordered based on message group ID. You could for instance have the message ID as customerId, then all the messages for a given customerId is grouped and ordered within the group is preserved based on their arrival.
With Lambda as a consumer, the order within the group is preserved - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/fifo-queue-lambda-behavior.html

regards, 
sash

Maintaining a single running total serverless

Add your answer

Relevant content