DynamoDB race conditions with TransactWriteItems and Condition Check

Question

Hi, I wanted to check my understanding of what happens in the following scenario, and to find out if there's a way around it.

I have a bunch of global DynamoDB tables set up to store metadata about Documents and their Versions. When a new Document gets created I use TransactWriteItems to write both to the documents table and an ownership table (using PutItem). Each of the actions has a Condition Expression which fails if the item already exists in the table. Let's say the tables are available in regions A and B. Is the following series of events possible?

1. A user in region A creates a new document, which invokes the TransactWriteItems operation
2. Simultaneously, a user in region B creates a new document with the same ID
3. The PutItem to the documents table in region A succeeds, but for some reason there's a delay calling the PutItem to the ownership table
4. The PutItem to the documents table in region B succeeds
5. The PutItem to the ownership table in region B succeeds
6. DynamoDB replicates the information across from region B to region A
7. The PutItem to the ownership table in region B gets called, but because the item from region A has been replicated over, the Condition Check fails

What happens in this scenario? When step 6 happens, do we end up with a last-writer-wins resolution where the item in the documents table in region A gets overwritten? Does the DynamoDB instance in region A attempt to rollback the transaction (which presumably has been overwritten already)? What if the PutItem to the ownership table in region A happens after the PutItem to the ownership table in region B, but before the information gets replicated across regions? Is there a way to ensure we don't have a situation where the user sees the transaction succeed, but ends up with different data in the table from the one they submitted?

(I've looked at https://repost.aws/questions/QUP7oi53yZTnaQqOIFJAYboA/concurrent-dynamo-db-transact-write-items-requests-with-same-condition-check but their question is different and doesn't involve global tables)

Answer

When using global tables you shouldn't write to the same item in different regions at around the same time (within the replication window) unless you don't care the order in which the writes apply. One write is going to win, the other is going to lose. If both regions at around the same time increment an item's value by 1 the final value might be +1 or +2, depending on if there was a last writer wins conflict. So best not to try to do such things.

Furthermore, transactional writes are only ACID in the region where the transaction runs. The remote region will get the data replicated in once the transaction completes, but the items will replicate individually not as a unit. If you try to run a transaction on the remote region it may run against a partial view of the original transaction (half the items updated, half still replicating in) and will make all its conditional decisions against that torn transactional view.

Your example talks about using transactions but then talks about delays in the writes within one transaction. That's not how transactions work. You send the one request and either all work succeeds or all fails. Just remember that even if the work succeeds (and is ACID) the individual items replicate independently and the receiving end might will have a partial view as things replicate over.

Easiest approach is to route all writes for a given item (or all items) to one region at a time, so you're always working against the latest data.

DynamoDB race conditions with TransactWriteItems and Condition Check

Relevant content