DynamoDB consistent Query misses an Item when an ongoing transaction has replaced it

0

Hello all,

Let's assume we have a set of Items like:

PKSK
11
12
13

We're making a low level Query like PK = 1 with consistent reads. This query responds with 3 items, as the table above. All good so far.

But say if there is an going transaction that does:

Operation
DELETE PK 1 - SK 3
ADD PK 1 - SK 4

Very rarely the Query returns:

PKSK
11
12

Meaning, we're completely missing SK 3 and 4. My expectation is that we would have either SK 3 or 4 in the return list, because transactions are atomic and we're making a consistent read.

Read isolation transaction documentation mentions about:

The isolation level is read-committed between any transactional operation and any read operation that involves multiple standard reads (BatchGetItem, Query, or Scan). If a transactional write updates an item in the middle of a BatchGetItem, Query, or Scan operation, the subsequent part of the read operation returns the newly committed value (with ConsistentRead) or possibly a prior committed value (eventually consistent reads).

But here it's not an old or new item case: an item is completely missing.

Tuna
質問済み 2ヶ月前568ビュー
2回答
1

Transactions are atomic, but they so not offer the isolation levels you are seeking with Read Committed:

If a transactional write updates an item in the middle of a BatchGetItem, Query, or Scan operation, the subsequent part of the read operation returns the newly committed value (with ConsistentRead) or possibly a prior committed value (eventually consistent reads).

It does not state that you will get either OLD or NEW image for all items in a transaction. You will be either returned OLD or NEW for each item, independently. When a transaction is committed, it's done so in parallel, which means your Query with Read Committed isolation can return either old or new for each.

To simplify it - If you receive a 200 SUCCESS from the transaction, you will be guaranteed all NEW images when using strongly consistent reads of that batch of items. If the transaction is still in-flight, it is not deemed committed, so you may see some NEW and some OLD images in a strongly consistent multi-read request (Query, Scan, BatchGetItem) as those only offer Read Committed Isolation levels.

If you require higher isolation levels, change your Query to TransactGetItem which offers Serializable isolation levels.


If you take the below info-graph based on a transaction across time.

  • T0 - You initiate the TransactWrite request
  • T1 - You initiate the Query with strong consistency
  • T2 - When all items reach T2, the TransactWrite will return a 200 SUCCESS

You can see your Query would see the OLD image for item 2 and item 4. All other items will return the NEW image. In contrast, if you make your strongly consistent request at time T3, in which the transaction had committed all items, then you would see all NEW images.

Enter image description here

profile pictureAWS
エキスパート
回答済み 2ヶ月前
  • When a transaction is committed, it's done so in parallel, which means your Query with Read Committed isolation can return either old or new for each.

    So in this case PK1-SK3 is deleted, that it's new state, so we haven't received this item. I understand that, but how come we haven't received PK1-SK4 then? What I mean is, the same transaction has deleted SK3 and added SK4 under the same PK, so a consistent read should have returned us the SK4 (if it deletes the SK3). That's why I think what we get here is not consistent at all.

    Sadly TransactGetItems doesn't work for us, because we don't know their SK. We're following Heterogeneous Item Collections approach, which assumes multiple types are under the same PK.

  • Just because they were in the same transaction doesn't change the isolation levels. When the transaction deems that all items will be committed, then it commits them in parallel. Your query, picked up SK3 was missing which was the NEW state and SK4 didn't exist, which was the OLD state. Query does not provide the isolation guarantees to enforce all NEW or all OLD for a given batch, only TransactGetItems can.

-2

Answer given by Leeroy Hannigan above seems to be perfectly correct.

AWS
回答済み 2ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ