Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

How do ensure consistency in versioned items?

0

Here is my DDB table structure:

PK: <country>#<state>#<shardId> SK: <city>#<entityId>#<entityVersion> payload: base64 encoded binary string lastModified: epoch time for optimistic locking

Here, entityVersion will be an incrementing number; I will be padding it with 0s to ensure lexicographic sorting. Every time a new version of entityId is created, I create a new DDB item with new entityVersion.

shardId will be 0-10 by hashing entityId.

My access pattern:

  1. Find the latest version of the entity within the city. I plan to use begins_with <city>#<entityId># and SORT DESC SK LIMIT
  2. Find a specific version of the item within the city. A simple equality on the operation on PK and SK.
  3. Delete all idems within a city: perform SK begins_with <city>#
  4. delete all entities within country and state. Find all items where PK is from <country>#<state>#0 to <country>#<state>#10 and delete them.

Now the problem here is that,

PKSKpayloadlastModified
usa#california#8san-francisco#person1#0001data10000023
usa#california#8san-francisco#person1#0002new_data10000045

Now, the problem here is that while creating a new version, I retrieve the latest version from DDB, apply changes, and create a new version. When the same version is updated, I can write a conditional write on the date field.

However, when I want to create a new version, if I don't perform a strong, consistent read, I may end up applying changes on 0001 instead of the latest version, i.e., 0002.

I was wondering if there is a way to get around this without using a strong, consistent read, as DDB will return the latest majority of the time, even with eventual consistent reads.

Topics
Database
Tags
Amazon DynamoDB
Language
English

asked a year ago180 views

2 Answers
1
Accepted Answer

Why do you have to use an incrementing number, its simply adding complexity that you have to read the latest version. Why not use time sortable UUID such a ULID

A ULID (Universally Unique Lexicographically Sortable Identifier) is a 26-character string that's both unique and sortable by time.

Here's an example: 01ARZ3NDEKTSV4RRFFQ69G5FAV

This ULID breaks down as: • First 10 characters (01ARZ3NDEK) represent the timestamp • Last 16 characters (TSV4RRFFQ69G5FAV) are random

ULIDs are case-insensitive, use Crockford's Base32 encoding, and sort lexicographically by creation time, making them ideal for database primary keys and distributed systems.

PKSKpayloadlastModified
usa#california#8san-francisco#person1#01ARZ3NDEKTSV4RRFFQ69G5FAVdata10000023
usa#california#8san-francisco#person1#01ARZ3NDENTSV4RRFFQ69G5FAVnew_data10000045

Now instead of reading everytime you need a new version, you simply just call PutItem knowing your new version is larger and unique.

AWS
EXPERT

answered a year ago

-1

To ensure consistency when creating new versions of your entities in DynamoDB, you have several options:

  1. Version attribute with optimistic locking: Instead of relying solely on your entityVersion in the sort key, you could add a separate version attribute to each item. When creating a new version, you would first read the latest version, increment the version number, and then use a conditional write to ensure no one else has updated it in the meantime. This is a form of optimistic locking that DynamoDB supports well.

  2. Sort key versioning pattern: You could implement a pattern where you maintain two copies of each item - one with a special prefix (like "v0_") that always represents the latest version, and another with an incremental version number. When updating, you'd create a new versioned item and also update the "latest" item. This way, you can always easily retrieve the latest version without sorting.

  3. Use conditional expressions: When creating a new version, you could use a conditional expression to check if the version you're basing your changes on is still the latest. This would prevent creating inconsistent versions if someone else has updated the item.

  4. Leverage the lastModified attribute: Since you already have a lastModified attribute, you could use it in conditional expressions to ensure you're working with the most up-to-date version before creating a new one.

While eventual consistency reads will work most of the time, for version control scenarios where accuracy is critical, strongly consistent reads are recommended. The performance impact is typically minimal compared to the potential data consistency issues you might encounter.

If you're concerned about the performance impact of strongly consistent reads, you could implement a hybrid approach where you use eventual consistency for most operations but switch to strongly consistent reads specifically for the version creation flow.
Sources
Best practices for implementing version control in DynamoDB - Amazon DynamoDB
DynamoDB and optimistic locking with version number - Amazon DynamoDB
Best practices for using sort keys to organize data in DynamoDB - Amazon DynamoDB

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content