Upserts with version awareness in DynamoDB

Introduction

ElasticSearch has a concurrency control capability that enables writes to use a version value to determine whether to apply an update or discard it as being stale in comparison to some existing data.

As part of considering a migration away from ElasticSearch as a data store, I was interested in how other databases could be made to achieve the same type of version aware upsert capability.

In some earlier posts on this blog I have shared how the version aware upserting can be done with PostgreSQL, MariaDB and TitanDB.

This post is to share how the same capability can be achieved with AWS's DynamoDB document database, as an example of a non-relational database.

What does the DynamoDB API offer?

Insert, or update 

DynamoDB has putItem for creating an item in a DynamoDB table, and updateItem for updating an existing item.

On first look, we might expect some combination of putItem and updateItem to need to be applied, as that would resemble how the relational databases had to detect conflict and fall back to attempt the second type of operation.

It turns out that we can just use putItem, as the API documentation states:

"If an item with the same key already exists in the table, it is replaced with the new item."

So, that takes care of the insert otherwise update aspect of the implementation, but what about version awareness?

Conditional updating writing 

The putItem API offers us the option of specifying some conditional logic that includes the ability to compare existing data against the data being sent.

As this is not involving the updateItem functionality, I decided that it was inappropriate to have "updating" in the heading. Here we are either creating a new item or overwriting an existing one, whereas the updateItem call would be for updating an existing item. 

If we have a table called event, containing an id and a version then we can have a call like the following:

String idAsString = "event-123";
String version = "456";

Map<String, AttributeValue> eventDataUpdating = Map.of(
        "id", AttributeValue.builder().s(idAsString).build(),
        "version", AttributeValue.builder().n(version).build());

dbClient.putItem(
        PutItemRequest.builder().tableName("event")
                .item(eventDataUpdating)
                .conditionExpression("attribute_not_exists(id) OR (version < :version)")
                .expressionAttributeValues(Map.of(":version", AttributeValue.builder().n(version).build()))
                .returnValuesOnConditionCheckFailure(ReturnValuesOnConditionCheckFailure.ALL_OLD)
                .build());

In that code, conditionExpression and expressionAttributeValues combine to express the two situations that determine whether the content should be written into the event table:

• attribute_not_exists covers the insert case, as there is no existing record with the specified id;
• version < :version covers the situation where an existing item exists but has a lower version value than what is being provided now.

Try it out

I've been experimenting in Java, using the Localstack Docker container as a standalone environment for interacting with DynamoDB, so you can grab the code and try running it for yourself.

At the time of this post, it is just a single class that:

• creates the table
• writes an initial low version
• sets up 100 randomly ordered version values
• spins up virtual threads that each pick up one of the version values and concurrently attempt to apply the update using the condition check
• prints out when a conflict has prevented an attempted update (as expected) 
• verifies that when the dust has settled we ultimately end up with the highest version being written

Code in GitHub  

(You'll need Java 21 or later, Maven, and Docker). 

Disclaimer

So far I have only scratched the surface of how to achieve the desired functionality.

I would not recommend applying this approach without also diving deep into the documentation for further layers of potential limitations and situations where eventual consistency may make this less appropriate than it appears.

Follow-on Curiosity

Is this how global tables address conflict?

I wonder if DynamoDB global tables apply similar logic when writes to the same item occur concurrently in different regions.

From the documentation about multi-region strong consistency, "Conditional write operations always evaluate the condition expression against the latest version of an item. Updates always operate against the latest version of an item."

I suppose the ReplicatedWriteConflictException could simply be a mapping from when ConditionalCheckFailedException is encountered.