Tech Blog of Stephen Souness

Learning from the mistakes of others A few years ago a team that worked across the office from my team came up with a neat way of introducing a cache to speed up the performance of the most business critical pages on the company's main money earning website. Page load time had been identified as a particularly significant aspect of how search engines would rank the value of websites, so getting a few milliseconds off that metric was a great achievement. Fast forward several months, the unthinkable happened as the infrastructure component that was at the heart of the caching implementation had a temporary outage, taking away the possibility of loading pages at all. It was quite a while ago, and I wasn't directly involved in the recovery process but I expect that it would have taken a stressful hour or three to recover. Avoid introducing points of failure As inevitably happens, the project that I was working on required some performance improvements in order for it to be consider...

Hello world in Java 25 can be very different to any earlier version of Java. void main() { IO.println( "Hello world" ); } That's it, no package, no import, not even having to specify a class. Even that  IO  represents a class that was only introduced in Java 25. If you saw that code in a multi-choice lineup of code that should and should not work in Java 10 years ago you wouldn't dream of regarding it as valid Java. The main method is also so different to traditional Java. There's no "public", no "static", and no parameter list. It's still early days, but I'm not a fan. 

Introduction  A few recent posts on this blog have outlined how we could achieve version aware upserting of data into various databases. In this post let's consider situations where that might be an unsuitable approach.  An assumption about Id uniqueness When we attempt to take an entity and write it into a database, we have an expectation that the attribute or attributes that are used to uniquely identify that entity are safe to handle as trustworthy within the business domain. Let's consider a situation where that assumption has been known to fall down in real production systems. Generation of a value to be used as the primary key in a relational database can seem to be a solved problem, given that we now have UUIDs that can be generated and passed around for use in our applications and services. Some earlier implementations for generation of UUID involved combining the MAC address associated with the network device of the machine and the current time as a way of combining t...

Introduction  I recently listened to a podcast episode that offered an introduction to some of the core concepts of retrieval-augmented generation (RAG) for artificial intelligence systems. One of the many points covered was the opportunity to prompt the system to inform the user if it could not determine a crorect answer. In this post I will share some real world experience of how attention to detail in API design can help or hinder this capability. Example of an unintended limitation  "A chain is only as strong as its weakest link" . Databases and search engines have different use cases and so have different associated expectations around what type of behavior is expected when the system is not fully operational. Most databases are based on a concept of indexes and records, where it is only appropriate to present back a query result if the authoritative data store has successfully been contacted and produced a result. Search engines can be a little bit looser, on the basis ...

Introduction This post will describe an approach to balancing the distribution and redistribution of data in horizontally scaled databases. It is something that I came across as part of preparing for system design interviews, which often involve discussions weighing up the pros and cons and risks of applying particular approaches to storing data. Storing structured data at scale What is structured data? Structured data is at the core of most applications on the Internet. Without going into too much detail, let's just expect that it is something that is stored in a way that enables us to consistently be able to retrieve it for later use and be in a particular known shape. How does retrieval work?  Retrieval of structured data basically involves taking an identifier, such as a primary key, and being able to quickly look up an index to the one location that is expected to give an authoritative perspective of the current state of the corresponding record. In the simplest case of a sing...

Introduction ElasticSearch has a concurrency control capability that enables writes to use a version value to determine whether to apply an update or discard it as being stale in comparison to some existing data. As part of considering a migration away from ElasticSearch as a data store, I was interested in how other databases could be made to achieve the same type of version aware upsert capability. In some earlier posts on this blog I have shared how the version aware upserting can be done with PostgreSQL, MariaDB and TitanDB. This post is to share how the same capability can be achieved with AWS's DynamoDB document database, as an example of a non-relational database. What does the DynamoDB API offer? Insert, or update  DynamoDB has putItem for creating an item in a DynamoDB table, and updateItem for updating an existing item. On first look, we might expect some combination of putItem and updateItem to need to be applied, as that would resemble how the relational databases had t...