Introduction Just jotting down some thoughts about what might be involved for addressing an issue being faced in my last project. The problematic situation Multiple sources of updates arriving in concurrently, being picked up for inclusion in an aggregated representation of the data. There are multiple worker processes, each with multiple worker threads that pick up changes and apply them without any awareness or consideration of what other work is underway. Potential solution approaches Debouncing of updates Using Redis as a store for keeping track of the identifier of records that are currently being processed has been successfully applied for reducing race conditions for updates of some attributes, so the pattern could be applied more broadly. The debouncing approach can be thought of as a type of transaction lock restricting access to the record that is required to contain the full representation of state. Partitioning workload processing This would most likely involve needing to s...
Professional software developer, producing scalable applications on the Internet