System Design

Saved, but still seeing old data? Replication lag and read-your-writes

TL;DR

A write to the primary returns OK immediately, but changes reach replicas asynchronously, so there’s a delay. If the app reads from a replica too soon, users see stale data. Read-your-writes fixes this by routing a user’s reads to the primary right after they write.

What actually happened

  1. The user renames Ann → Zoe; the write hits the primary, commits and returns “saved”.
  2. The change is shipped to the replica asynchronously and takes about 1200ms to apply (lag varies with load).
  3. The page reloads; the read lands on the replica when it’s only 480ms in → it returns Ann.

No data was lost: the primary has held “Zoe” since the commit.

The fix: read-your-writes

ApproachProsTrade-off
After a write, route that user’s reads to the primary for a short window (e.g. 1.5s)Simple; users always see their own changesSlightly more load on the primary
Record the log position (LSN/GTID) after writing; only read from replicas that have caught upPrecise, no guessed timeoutsMore complex to implement
Optimistically update the UI from the submitted dataInstant feedbackMust handle failed writes

How to put it to work

  • Developers: Measure real replication lag (p95/p99) to pick a sensible primary-read window — don’t guess.
  • PMs / QA: “Saved but not showing” bugs are rarely data loss — check the read/write architecture before escalating.