The promise that caused the trouble
PRAMAAN reads a litigation matter's record and keeps a structured Brief of it: the parties, the chronology, what each document asserts and where. The promise we make counsel is that the Brief is simply current. A litigator uploads three PDFs and walks away. Some minutes later the Brief reflects them. No rebuild button, no spinner, no "syncing" badge. Nothing to think about.
That promise is expensive. Anything with no button is running on background machinery, and background machinery fails where nobody is looking. If a rebuild silently loses, counsel preps for a hearing off a Brief that is missing evidence, and every status light in the system says green. For most software that is a caching bug. For us it is the exact failure the product exists to prevent. So before writing the rebuild pipeline we sat down and tried to break every design we could think of, on paper, before any of them could break a matter.
This post is the record of that exercise. It ends at a pattern called a fencing token, which we did not invent, and the second half traces who did.
The question that kept killing designs
Our first design was the one everybody writes. Put a boolean on the matter: needs_rebuild. An upload flips it true. A worker sees it, rebuilds, flips it false. It works in the demo, and it survives until the first slow build.
A Brief build is not a quick UPDATE. It reads the whole record and calls a model panel, which means minutes, which means the worker can die mid-build, which means you need a lease and a sweep: if a worker goes quiet past a deadline, reclaim the job and give it to someone else. Fine. Standard. We drew the sweep on the whiteboard and someone asked the question that went on to kill three designs in a row.
What if the first worker isn't dead?
Suppose it just stalled: a slow model call, a network hiccup, a garbage-collection pause, the runtime equivalent of a long blink. The lease expires. The sweep, correctly, hands the job to a second worker, which rebuilds the Brief with the three new PDFs and writes a fresh frame. And then the first worker wakes up. Nobody told it anything. As far as it knows it has been working diligently the whole time, and now, dutifully, it writes its stale frame over the fresh one. Engineers call this worker a zombie, and the name is fair. It is dead to the system and doesn't know it.
Notice what the zombie did to our boolean. Both workers read the flag. Both believed they were the one. The flag has two states, and this problem has a timeline: it needs to say which rebuild is the real one, and a boolean cannot say that no matter how carefully you set it.
The deeper problem is the question itself. "Is that worker dead or just slow?" has no reliable answer; from the outside, a dead process and a paused one look identical until the paused one moves again. Any design that needs that answer is built on sand. What we actually needed was a design that never has to ask.
The designs we rejected
We worked through the obvious candidates before reaching for anything exotic, and each rejection taught us something.
Hold a database lock for the whole build. Postgres has advisory locks; take one per matter, hold it while building, and the lock itself keeps builds exclusive. But holding a lock means holding a connection, and our build spends its minutes inside an external model call. Pinning a pooled connection per matter for minutes at a time starves the pool. Worse, the zombie keeps its lock precisely because it isn't dead: the connection is alive, the worker is just stuck, and now nothing can rebuild that matter at all. The lock turned a stale-write problem into a stuck-matter problem.
Rent a distributed lock. Redis with Redlock, ZooKeeper, a DynamoDB lock client. This is where the literature got interesting, because Martin Kleppmann had already fought this fight in 2016 and his argument changed our design. A distributed lock is lease-based too, so the zombie scenario comes straight back: the pause can land after every "do I still hold the lock?" check you write and before the write it was guarding. No check inside the worker can be tight enough. His conclusion is that safety cannot live in the lock at all. It has to live in the thing being written to, which must reject writes whose credentials have gone stale. And the moment you accept that, the lock service becomes decoration. Our thing-being-written-to is Postgres. If Postgres has to check every write anyway, why are we running a Redis cluster to protect it?
So the design inverted. Stop guarding the worker. Guard the write.
Take a number
The shape we shipped is the one a busy counter clerk figured out long before software: don't argue about who was in line first. Hand out numbered tickets, and only serve the current number.
The ticket dispenser is one Postgres row per matter, in a table called ops.matter_frame_jobs, and the entire mechanism is two integers, a timestamp, and a boolean:
requested_generation: the dispenser. Every demand for a rebuild bumps it by one.claimed_generation: the number now being served. A worker adopts it at the moment it atomically claims the job.claimed_at: the lease. A sweep fails any claim older than the lease horizon (fifteen minutes, chosen to outlast our worst healthy build).rebuild_needed: a breadcrumb for demands that arrive mid-build. Its story comes below.
The claim is a single statement, so any number of workers can race it and Postgres picks exactly one winner:
INSERT INTO ops.matter_frame_jobs (...)
VALUES (..., 'running', false, 1, 1, now())
ON CONFLICT (firm_id, matter_id) DO UPDATE SET
run_status = 'running',
requested_generation = matter_frame_jobs.requested_generation + 1,
claimed_generation = matter_frame_jobs.requested_generation + 1,
claimed_at = now()
WHERE matter_frame_jobs.run_status <> 'running'
OR matter_frame_jobs.claimed_at < now() - make_interval(secs => :lease_seconds)
RETURNING claimed_generation
The returned claimed_generation is the worker's ticket: its fencing token. It carries the number through the whole build, and the final write of the finished frame is a compare-and-swap against it, in the same transaction as the frame insert:
UPDATE ops.matter_frame_jobs
SET run_status = 'idle'
WHERE firm_id = :firm_id AND matter_id = :matter_id
AND claimed_generation = :my_token
AND run_status = 'running'
Zero rows updated means the token is stale, and the frame aborts with it. Now replay the zombie. Worker A claims at token 6 and hangs. The lease expires; the sweep reclaims; worker B claims and the bump hands it token 7. B writes, the row says 7, the write lands. A wakes and writes with token 6. The row says 7. A's write matches zero rows and dies, harmlessly, at the ledger.
Nobody ever asked whether A was dead. That is the whole trick, and it is Kleppmann's core insight: the storage checks the token, never the process. A fencing token is, in his words, "simply a number that increases... every time a client acquires the lock," and the storage must take "an active role in checking tokens, and rejecting any writes on which the token has gone backwards." You stop asking "is this worker alive?", a question no distributed system can answer, and start asking "is this write current?", a question a single Postgres row answers for free.
The machinery around the row
The row needs a delivery system, and ours is worth a short tour because the same trust principle runs through it: the queue is never the truth.
When a matter is created, the same transaction that inserts it also inserts an event row into ops.domain_event_outbox. Fact and record commit together or not at all. A relay drains that table once a minute onto an EventBridge bus, and the bus fans out to SQS. But the SQS message carries only IDs and trace context. It is a nudge, a tap on the shoulder that says "go look at the ledger." A dropped nudge loses nothing; the row is still there. A duplicated nudge corrupts nothing; the consumer dedupes on the event's ID. Even the event names are grammar-checked by the database, a CHECK constraint pinning every one to a lowercase noun.verb fact like matter.created. You cannot put a wish on our bus. Only history.
In our vocabulary, the whole standing lane is the Brief rail: its trigger (the rule with its gates that turns bus facts into work), its jobs (the leased rows), its runs (individual attempts, the thing you debug), and its persistence. The bus stays outside; a rail hears a shared fact only through its own trigger. Three PDFs land as three events, the trigger coalesces them, and one job serves them all.
Two lawyers, one build
The row also absorbs the human races, which is where that fourth column earns its place.
Two lawyers on the same matter click Regenerate in the same second. Both nudges race the claim; Postgres serializes them on the row; one wins and starts building. The loser doesn't error and doesn't start a parallel build. It sets rebuild_needed = true and exits. When the running build finishes, that breadcrumb is served by exactly one follow-up build with a fresh token, and the flag is cleared in the same statement that requeues it, so a doomed rebuild can't requeue itself forever. A click that lands mid-build takes the same path. However many demands pile up during one build, they collapse to one pending rebuild.
What did all this buy us? Four things we can state as guarantees rather than hopes. Two builds for one matter never run in parallel. A zombie's stale frame can never land, because the token check and the frame write share a transaction. N demands during a build cost one model invocation, not N. And "is there unserved demand?" is a fact you can read off the row, because claimed_generation < requested_generation says so in one comparison. A boolean can only say that someone, at some point, wanted something. The counter says how many times, and whether the work has caught up with the want. One of those is evidence. We are a company built on the difference.
Why "fencing"
The word is older than the token. In high-availability clustering, fencing meant literally fencing a suspect node off from the shared disks, because a node that looks dead is only suspected dead, and if it is merely slow it will come back and corrupt shared storage. Same zombie, 1990s edition. The Linux-HA community's bluntest tool was STONITH, "Shoot The Other Node In The Head": power-cycle the machine and make the speculation true. (The acronym began as a misspelling of STOMITH, for machine, and stuck because it sounded like the King James verb stoneth.) The gentler tools revoked access instead: SCSI persistent reservations, disabled SAN ports. Build the fence at the resource, not in the node's good intentions.
A fencing token is that same fence built out of arithmetic. Nobody power-cycles our zombie worker. The WHERE clause just declines to open the gate.
The same trick, under other names
Once you know its shape you find it everywhere, usually renamed. Google's Chubby lock service (2006) called it a sequencer: a byte-string with a lock generation number that a client passes to servers, which "test whether the sequencer is still valid... if not, reject the request." Kafka calls it the producer epoch and borrowed the whole word: stale transactional producers "are fenced off" and die with ProducerFencedException. Raft calls it the term; Paxos the ballot number; Viewstamped Replication the view number; half the industry just says epoch. ZooKeeper's zxids, etcd's revisions, DynamoDB's condition expressions, HTTP's If-Match and the 412 status code, S3's conditional writes (finally, in 2024): all the same move, a monotonic number checked at the resource. Hazelcast ships the only mainstream API that says the quiet part out loud: FencedLock.getFence().
Kleppmann's 2016 essay, and chapter 8 of his Designing Data-Intensive Applications, gave the pattern its modern name and its clearest diagram. The debate it kicked off is worth your evening. antirez's rebuttal argues that storage able to check monotonic tokens was the real arbiter all along, which is less a refutation than a restatement of why we skipped the lock service entirely. And Lorin Hochstein's formal modeling shows the fine print: tokens guarantee ordering, not mutual exclusion. We read all three before committing. The pattern survived the reading.
Nothing here is new
Count what the Brief rail is actually made of: a transactional outbox, a lease with a sweep, a compare-and-swap, a fencing token. Four named patterns, all published and battle-tested by other people, most of them older than our company. We invented none of it, and that is the point. Our engineering was the assembly: making Postgres the single ledger and demoting every queue to a nudge, and wiring the token check into the same transaction as the frame write so a stale build cannot half-land.
If you take one line away, take the clerk's. Don't ask a worker whether it still owns the work. Hand out numbers, and let the counter refuse to serve a stale ticket.
References: Martin Kleppmann, "How to do distributed locking" (2016) and Designing Data-Intensive Applications (O'Reilly, 2017), ch. 8; Mike Burrows, "The Chubby lock service" (OSDI 2006), §2.4; Salvatore Sanfilippo, "Is Redlock safe?"; Lorin Hochstein, "Locks, leases, fencing tokens, FizzBee!"; Mehta & Gustafson, "Transactions in Apache Kafka" (Confluent, 2017).