{"id":47129394,"title":"Show HN: Crash-safe job queue – lease-expiry race and fencing fix","points":1,"user":"kritibehl","time":1771883188,"time_ago":"8 hours ago","type":"link","content":"

Most lease-based job queues look correct until you test them adversarially.

I built Faultline, a PostgreSQL-backed distributed job execution engine using:

- Lease-based claims\n- Retry scheduling\n- Idempotent side effects via a ledger table\n- A deterministic race reproduction harness

The interesting part wasn’t the happy path.\nIt was the lease-expiry race.

Setup:

- Lease TTL: 1s\n- Worker A sleeps 2.5s (forces expiry)\n- Barrier enforces deterministic ordering\n- Worker B reclaims the job

Structured trace:

{"event": "lease_acquired", "job_id": "...", "token": 1, "forced": true}\n{"event": "execution_started", "job_id": "...", "token": 1}\n{"event": "lease_acquired", "job_id": "...", "token": 2, "forced": true}\n{"event": "execution_started", "job_id": "...", "token": 2}\n{"event": "stale_write_blocked", "job_id": "...", "stale_token": 1, "current_token": 2, "reason": "token_mismatch"}\n{"event": "worker_exit", "reason": "stale"}\n{"event": "worker_exit", "reason": "success"}

Worker A believed it still owned the lease.\nWorker B legitimately reclaimed it.

Without fencing, Worker A could still attempt mutation.

UNIQUE(job_id) alone is insufficient — it prevents duplicate rows but does not encode lease epoch ownership.

The fix:

- Add `fencing_token BIGINT`\n- Increment atomically on every lease acquisition\n- Bind side effects to `(job_id, fencing_token)`\n- Enforce a write gate before mutation

Claim logic:

UPDATE jobs\nSET state='running',\n lease_owner=$1,\n lease_expires_at = NOW() + make_interval(secs => $2),\n fencing_token=fencing_token+1,\n updated_at=NOW()\nWHERE id=$3\n AND (\n state='queued'\n OR (state='running' AND lease_expires_at < NOW())\n )\nRETURNING id, fencing_token;

Lease validity depends solely on DB time (`NOW()`); workers never use local clocks for correctness.

Guarantees under forced expiry + reclaim:

- No duplicate side effects\n- No stale worker mutation\n- Deterministic reproduction of the race\n- DB-enforced epoch ownership via `(job_id, fencing_token)`

The harness forces this race deterministically via barrier gating and forced TTL expiry.

Curious how others handle fencing under lease-based execution — specifically how teams handle fencing token overflow at scale and whether renewal logic changes the fencing guarantee.","url":"https://github.com/kritibehl/faultline","domain":"github.com","comments":[{"id":47129524,"level":0,"user":"kritibehl","time":1771883744,"time_ago":"8 hours ago","content":"

The race is deterministic via barrier + forced TTL expiry. Happy to paste the exact pytest harness if useful.","comments":[]}],"comments_count":1}