jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-11-01 23:11:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: fold review round 1 into cluster-mode spec (deployment gate, slot-rules corrections, drain fix)

Browse Source 
Review blockers from Wren (three: E1 deployment-gate assumption, compatible-
mode cross-slot legality unsupported, proof-surface overclaim) and Dawn (two:
CROSSSLOT enforced on single-shard cluster-enabled nodes falsifies the A/B-in-
compatible premise; renew=PEXPIRE makes any temporal drain gate unsound).

Spec: A2/A3 rewritten (declared-keys slot check, undeclared-KEYS prohibition,
slot rules from compatible onward as working assumption + pre-G3 probe);
migration reordered A->B->backfill->drain->C entirely in disabled mode with
compatible after C; C-gate now participation-based (B renew-migrates old-key
leases; observed-zero, never wait-a-TTL); per-phase operation table for all
six directory ops; proof scope stated honestly (write-side under E1);
Deployment Gate section discharging E1 with rollback matrix and phase gauge;
1.5.0 receipts (verified free single-node); prior-art section (Stripe,
GitLab, BullMQ, Sidekiq, Grafana, ioredis #1842) with locality contract and
executable cross-slot validator in G2; failure-modes table extended.

Model: mode variable disabled/compatible/enabled with slot rules from
compatible; MigrateB (renew-migrate, same generation); RollbackPhase +
DowngradePod with drain + reverse-max-merge guards; EnterCompatible/
RevertCompatible/EnableCluster doors; Inv_ClusterSafe -> Inv_SlotRulesSafe.
A's generation dual-write removed: mutation testing proved it redundant.
TLC green (12636 states, 3637 distinct); 9 mutants (M1-M9) all killed by
their intended invariants.

Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
This commit is contained in:
npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d
2026-07-30 11:49:10 -04:00
parent 549fd58150
commit 2c7fdc55fa
3 changed files with 495 additions and 131 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/spec/RedisClusterFencingMigration.cfg
View File

@@ -7,5 +7,5 @@ INVARIANTS
TypeOK
Inv_SingleAuthority
Inv_Monotonic
Inv_ClusterSafe
Inv_SlotRulesSafe
Inv_LeaseGenGrounded

200
docs/spec/RedisClusterFencingMigration.tla
View File

@@ -16,12 +16,40 @@
(* generation = max(oldGen,newGen)+1 written to BOTH. *)
(* version B (2): union lease check; lease written to NEW key; *)
(* generation = max(oldGen,newGen)+1 written to NEW. *)
(* B's renewer MIGRATES any old-key lease it owns to the *)
(* new key (same generation, atomic pre-cluster Lua) — *)
(* this is what makes the old keyspace DRAIN by *)
(* participation instead of by hope: renew is PEXPIRE, *)
(* so a renewed lease never expires and no fixed wait *)
(* drains anything (spec S:B2). *)
(* backfill: after full-B, fold oldGen into newGen (max-merge). *)
(* version C (3): new keys only (cluster-safe: no cross-slot reads). *)
(* enable: cluster mode enabled; old keys become unreachable. *)
(* version C (3): new keys only (cluster-safe: no cross-slot access). *)
(* mode: 0 = disabled, 1 = compatible, 2 = enabled. *)
(* SLOT RULES (CROSSSLOT rejection) are enforced from *)
(* mode 1 onward: "one shard owns all slots" grants no *)
(* cross-slot exemption (falsified live on a single-shard *)
(* cluster-enabled node — spec S:B1). O/A/B/migrate are *)
(* cross-slot scripts, so they are guarded on mode = 0; *)
(* under slot rules those scripts fail loudly and mutate *)
(* nothing. 1 -> 0 is a supported revert; 1 -> 2 is the *)
(* one-way door. *)
(* *)
(* Rolling deploys mean adjacent versions coexist; the phase variable *)
(* enforces that the fleet spans at most versions {phase-1, phase}. *)
(* ENVIRONMENT ASSUMPTION E1 (deployment gate): the phase machinery below *)
(* — AdvancePhase requiring the whole fleet at the current phase, and *)
(* RollbackPhase requiring the downgrade to settle before rolling back *)
(* further — is NOT a property of Kubernetes RollingUpdate. It must be *)
(* discharged operationally (spec §Deployment Gate): one image per phase, *)
(* next image change blocked until all live pods report the expected *)
(* phase and old ReplicaSets are at zero, skipped-phase rollback *)
(* prohibited. Mutations M6–M8 show the invariants fail without E1. *)
(* *)
(* SCOPE OF PROOF: the model covers acquisition authority, generation *)
(* issuance, lease migration/loss, and the mode doors. Renew (TTL *)
(* extension) is subsumed by the untimed nondeterminism (a renewed lease *)
(* is one where Expire has not yet fired); release is subsumed by Expire *)
(* (both are lease loss without generation change); validate is read-only *)
(* and its per-phase read rule is specified and tested, not modeled *)
(* (spec §Per-phase operation table). *)
(* *)
(* One session is modeled; sessions are independent (per-session keys). *)
(***************************************************************************)
@@ -43,12 +71,12 @@ VARIABLES
oldGen, \* OLD non-expiring generation counter
newGen, \* NEW non-expiring generation counter
backfilled, \* one-time oldGen -> newGen max-merge has run
clusterOn, \* cluster mode enabled (the one-way door)
mode, \* 0 = disabled, 1 = compatible, 2 = enabled (one-way)
lastIssued, \* history: highest generation ever issued to a lease
monoOk \* history flag: FALSE iff an issue was <= a prior issue
vars == <<phase, ver, oldOwner, oldLeaseGen, newOwner, newLeaseGen,
oldGen, newGen, backfilled, clusterOn, lastIssued, monoOk>>
oldGen, newGen, backfilled, mode, lastIssued, monoOk>>
Max(a, b) == IF a > b THEN a ELSE b
@@ -62,7 +90,7 @@ TypeOK ==
/\ oldGen \in 0..MaxGen
/\ newGen \in 0..MaxGen
/\ backfilled \in BOOLEAN
/\ clusterOn \in BOOLEAN
/\ mode \in 0..2
/\ lastIssued \in 0..MaxGen
/\ monoOk \in BOOLEAN
@@ -73,31 +101,60 @@ Init ==
/\ newOwner = None /\ newLeaseGen = 0
/\ oldGen = 0 /\ newGen = 0
/\ backfilled = FALSE
/\ clusterOn = FALSE
/\ mode = 0
/\ lastIssued = 0
/\ monoOk = TRUE
(***************************************************************************)
(* Deployment machinery. *)
(* Deployment machinery (environment assumption E1 — see header). *)
(***************************************************************************)
\* The fleet may only chase the current phase; a phase only advances when
\* every pod has reached it. Gate into C (phase 3): backfill has run AND no
\* live old-key lease remains (operationally: wait >= lease TTL after
\* full-B before deploying C).
\* live old-key lease remains. The drain conjunct is participation-based
\* (B migrates owned old-key leases; orphaned leases expire by TTL), NOT a
\* fixed wait — renew resets the TTL, so "wait one TTL" drains nothing.
AdvancePhase ==
/\ phase < 3
/\ \A p \in Pods : ver[p] = phase
/\ (phase = 2) => (backfilled /\ oldOwner = None)
/\ phase' = phase + 1
/\ UNCHANGED <<ver, oldOwner, oldLeaseGen, newOwner, newLeaseGen,
oldGen, newGen, backfilled, clusterOn, lastIssued, monoOk>>
oldGen, newGen, backfilled, mode, lastIssued, monoOk>>
UpgradePod(p) ==
/\ ver[p] < phase
/\ ver' = [ver EXCEPT ![p] = phase]
/\ UNCHANGED <<phase, oldOwner, oldLeaseGen, newOwner, newLeaseGen,
oldGen, newGen, backfilled, clusterOn, lastIssued, monoOk>>
oldGen, newGen, backfilled, mode, lastIssued, monoOk>>
DowngradePod(p) ==
/\ ver[p] > phase
/\ ver' = [ver EXCEPT ![p] = phase]
/\ UNCHANGED <<phase, oldOwner, oldLeaseGen, newOwner, newLeaseGen,
oldGen, newGen, backfilled, mode, lastIssued, monoOk>>
\* Emergency rollback: one adjacent phase at a time, only in disabled mode
\* (from compatible, revert the mode first), never past a running
\* downgrade (no pod may still be above the current target — dropping
\* this guard is mutation M8). Rolling back into O (phase 0) additionally
\* requires the new keyspace to be lease-drained (dropping this is M6) and
\* re-syncs the old counter by reverse max-merge: O issues from the old
\* counter alone, so without the merge it re-issues generations B already
\* issued (dropping the merge is M7). Every rollback conservatively
\* invalidates the backfill; it must re-run before the C-gate can pass.
RollbackPhase ==
/\ phase > 0
/\ mode = 0
/\ \A p \in Pods : ver[p] <= phase
/\ IF phase = 1
THEN /\ newOwner = None
/\ oldGen' = Max(oldGen, newGen)
ELSE UNCHANGED oldGen
/\ phase' = phase - 1
/\ backfilled' = FALSE
/\ UNCHANGED <<ver, oldOwner, oldLeaseGen, newOwner, newLeaseGen,
newGen, mode, lastIssued, monoOk>>
\* One-time backfill after the whole fleet is on B: fold the old counter
\* into the new one. Idempotent max-merge.
@@ -107,20 +164,39 @@ Backfill ==
/\ newGen' = Max(newGen, oldGen)
/\ backfilled' = TRUE
/\ UNCHANGED <<phase, ver, oldOwner, oldLeaseGen, newOwner, newLeaseGen,
oldGen, clusterOn, lastIssued, monoOk>>
oldGen, mode, lastIssued, monoOk>>
\* The one-way door. Only after the whole fleet runs C scripts (which
\* never touch old keys, so no cross-slot access exists to break).
EnableCluster ==
/\ ~clusterOn
(***************************************************************************)
(* Mode doors. Slot rules turn on at compatible (S:B1), so the whole *)
(* fleet must already run C — the only slot-clean script version. *)
(* compatible -> disabled is the supported revert; enabled is one-way. *)
(***************************************************************************)
EnterCompatible ==
/\ mode = 0
/\ phase = 3
/\ \A p \in Pods : ver[p] = 3
/\ clusterOn' = TRUE
/\ mode' = 1
/\ UNCHANGED <<phase, ver, oldOwner, oldLeaseGen, newOwner, newLeaseGen,
oldGen, newGen, backfilled, lastIssued, monoOk>>
RevertCompatible ==
/\ mode = 1
/\ mode' = 0
/\ UNCHANGED <<phase, ver, oldOwner, oldLeaseGen, newOwner, newLeaseGen,
oldGen, newGen, backfilled, lastIssued, monoOk>>
EnableCluster ==
/\ mode = 1
/\ mode' = 2
/\ UNCHANGED <<phase, ver, oldOwner, oldLeaseGen, newOwner, newLeaseGen,
oldGen, newGen, backfilled, lastIssued, monoOk>>
(***************************************************************************)
(* Acquire, per script version. Each is one atomic Lua script. *)
(* Acquire, per script version. Each is one atomic Lua script. O, A, B *)
(* and the B-migrate touch both keyspaces (cross-slot), so they are *)
(* guarded on mode = 0: under slot rules they fail loudly and mutate *)
(* nothing, which the guard models by absence. *)
(***************************************************************************)
RecordIssue(g) ==
@@ -130,43 +206,66 @@ RecordIssue(g) ==
\* Version O (legacy, deployed today): old keys only.
AcquireO(p) ==
/\ ver[p] = 0
/\ mode = 0
/\ oldOwner = None
/\ oldGen + 1 <= MaxGen
/\ oldGen' = oldGen + 1
/\ oldOwner' = p /\ oldLeaseGen' = oldGen + 1
/\ RecordIssue(oldGen + 1)
/\ UNCHANGED <<phase, ver, newOwner, newLeaseGen, newGen, backfilled,
clusterOn>>
mode>>
\* Version A: union lease check; lease still on OLD key; generation is
\* max-merged and written to BOTH counters. (The max-merge here is
\* load-bearing: without it, an A-pod acquiring after a B-pod re-issues a
\* generation B already issued — see mutation M5.)
\* max-merged over BOTH counters but written only to the OLD one. (An
\* earlier draft dual-wrote the generation to both counters; mutation
\* testing showed the dual-write redundant — the mutant removing it
\* passes the full state space, because B's union max-merge read and the
\* backfill already carry A's issues forward. The max-merge READ is the
\* load-bearing part: dropping it is mutation M5.)
AcquireA(p) ==
/\ ver[p] = 1
/\ mode = 0
/\ oldOwner = None /\ newOwner = None
/\ LET g == Max(oldGen, newGen) + 1 IN
/\ g <= MaxGen
/\ oldGen' = g /\ newGen' = g
/\ oldGen' = g
/\ oldOwner' = p /\ oldLeaseGen' = g
/\ RecordIssue(g)
/\ UNCHANGED <<phase, ver, newOwner, newLeaseGen, backfilled, clusterOn>>
/\ UNCHANGED <<phase, ver, newOwner, newLeaseGen, newGen, backfilled,
mode>>
\* Version B: union lease check; lease moves to the NEW key; generation
\* max-merged into the NEW counter.
AcquireB(p) ==
/\ ver[p] = 2
/\ mode = 0
/\ oldOwner = None /\ newOwner = None
/\ ~clusterOn \* B reads old keys: cross-slot, pre-cluster only
/\ LET g == Max(oldGen, newGen) + 1 IN
/\ g <= MaxGen
/\ newGen' = g
/\ newOwner' = p /\ newLeaseGen' = g
/\ RecordIssue(g)
/\ UNCHANGED <<phase, ver, oldOwner, oldLeaseGen, oldGen, backfilled,
clusterOn>>
mode>>
\* Version C: new keys only. Cluster-safe.
\* Version B renew-migrate: a B-pod that still owns an old-key lease
\* (acquired before its upgrade) moves it to the new key at its next renew
\* tick — same owner, SAME generation (no new issuance; monotonicity is
\* untouched), atomic cross-slot Lua, legal only in disabled mode. The
\* script requires the new key empty; the counter is max-merged so the
\* moved lease stays grounded. This action is what discharges the C-gate's
\* oldOwner = None conjunct by participation.
MigrateB(p) ==
/\ ver[p] = 2
/\ mode = 0
/\ oldOwner = p
/\ newOwner = None
/\ newOwner' = p /\ newLeaseGen' = oldLeaseGen
/\ newGen' = Max(newGen, oldLeaseGen)
/\ oldOwner' = None /\ oldLeaseGen' = 0
/\ UNCHANGED <<phase, ver, oldGen, backfilled, mode, lastIssued, monoOk>>
\* Version C: new keys only. Slot-clean; runs in every mode.
AcquireC(p) ==
/\ ver[p] = 3
/\ newOwner = None
@@ -175,32 +274,37 @@ AcquireC(p) ==
/\ newOwner' = p /\ newLeaseGen' = newGen + 1
/\ RecordIssue(newGen + 1)
/\ UNCHANGED <<phase, ver, oldOwner, oldLeaseGen, oldGen, backfilled,
clusterOn>>
mode>>
(***************************************************************************)
(* Lease loss: TTL expiry or explicit release. Generations survive. *)
(* Lease loss: TTL expiry or explicit release (same state effect). *)
(* Generations survive. *)
(***************************************************************************)
ExpireOld ==
/\ oldOwner /= None
/\ oldOwner' = None /\ oldLeaseGen' = 0
/\ UNCHANGED <<phase, ver, newOwner, newLeaseGen, oldGen, newGen,
backfilled, clusterOn, lastIssued, monoOk>>
backfilled, mode, lastIssued, monoOk>>
ExpireNew ==
/\ newOwner /= None
/\ newOwner' = None /\ newLeaseGen' = 0
/\ UNCHANGED <<phase, ver, oldOwner, oldLeaseGen, oldGen, newGen,
backfilled, clusterOn, lastIssued, monoOk>>
backfilled, mode, lastIssued, monoOk>>
Next ==
\/ AdvancePhase
\/ EnableCluster
\/ RollbackPhase
\/ Backfill
\/ EnterCompatible
\/ RevertCompatible
\/ EnableCluster
\/ ExpireOld
\/ ExpireNew
\/ \E p \in Pods : UpgradePod(p) \/ AcquireO(p) \/ AcquireA(p)
\/ AcquireB(p) \/ AcquireC(p)
\/ \E p \in Pods : UpgradePod(p) \/ DowngradePod(p)
\/ AcquireO(p) \/ AcquireA(p) \/ AcquireB(p)
\/ MigrateB(p) \/ AcquireC(p)
Spec == Init /\ [][Next]_vars
@@ -209,19 +313,29 @@ Spec == Init /\ [][Next]_vars
(***************************************************************************)
\* T1: fencing authority never forks — at most one live lease per session
\* across BOTH keyspaces, at every instant of the migration.
\* across BOTH keyspaces, at every instant of the migration, including
\* rollbacks.
Inv_SingleAuthority == oldOwner = None \/ newOwner = None
\* T2: generations are strictly monotonic per session — no generation is
\* ever issued twice, across the old/new counter handoff.
\* ever issued twice, across the old/new counter handoff and across
\* rollback re-merges.
Inv_Monotonic == monoOk
\* T3: the one-way door is only crossed with no old-keyspace authority
\* left: backfill done and no live old-key lease. (After this point C
\* scripts never read old keys, so any residual old-key state is inert.)
Inv_ClusterSafe == clusterOn => (backfilled /\ oldOwner = None)
\* T3: slot rules (compatible or enabled) are only ever on with the old
\* keyspace fully drained and the whole fleet on slot-clean scripts —
\* so no post-door execution ever attempts a cross-slot fencing script,
\* and any residual old-key state is inert garbage.
Inv_SlotRulesSafe ==
(mode >= 1) => /\ backfilled
/\ oldOwner = None
/\ phase = 3
/\ \A p \in Pods : ver[p] = 3
\* Sanity: a live lease always carries the generation its counter issued.
\* The door is one-way: this is an axiom of the environment (A3), recorded
\* as an action guard (no transition leaves mode = 2), not an invariant.
\* Sanity: a live lease always carries a generation its counter dominates.
Inv_LeaseGenGrounded ==
/\ (oldOwner /= None) => (oldLeaseGen > 0 /\ oldLeaseGen <= oldGen)
/\ (newOwner /= None) => (newLeaseGen > 0 /\ newLeaseGen <= newGen)