mirror of
https://github.com/block/buzz.git
synced 2026-10-27 04:30:52 +00:00
main
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Wes
|
6a86531ac9 |
feat(relay): add atomic complete read-state snapshots (#7572)
Opened by Brain on behalf of Wes (GitHub: wesbillman). ## Summary Add an opt-in, host-scoped completeness primitive for NIP-RS read-state clients. Ordinary query arrays and WebSocket behavior remain unchanged. - Advertise a version-1 `read_state_snapshot` capability in NIP-11 only when the request resolves a community. - Accept exactly one self-author kind-30078 snapshot filter on authenticated `POST /query`, after the existing admission, replay and membership gates. - Read all retained, nondeleted self-author kind-30078 coordinates from the writer at one SQL-statement MVCC cut—including unrelated application coordinates and events without a read-state tag. - Return a complete versioned envelope bound to community, signer and a content-identity hash. Reject malformed contracts, reconstruction/signature/storage failures and whole-request overflow; never return a truncated complete response. - Bound snapshots at 4,096 events and 8 MiB of compact encoded event-array bytes, advertised as `max_event_array_bytes`; envelope metadata is additional. Retain the separate stored-payload preflight budget. ## Safety and scope This is point-in-time enumeration, **not** a live freshness guarantee, cursor, CAS token, cross-relay transaction, or unread total. NIP-RS now explicitly requires fresh complete enumeration from every write relay before each override action, canonical publication, carry-forward or deletion. It does not enable synchronized manual-unread overrides in any client. No schema migration or deployment is included. ## Validation Executed against the source committed as `1fd5a5e0c0d893c1259386885517c0c44ae7630f`: - Four real-router/PostgreSQL snapshot cases: beyond-page-cap enumeration and overflow; corruption/storage failure; production payload-bound NIP-98, Redis replay guard and membership enforcement; host discovery/scope/strict request shape. - Two database cases: replacement during preflight preserves a single statement cut; encoded-byte overflow and writer-not-replica routing. The cluster-wide probe uses the required `cluster_global_` serialization prefix. - Production-call-site mutations killed for dispatch, strict filter validation, discovery tenant, snapshot tenant, encoded-byte guard, membership enforcement and replay enforcement; isolated mutation tree restored. - Both affected crates pass all-target Clippy; formatting, differential file-size and repository PostgreSQL test-discovery checks pass. - Normal pre-push gates pass without bypass: branch skew, push-head scope, file size, Rust unit lanes and desktop Tauri checks. - Independent read-only review by Pinky found no remaining relay blocker at this commit. The execution/mutation results above were run by Brain, not duplicated by the reviewer. Hosted CI and maintainer review remain separate gates. No production traffic or deployment was used for these fixture tests. Originating Buzz channel: `a394ecdb-9c67-4c4e-a3c6-4a95f69e8cc8` (unread-but-good), thread `2413d0ba7fa5b2ac5c777df6b3754e2099b042ec8fb96ca64302a0919a0550b3`. ## Review repair (2026-09-21) Carl updated this existing branch to `2f9c9b11266c24947412cf28cefae4e96e7167b7`, merging main `5079c770fe30bb3d8204822ce6c2431eacac6d4b` without rewriting published history. - Resolved the DB export conflict, preserving both snapshot and storage-accounting exports. - Addressed the blocking review using its accepted alternative: rename `max_bytes` to `max_event_array_bytes` and explicitly document that success-envelope metadata is outside the array budget. No auth, query, runtime-limit, or client-behavior redesign. - Added a PostgreSQL/router regression proving discovery, exact 8,388,608-byte array success with a larger envelope, and one-byte-over rejection (413, no partial complete response). Exact-head validation: all seven snapshot PostgreSQL/router tests passed in an isolated local database; both affected crates passed all-target Clippy; mandatory pre-commit/pre-push gates passed without bypass (Rust unit lanes, Tauri default/mesh Clippy and tests, differential file size, branch-skew/ref checks). Reverting only the descriptor rename made the new regression fail before the final commit. Independent bounded source-delta review by Mordecai found no blockers; execution was Carl's. A broader `cargo test -p buzz-db -p buzz-relay --all-targets` run on the merged candidate passed 131 DB tests and 1,057 relay tests, then encountered the same unrelated `mesh_demo::demo_join_forwarded_arm_round_trips_echo` 504 recorded in the prior review. Full repository `just ci` / `just test` were not rerun locally; hosted CI remains a separate gate. No deployment or production traffic was exercised. --------- Signed-off-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz> Signed-off-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> |
||
|
Will Pfleger
|
209536ade6 |
docs(nips): add single-coordinate manual-unread override layer and verification model to NIP-RS (#2864)
## Summary
Amends `docs/nips/NIP-RS.md` with the manual mark-as-unread override
layer and includes `docs/formal/nip-rs-unread/`, the bounded exhaustive
verification model that preceded and informed the spec.
All `ov_*` override state lives in exactly one coordinate per
installation. That single constraint is what makes the rest of the
amendment small: override state never moves between coordinates, so
there is no slot lifecycle to make crash-safe, and the only durability
obligation is carry-forward on `client_id` rotation.
## Spec changes (`docs/nips/NIP-RS.md`)
- **Non-Goals:** drop the stale line stating mark-as-unread is out of
scope; state the `ov_*` durability exception to the
best-effort/time-horizon model.
- **Reserved Namespace:** `ov_` stem and `esc:` escape marker reserved.
Escape on publish (prepend `esc:` to raw IDs beginning with `ov_` or
`esc:`), unescape on receive (strip exactly one `esc:`). Bijection, with
the pre-amendment backward-compat residual documented as a stated
limitation.
- **Content Validation:** override entries are collected and validated
as a complete logical group *before* any decoding, zero-filling,
merging, or canonicalizing. Only two wire shapes are accepted — a
complete live three-key group, or an `ov_c:`-only tombstone floor. Any
other shape rejects the whole group while retaining the frontier entry;
applying the generic per-entry discard rule first is prohibited.
- **`d` Tag:** `<slot-id>` is exactly 32 lowercase hexadecimal
characters, replacing "a random opaque string" of 1–64 ASCII characters.
The fixed shape lets a relay recognize a read-state coordinate
structurally from the `d` tag alone, without decrypting anything, and
apply per-coordinate protections to it — under the old wording a
conforming client could pick a shape that silently forfeits them.
Recognizable coordinates are also what let a relay replace superseded
versions outright rather than accumulating one retained row per publish,
which keeps the coordinate count a full-state load must enumerate near
one per installation. Every client designates one **primary** coordinate
with a stable `<slot-id>` for the installation's lifetime. All `ov_*`
entries, and the frontier entries of the contexts they belong to, MUST
live in the primary. Additional coordinates remain legal for frontier
volume but MUST NOT carry `ov_*`, which keeps them freely rewritable and
freely deletable.
- **`t` Tag:** described as a discoverability marker rather than a
guarantee of relay-side selectivity. A relay MAY apply tag constraints
after its result cap, and `kind:30078` is shared with unrelated
application data, so clients MUST apply the tag as a correctness filter
locally, MUST NOT infer completeness from a short result, and MUST omit
the tag entirely when performing a full-state load.
- **Fetching / Full-State Load:** clients implementing the override
layer MUST NOT apply a finite `since` filter — an encrypted payload
means a relay filter cannot select for override-bearing events, so any
event-level window can exclude the only coordinate holding a tombstone
floor. Removing `since` is not sufficient: relays MAY cap historical
results, MAY cap below the requested `limit`, and emit
end-of-stored-events after the capped query, so neither EOSE nor a short
page proves completeness. No test against the client's requested `limit`
can detect truncation either: the effective cap belongs to the relay, a
relay MAY cap below what was requested, and an advertised maximum limit
is not necessarily the limit enforced.
A full-state load is therefore enumerated on `{"kinds": [30078],
"authors": [<pubkey>], "limit": <n>}` with **no tag constraint**. A
relay MAY apply tag constraints only after its result cap and withhold
the events that fail them, so under a tag-constrained filter the
delivered count is not the count the cap selected — a delivered page can
be empty while older coordinates still exist below it, and `kind:30078`
is arbitrary application data whose `d` tag namespace is open to every
application that has written under the user's key. Omitting the tag
makes delivery observable; read-state selection moves client-side, where
the validation rules already place it.
Completeness is then established by enumeration on a strictly decreasing
cursor: collect a page, descend on the lowest `created_at` across all
delivered events, exhaust that second with a window pinned to it,
continue below it, and treat only an empty delivery as complete. Every
query carries the same explicit `limit` `n` with `n >= L`. Per-second
exhaustion is discharged by comparing the pinned window's delivery
against the largest delivery the relay has already demonstrated in the
same load, floored at `L = 2` so that the ordinary single-coordinate
installation can reach *complete* at all. The comparison fails safe: an
inconclusive window reports *cannot prove complete* rather than
*complete*, and that verdict is terminal for the load.
Because these are addressable events, a coordinate republished mid-load
moves *above* the descending cursor while its previous version stops
existing, so neither is reachable by any later query. A full-state load
is therefore fenced by a live subscription on the same tag-free filter,
established — defined as receipt of end-of-stored-events — before the
first enumeration query and held unbroken on the same connection for the
load's duration. Fence deliveries are collected like enumerated events
but do not contribute to the cursor or to the demonstrated-delivery
bound. Collection deduplicates coordinates on the full NIP-01
addressable ordering — greatest `created_at`, lowest event id on ties —
because an equal-timestamp replacement is legal and is the version the
relay retains. A lapsed or reconnected fence makes the load potentially
incomplete, and a client MUST NOT publish to its own coordinates during
its own load.
Five relay behaviours the *complete* verdict rests on are stated as
normative conformance preconditions rather than assumptions, because
none is verifiable from the responses a client receives: newest-first
prefix delivery with lowest-id tie-breaking (what NIP-01 already
specifies for `limit`), a non-decreasing effective cap within a load,
the floor `L`, push delivery on an open subscription, and a delivery
barrier ordering accepted matching events ahead of a query's
end-of-stored-events on the same connection. Conditioning *complete* on
positive proof of these instead would withdraw the override layer from
every client rather than from the non-conforming relays. A client MUST
NOT load against a relay it has evidence violates them, and MUST treat
any such load as potentially incomplete.
A load that is potentially incomplete, or that failed on any relay the
client publishes to, MUST NOT authorize canonical compaction, publishing
a canonicalized override blob, deleting or abandoning a coordinate, or
reporting a mark-read as successful; the client falls back to local
state.
- **Client-ID Rotation / Orphaned Blob Deletion:** rotation is the only
event that changes an override-bearing coordinate. Before deleting or
abandoning its previous primary, a client MUST republish the
componentwise `max()` of every register that primary holds — every
tombstone ceiling included — under its new primary, and MUST confirm
acceptance on **every relay** from which the old primary will be deleted
or allowed to lapse. Acceptance on one relay does not authorize deletion
on another. Frontier-only orphans are deletable unconditionally; an
unknown same-`client_id` coordinate is treated as a live carrier until
merged.
- **Live Subscription and Convergence:** the re-publish trigger and its
suppression are evaluated on canonicalized state, so a retained live
peer blob the client has already tombstoned cannot trigger an identical
write on every replay.
- **Manual-Unread Override Layer** (new section):
- **Wire encoding:** `ov_s:<ctx>`, `ov_c:<ctx>`, `ov_b:<ctx>` as uint32
siblings in the existing `contexts` map.
- **Merge rule:** componentwise `max()` per counter — no new wire merge
logic.
- **Liveness predicate:** `S > 0 AND F <= B AND S > C`, transcribed from
`model.py::override_set_b`.
- **Actions:** mark-unread bumps S and captures the effective frontier
as B; mark-read bumps C; a natural frontier advance past B deactivates a
stale set with no counter update. Every action requires a complete
full-state load. At the uint32 ceiling, wrapping and resetting are
prohibited: mark-unread is refused, and mark-read completes only if the
resulting state has `override_active == false` — otherwise it fails
visibly rather than reporting success over a still-live override.
- **Tombstone floor:** a dead ever-active register compacts to `RegB(0,
max(S,C), 0)` — a single `ov_c:` key. A virgin register is omitted
entirely. This blocks counter reuse and the resulting resurrection.
- **Mandatory canonical publication:** a protocol requirement, not an
optimization. Publishing raw dead registers lets two independently-dead
registers from different devices produce a live join.
- **Override group co-location rule:** a context's frontier entry and
all its `ov_*` siblings MUST travel in the same event, and that event
MUST be the primary coordinate. An override-bearing context therefore
has exactly one legal destination for its whole group; only
frontier-only groups may be distributed across additional coordinates.
Grouping is per logical context, never per key.
- **Unescape-before-group rule:** the frontier wire key MUST be
unescaped to its raw logical context ID before use as group identity.
Equal normative weight to atomic grouping.
- **Tie policy:** clear-wins is MUST. The tie verdict is not encoded on
the wire, so a selectable policy makes two conforming clients diverge
permanently on both the unread verdict and the canonical wire form.
- **Override State Durability:** `ov_*` entries are exempt from age
pruning and budget eviction permanently, and durability is defined over
retrievable logical state — the containing event must stay reachable and
the load must establish completeness, not merely retain keys. There is
no safe finite GC horizon.
- **Bounds and budget:** byte/key analysis at both small-counter and
uint32-maximum values. Confining `ov_*` to one blob makes its plaintext
budget a hard lifetime ceiling on ever-overridden contexts — roughly 600
tombstones at the worst-case ~54 bytes against 32 KiB, ~730 at the
common ~45 bytes, ~199 simultaneously live overrides at ~164 bytes. At
the ceiling a client MUST refuse mark-unread and MUST NOT split override
state, drop floors, or publish a truncated override set. Same policy
shape as counter exhaustion: visible failure, never silent degradation.
- **Verification artifact:** `docs/formal/nip-rs-unread/`. The model is
a broader predecessor of this NIP: its `split_blob_into_slots` permits
override groups in any slot, so verified atomicity covers every
arrangement this NIP allows, but the converse does not follow. The model
does not verify the single-primary rule, the completeness procedure, the
relay conformance requirements or the mutation fence, or carry-forward;
malformed-group wire validation is likewise normative but outside
verified scope.
- **Abstract / Non-Goals / Backwards Compatibility:** the absolute "no
relay-side logic" and "no relay behavior changes" claims are narrowed to
what remains true — no new event kind, no new wire message, no
relay-stored read-state logic — with the override layer's relay
conformance contract named as the exception. Frontier sync and clients
that skip the override layer are unaffected on any relay.
## Verification model (`docs/formal/nip-rs-unread/`)
Four Python files constituting a bounded exhaustive verification model
for the override layer's register algebra.
**What it does:** constructs a toy universe — 2–3 devices, 2 channels,
every action that can happen (mark-unread, mark-read, late/duplicate
syncs, app reinstall, storage compaction) — and brute-forces every
reachable ordering (14,258 BFS states; 672-point deep-history parameter
cube; 9-mutant harness over ~45,000 merge pairs). After each world-state
it asks: did all devices converge? Did any unread flag get resurrected
after being cleared, or vanish while live?
**What it found and fixed:**
1. **Killed candidate A.** The model produced a concrete kill sequence:
an old client that doesn't know about the new field rewrites its
read-state blob and silently erases unread flags. That witness is why
the spec uses candidate B (two counters that only count up, plus a
snapshot) instead.
2. **Candidate B passes everything.** All delivery orders converge; the
frontier high-water mark never regresses; duplicated/replayed syncs are
harmless; old clients can't destroy it; compaction never resurrects a
dead unread or drops a live one, including
cleanup-followed-by-weeks-late-stale-sync and
tombstone-landing-on-unrelated-live-state corner cases.
3. **Caught a second real bug late.** Two devices each publishing "this
unread is cleared" could, on merge, reactivate it. The fix (canonicalize
before publishing) is a mandatory rule in the spec; the model re-checks
it across ~45,000 merge pairs.
**Scope and caveats:** bounded to 2–3 devices and 2 channels. Can't
prove the infinite case. `NOTE.md` documents the exact verification
scope and the gap between the model's `split_blob_into_slots` generality
and the single-primary rule the spec adds on top.
**Why it's in the repo:** the spec asserts "verified by bounded
exhaustive model checking." Keeping the artifact in-repo means anyone
who later amends the merge/compaction rules can `python3 exhaustive.py
&& python3 mutation.py` (deterministic, exit 0) and confirm the
guarantees hold. Without it the spec claims a proof nobody can check.
## Diff scope
`docs/nips/NIP-RS.md` — spec amendment, zero product code.
`docs/formal/nip-rs-unread/{NOTE.md,model.py,exhaustive.py,mutation.py}`
— bounded exhaustive verification model, zero product code.
`.gitignore` — `__pycache__/` and `*.pyc` entries for the model
directory.
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
|
||
|
Wes
|
733c766eba |
fix(desktop): clear unread projections from message read markers (#1242)
Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Pinky <44b8e82baa6e0e254e0208d68f335c283c94e7b78dd1fa10d5a49d3f13dd0435@sprout-oss.stage.blox.sqprod.co> |
||
|
Will Pfleger
|
2a2c1c8001 |
feat(desktop): in-channel and in-thread unread indicators (#1008)
Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@sprout-oss.stage.blox.sqprod.co> |
||
|
Will Pfleger
|
43d1ce3532 |
docs(nip-rs): add optional thread read context scheme (#1006)
Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub16v54tttfqacx9ycvc3k0ut0npj564ahcuajzy6qjvh57ntmsf4uq4806j2 <d32955ad69077062930cc46cfe2df30ca9aaf6f8e76422681265e9e9af704d78@sprout-oss.stage.blox.sqprod.co> |
||
|
Wes
|
e0fdeb2ff5 | [codex] clean up root docs and assets (#445) |