jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 04:30:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,706 Commits 1,194 Branches 231 Tags
4544ba1713d8d7f5959d7d63886ff3927357e1ac
Commit Graph

102 Commits

Author SHA1 Message Date
Kalvin C
5a349db91c feat(relay): add opt-in newest-first thread windows (#7823) 
## Summary

Add an opt-in, newest-first thread window to the authenticated HTTP
query path.

A filter with `thread_window: true` returns:

- reply rows ordered by `(created_at DESC, id ASC)` with an exact
composite cursor;
- optional bounded reactions, edits, deletions, and delete-of-aux
closure;
- one relay-signed `kind:39007` bounds event bound to the normalized
host, authenticated reader, channel, root, request cursor, kinds, depth,
limit, and aux choice.

Filters without the flag keep the existing oldest-first thread behavior.
Channel windows and `kind:39006` are unchanged. No client opts into this
mode in this PR.

### Why extend NIP-CW

NIP-CW already defines relay-computed, cursor-paged conversation views
over extended NIP-01 filters. Thread windows use the same protocol
ideas: ordinary signed rows, composite keyset pagination, optional aux
closure, and relay-signed bounds. Adding a thread mode keeps those
related wire contracts together instead of creating a second NIP with
duplicated terminology and compatibility rules.

Extending a draft, optional NIP with an additive mode is normal protocol
evolution. Existing fields and meanings are not reinterpreted:

- `top_level: true` remains channel mode;
- `kind:39006` remains channel-window bounds;
- `thread_window: true` selects the new mode;
- `kind:39007` has a distinct identity and root/request binding;
- extension-unaware clients never send the selector;
- legacy thread queries remain unchanged.

NIP-CW documents only the wire contract and relay semantics. Database
plans, replica internals, and rollout procedures remain implementation
concerns.

### Why thread windows use independent modules

The implementation deliberately keeps `thread_window.rs` separate in
`buzz-core`, `buzz-db`, and `buzz-relay` rather than adding branches
throughout the existing channel-window path.

The modes share pagination concepts, but their invariants differ:

- channel mode selects top-level rows; thread mode requires one root and
depths `1..=depth_limit`;
- thread mode accepts only conversation row kinds and rejects
unknown/conflicting fields;
- thread bounds bind the root and authenticated reader because
cross-channel aux visibility can differ by reader;
- thread aux includes the root and requires complete two-hop closure;
- thread mode refreshes writer authorization before selection and before
signing;
- historical cursor pages may use a proved replica transaction, but
fallback must restart both aux hops on the writer;
- legacy thread traversal is oldest-first and must not inherit
descending cursor semantics.

Forcing these rules into channel-window code would create a mode flag
across parsing, SQL predicates, bounds construction, authorization,
replica fallback, and tests. That abstraction would not remove
complexity. It would distribute it. Separate modules keep each contract
local and leave the established channel and legacy paths untouched.

### Technical design

- Strict request parsing normalizes UUIDs, event IDs, kinds, defaults,
and cursors before computing the response binding.
- SQL applies community, root, channel, depth, kind, deletion, and
cursor predicates before `limit + 1`.
- Ordered metadata scanning plus exact event-key lookup avoids the
stale-statistics plan that previously exceeded the four-second SQL
budget on a 10k-reply thread.
- Aux work is bounded by target batches, query count, raw candidate
count, response bytes, SQL timeouts, and one shared HTTP deadline.
- Fresh authorization-set comparison catches both grants and revocations
during closure. The relay returns retryable `503` rather than signing an
incomplete page.
- Replica failure permanently degrades the request to the writer and
restarts closure from its original targets without resetting budgets.
- Migration `0049_thread_window_index.sql` adds the mixed-direction
index used by newest-first scans. It validates a prebuilt index and
bounds startup lock/build time.

### Tradeoffs

- The stale-statistics-safe plan is intentionally less clever. Measured
warm 100k deep-cursor execution increased from roughly 0.30 ms to 4.80
ms, plus about 8 ms planning, in exchange for avoiding a reproduced
multi-second timeout after rapid 10k-thread insertion.
- Bounds are authoritative only for the served page. There is no
cross-page snapshot guarantee, and replica insertion coverage does not
imply latest edit/deletion/aux visibility.
- Strict parsing rejects unsupported constraints instead of silently
serving a different row set.
- Required aux closure fails closed. The relay does not omit aux and
sign plausible-but-incomplete bounds.
- This PR ships relay capability only. Client rendering, retry UX,
live-history overlap, and compatibility fallback remain separate work.

### Related issue

No existing issue found. This work follows the thread-loading design and
review discussion.

### Testing

- `cargo test -p buzz-core`: 258 unit tests and 2 doctests passed.
- `cargo check --workspace --all-targets`: passed at the rebased
implementation head.
- `cargo test -p buzz-db`: 131 non-PostgreSQL tests passed; 275
infrastructure tests were ignored as declared.
- Focused PostgreSQL migration/runtime regressions: 3/3 passed against
an isolated PostgreSQL container, including both corrected
migration-ledger tests.
- `cargo fmt --all --check`: passed.
- Pre-rebase feature validation passed 398/398 PostgreSQL tests,
signed-wire multi-tenant/auth/pagination/aux/revocation probes, clean
Compose boot/reboot, and full core tests.
- Independent HA validation exercised three relay pods and a physical
PostgreSQL standby on the same runtime logic before the final
documentation-only cleanup.
- Independent final-delta review confirmed exactly two changed files,
byte-identical executable migration SQL after comments/diagnostic
normalization, matching prebuild/catalog/recovery instructions, and `git
diff --check`; no runtime receipts were relabeled to the new SHA.
- The pre-push hook ran repository checks but its Rust unit lane exited
100 without exposing an individual failed test in the captured
multiplexed output. The independently run affected-package checks above
passed; current-head repository CI remains required.

Known unrelated baseline behavior remains: the mesh-demo echo test can
return 504, and same-second archive → unarchive can deduplicate its
membership notification. Neither path is changed here.

---------

Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Signed-off-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Co-authored-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
Co-authored-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
 2026-09-23 12:01:45 -07:00
Will Pfleger
b6a26556ec feat(canvas): add version history with atomic restore (#6780) 
🤖 Adds append-only channel canvas history, restore, and
relay-authoritative optimistic concurrency so stale edits cannot
silently replace a newer canvas.

## What this adds

- **Desktop history and restore** — browse revisions with author,
timestamp, preview, and diff; restore an older revision as a new signed
head after explicit confirmation. Persisted empty canvases remain
reachable, mutation outcomes are announced accessibly, and focus moves
to a stable destination after save or restore.
- **CLI history and restore** — `buzz canvas history --channel …`, `buzz
canvas get --channel … --revision <id>`, and `buzz canvas restore
--channel … --revision <id>` use keyset pagination over the retained
revision stream.
- **SDK canvas builders** — optional `expected-revision` tags, bounded
ancestry classification, and shared timestamp discipline keep
first-party writes ahead of the head they read without propagating
poisoned future timestamps.

## Authoritative concurrency contract

Tagged canvas writes use compare-and-swap semantics in the relay and
database:

- `expected-revision=none` succeeds only when no live canvas exists.
- An event ID succeeds only when it names the canonical live head.
- A byte-identical replay succeeds only when that event is already the
canonical live head.
- A successful candidate must sort strictly ahead of the current head
under `created_at DESC, id ASC`.
- Malformed preconditions and stale or non-advancing writes are rejected
before event rows, mentions, or fan-out are committed.

The database serializes tagged writes, unconditional kind `40100`
writes, and canvas soft deletion on the same `(community, kind,
channel)` advisory key. Untagged `buzz canvas set` remains an
unconditional write, but participates in that serialization boundary so
it cannot race a tagged save or deletion. Deletion classification is
derived inside the datastore transaction; unrelated event kinds do not
take the canvas lock.

## Read and timestamp consistency

Precondition and post-write ancestry reads request strong consistency
from `/query`, pinning them to the writer when replica routing is
enabled. Display-only history remains replica-eligible.

First-party canvas writers stamp `created_at = max(now, head + 1)` with
a conservative client skew limit. The relay applies a canvas-specific
future-timestamp ceiling before persistence, preventing an accepted
boundary head from making subsequent first-party writes invalid while
preserving the broader ingest limit for other event kinds.

## Durable outcomes and recovery

After an accepted publish, a failed verification read is reported as
durable success with `verified: false` rather than a failed save.
Desktop and CLI preserve the new event ID and direct users to History
when verification is unavailable. A successful read walks the bounded,
cycle-guarded `expected-revision` ancestry chain so transitive
descendants are not misclassified as supersession. Canvas caches
invalidate for every settled mutation outcome, including a retained but
superseded revision.

The 409 reconciliation path in `cmd_restore_canvas` now separately
establishes persistence evidence from ancestry classification.
`canvas_write_survived` checks reachability only; a secondary
writer-pinned IDs lookup distinguishes "A absent" from "A present but
superseded by a legacy write." The four-way outcome (survived /
persisted-but-superseded / genuinely-absent / read-fail) is now correct
for all lost-response shapes including unconditional legacy writes.

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
 2026-09-23 11:03:54 -04:00
Jordan Mecom
7881447c19 fix(audit): frame hash inputs with TLV (#7492) 
Hash new audit entries from a TLV buffer built with one-byte field tags,
big-endian u64 byte lengths, and raw values. Absent optional fields are
omitted; present empty values retain their tag and zero length. This
prevents bytes from moving between fields without restricting object IDs
or inferring boundaries from field contents.

Store an explicit `hash_version`. Migration 0048 labels existing rows as
version 1, and new writers use version 2. Historical hashes are
unchanged and mixed-version chains verify. The legacy encoding retains
its original ambiguity; TLV protects new entries. Apply the migration
before deploying the updated writers and verifiers.

Tests cover both boundary collisions, absent versus empty fields, stable
digests, unknown versions, mixed-version chains, and stored-row
tampering. All 16 audit and 128 database unit tests passed. The six
PostgreSQL audit tests passed against both migrated and fresh schemas,
and a populated-row migration check preserved the historical hash. Full
`just ci` passed.

Related audit fixes: #4216 and #4249; issue #4173.

Signed-off-by: Jordan Mecom <jm@squareup.com>
 2026-09-22 11:19:11 -07:00
Will Pfleger
77729abfb6 feat(relay): add admin HTTP routes for member restriction management (#7302) 
## What

Three admin HTTP routes under `GET|DELETE /api/admin/v1/members/...`:

- `GET
/members/restrictions?communityId=<uuid>[&limit={1-200}][&cursor=<token>]`
— lists active bans and timeouts with stable keyset pagination
- `DELETE /members/{pubkey}/ban?communityId=<uuid>` — lifts an active
ban; 409 if no active restriction
- `DELETE /members/{pubkey}/timeout?communityId=<uuid>` — lifts an
active timeout; 409 if no active restriction

## Auth

All three routes pass through the existing NIP-98 authorization gate.
Both DELETEs call `require_mutation_principal`; GET follows the
established read-authorization pattern.

## Atomicity

Both DELETE routes use transactional DB helpers
(`unban_member_with_audit` / `untimeout_member_with_audit`) that perform
the conditional lift UPDATE and the `moderation_actions` INSERT in one
SQL transaction. If the audit INSERT fails the lift rolls back; the
restriction stays active.

## Expired-ban predicate

`unban_member` WHERE clause requires `banned AND (ban_expires_at IS NULL
OR ban_expires_at > now())` — matches all read paths. An already-expired
ban returns `false` → 409.

## Pagination (GET /members/restrictions)

`GET /members/restrictions` returns `{ "items": [...], "nextCursor":
"<token>"|null }`.

- `limit` — page size 1–200 (default 200), enforced as a SQL `LIMIT`.
Returns 400 for values outside range.
- `cursor` — opaque continuation token from a prior page's `nextCursor`.
Omit for the first page.
- Order: `ORDER BY updated_at DESC, pubkey DESC`. The `pubkey`
tie-breaker ensures a deterministic cursor even when multiple rows share
the same `updated_at`.
- Cursor encoding: base64url of `{updated_at_micros}_{pubkey_hex}`.
Treat as opaque.
- `nextCursor` is `null` only when the returned page is smaller than
`limit`. An exactly-full final page emits a non-null cursor; the
subsequent request returns an empty page with `null`.
- The existing `list_community_restrictions` DB method (unbounded) is
kept intact for the embedded dashboard bridge consumer (`GET
/moderation/restricted`), whose bare-array response contract must not
change.

## Tests

All Postgres-gated tests are in the `postgres_tests` module
(`api/admin/mod.rs`):

- `list_restrictions_returns_active_bans_and_timeouts` — seeds a ban +
timeout, GET returns both; `nextCursor` is null when all records fit in
one page
- `list_restrictions_pagination_exhaustive` — seeds 201 rows: 199 with
distinct timestamps (1s apart, newest first) plus a tied pair sharing an
identical older timestamp (`now()-1000s`), pinned to sort positions
200/201 so the default-200 page boundary splits the tie. Asserts: (1)
`limit=201` → 400; (2) a no-`limit` request returns exactly 200 items +
non-null `nextCursor` (falsifiable binding of default=200 and SQL cap —
reverting the route default to the shared helper's 50 fails this
assertion); (3) continuation walk to exhaustion with exactly-once
coverage over all 201 pubkeys, exercising the tie-breaker at the page
boundary.
- `unban_member_returns_204_clears_ban_and_inserts_audit` — 204, ban
cleared, timeout survives (restriction independence), audit row
verified, tenant isolation confirmed
- `untimeout_member_returns_204_clears_timeout_and_inserts_audit` — 204,
timeout cleared, ban survives (restriction independence),
second-community timeout survives (community_id predicate), audit row
verified
- `unban_member_returns_409_for_expired_ban` — expired ban → 409, no
audit row
- `unban_with_audit_rolls_back_lift_when_audit_insert_fails` (buzz-db) —
forced CHECK constraint failure proves Err + ban still active (rollback
proof)

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
 2026-09-21 14:53:04 -04:00
Will Pfleger
ce41be9c4b fix(relay): fire kick live side effects at convergence; persist target; fence re-add race with held lock (#7298) 
## What

Ensures admin kick live side effects (subscription eviction + workflow
disable) fire correctly on every path — fresh HTTP, crash recovery, and
upgrade recovery from pre-migration rows — without a race window where a
concurrent legitimate re-add can be revoked.

### Finding 1 — Lock held through effects

Replace `verify_member_still_removed` (which acquired + released the
advisory lock before returning) with `membership_removal_fence`, a RAII
guard that keeps the `pg_advisory_xact_lock` alive until the caller
drops it.

Update `apply_kick_live_side_effects` to use the fence on every path
(fresh and recovery), holding it through eviction and workflow-disable.
Remove the `is_recovery` parameter entirely.

**The bug:** the old code released the lock before returning a bool; a
concurrent `add_member` could commit between the check and the effects,
silently evicting a re-added member.

**New guarantee:** any `add_member` either serializes before the fence
(fence observes `removed_at IS NULL`, skips effects) or waits until
after effects complete.

### Finding 2 — Persisted-context path scoped to kick only

Gate the persisted-context branch in `admin_action_worker.rs` on `action
== "kick"`. Non-kick actions (ban, timeout, delete) always re-derive
from the report.

**The bug:** any action with `enforcement_target_pubkey` set took the
persisted-context branch, which hard-coded `target_event_id = None`. A
stranded delete either failed pre-marker recovery ("delete requires
target_event_id") or finalized post-marker without the tombstone outbox
row.

### Finding 3 — Legacy kick row upgrade recovery (Carl's
CHANGES_REQUESTED)

Migration 0047 (renumbered from 0045 to avoid collision with two main
merges: 0045_retain_push_revocation_tombstones and
0046_storage_accounting_snapshots) adds
`enforcement_target_pubkey`/`enforcement_channel_id` without
backfilling.

**The bug:** the convergence gate read only the persisted columns and
rejected NULL/NULL rows as corrupt, causing every recovery retry to fail
forever. A cancellation cannot release a post-mutation action, so these
rows remain permanently stuck.

**Fix:** when persisted columns are absent, fall back to the function
parameters (which the recovery worker re-derives from the report row).
Error only when both sources are absent (genuinely unresolvable). This
preserves new-writer safety while allowing pre-migration rows to
converge.

An old writer in a rolling deployment can also create NULL rows after
the DDL applies; the same fallback handles these.

### Finding 4 — Test moved to PG lane


`kick_live_side_effects_clears_membership_cache_and_evicts_subscription`
was in the infra-free unit lane but called `membership_removal_fence`,
which requires a real Postgres connection. Moved to the PG fixture lane
(`#[ignore = "requires Postgres"]`) with a real member row so the fence
query sees the expected `removed_at` state.

## Tests

- **`crash_recovery_redrive_fires_kick_live_side_effects`** (existing):
re-drive via real recovery worker path, asserts all three side effects.
-
**`crash_recovery_after_readd_preserves_membership_subscriptions_and_workflows`**
(updated): uses real `add_member` (not direct SQL) — re-add completes
before fence is acquired.
- **`membership_removal_fence_blocks_concurrent_add_member`**
(existing): holds the fence explicitly, spawns `add_member`, asserts it
blocks until the fence is dropped.
- **`stranded_delete_pre_marker_recovers_via_worker`** (existing):
stranded delete (crash before mutation) converges via `recover_one` to
succeeded.
- **`legacy_kick_row_pre_marker_recovers_via_worker`** (new): old-writer
shaped row (NULL columns, no step_marker) converges through the full
state machine via recovery worker. Falsifiable: reverting the
convergence-gate fallback leaves the action stuck in enforcing.
- **`legacy_kick_row_post_marker_recovers_via_worker`** (new): same but
with step_marker set (post-marker crash). Falsifiable: same revert.
-
**`kick_live_side_effects_clears_membership_cache_and_evicts_subscription`**
(moved to PG lane): verifies cache eviction and subscription removal
against a real member row.

## CI

`Rust / Rust Lint` and `Windows Rust` failures in the prior run were
main-inherited (same lanes fail on main at the prior base). The branch
is now at main's current head.

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
 2026-09-21 14:52:54 -04:00
Wes
6a86531ac9 feat(relay): add atomic complete read-state snapshots (#7572) 
Opened by Brain on behalf of Wes (GitHub: wesbillman).

## Summary

Add an opt-in, host-scoped completeness primitive for NIP-RS read-state
clients. Ordinary query arrays and WebSocket behavior remain unchanged.

- Advertise a version-1 `read_state_snapshot` capability in NIP-11 only
when the request resolves a community.
- Accept exactly one self-author kind-30078 snapshot filter on
authenticated `POST /query`, after the existing admission, replay and
membership gates.
- Read all retained, nondeleted self-author kind-30078 coordinates from
the writer at one SQL-statement MVCC cut—including unrelated application
coordinates and events without a read-state tag.
- Return a complete versioned envelope bound to community, signer and a
content-identity hash. Reject malformed contracts,
reconstruction/signature/storage failures and whole-request overflow;
never return a truncated complete response.
- Bound snapshots at 4,096 events and 8 MiB of compact encoded
event-array bytes, advertised as `max_event_array_bytes`; envelope
metadata is additional. Retain the separate stored-payload preflight
budget.

## Safety and scope

This is point-in-time enumeration, **not** a live freshness guarantee,
cursor, CAS token, cross-relay transaction, or unread total. NIP-RS now
explicitly requires fresh complete enumeration from every write relay
before each override action, canonical publication, carry-forward or
deletion. It does not enable synchronized manual-unread overrides in any
client. No schema migration or deployment is included.

## Validation

Executed against the source committed as
`1fd5a5e0c0d893c1259386885517c0c44ae7630f`:

- Four real-router/PostgreSQL snapshot cases: beyond-page-cap
enumeration and overflow; corruption/storage failure; production
payload-bound NIP-98, Redis replay guard and membership enforcement;
host discovery/scope/strict request shape.
- Two database cases: replacement during preflight preserves a single
statement cut; encoded-byte overflow and writer-not-replica routing. The
cluster-wide probe uses the required `cluster_global_` serialization
prefix.
- Production-call-site mutations killed for dispatch, strict filter
validation, discovery tenant, snapshot tenant, encoded-byte guard,
membership enforcement and replay enforcement; isolated mutation tree
restored.
- Both affected crates pass all-target Clippy; formatting, differential
file-size and repository PostgreSQL test-discovery checks pass.
- Normal pre-push gates pass without bypass: branch skew, push-head
scope, file size, Rust unit lanes and desktop Tauri checks.
- Independent read-only review by Pinky found no remaining relay blocker
at this commit. The execution/mutation results above were run by Brain,
not duplicated by the reviewer.

Hosted CI and maintainer review remain separate gates. No production
traffic or deployment was used for these fixture tests.

Originating Buzz channel: `a394ecdb-9c67-4c4e-a3c6-4a95f69e8cc8`
(unread-but-good), thread
`2413d0ba7fa5b2ac5c777df6b3754e2099b042ec8fb96ca64302a0919a0550b3`.

## Review repair (2026-09-21)

Carl updated this existing branch to
`2f9c9b11266c24947412cf28cefae4e96e7167b7`, merging main
`5079c770fe30bb3d8204822ce6c2431eacac6d4b` without rewriting published
history.

- Resolved the DB export conflict, preserving both snapshot and
storage-accounting exports.
- Addressed the blocking review using its accepted alternative: rename
`max_bytes` to `max_event_array_bytes` and explicitly document that
success-envelope metadata is outside the array budget. No auth, query,
runtime-limit, or client-behavior redesign.
- Added a PostgreSQL/router regression proving discovery, exact
8,388,608-byte array success with a larger envelope, and one-byte-over
rejection (413, no partial complete response).

Exact-head validation: all seven snapshot PostgreSQL/router tests passed
in an isolated local database; both affected crates passed all-target
Clippy; mandatory pre-commit/pre-push gates passed without bypass (Rust
unit lanes, Tauri default/mesh Clippy and tests, differential file size,
branch-skew/ref checks). Reverting only the descriptor rename made the
new regression fail before the final commit. Independent bounded
source-delta review by Mordecai found no blockers; execution was Carl's.

A broader `cargo test -p buzz-db -p buzz-relay --all-targets` run on the
merged candidate passed 131 DB tests and 1,057 relay tests, then
encountered the same unrelated
`mesh_demo::demo_join_forwarded_arm_round_trips_echo` 504 recorded in
the prior review. Full repository `just ci` / `just test` were not rerun
locally; hosted CI remains a separate gate. No deployment or production
traffic was exercised.

---------

Signed-off-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
Signed-off-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz>
Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz>
 2026-09-21 12:30:03 -06:00
Tom Brow
a61239f0d8 fix(mobile): avoid opening empty threads on message tap (#7756) 
Tapping a mobile channel message with no replies currently opens an
empty thread. Only enable message-row thread navigation when the
existing thread summary reports replies, including replies known to the
relay but not loaded locally. Long-press → Reply still starts a new
thread.

Added widget tests for zero replies, loaded replies, relay-only replies,
and starting a thread through Reply. The zero-reply test failed before
the fix and passes afterward.

Validation: all 227 channel-page tests, the full mobile pre-push checks,
and all PR CI checks passed. The full repository gate failed in
unchanged Rust ACP timing tests (idle reset, keepalive, and steering
deadlines); this PR changes only mobile Dart files.

No duplicate open PR or issue found.

---------

Signed-off-by: Tom Brow <tomb@block.xyz>
 2026-09-21 08:51:05 -07:00
Wes
5079c770fe fix(workflows): make deletion persistent and retryable (#7735) 
## Summary

Fix workflow deletion leaving its definition queryable, and make public
deletion acceptance agree atomically with executable/definition state.

- Delete the executable workflow and soft-delete its definition under
the existing coordinate lock, in the same transaction as kind-5
acceptance and mention indexing. Failed deletion leaves no public
accepted event; identical retries can succeed. Legacy already-stored
failed requests still repair.
- Choose one committed dispatch/audit owner for concurrent identical
requests. Completed duplicates remain silent. Preserve owner/community
isolation, newer revisions, orphan cleanup, and intentional backdated
recreation.
- Align Desktop confirmation and architecture documentation with
recreation semantics. Add deterministic PostgreSQL, live WebSocket, and
Desktop regressions. No schema migration or new retry/delivery
subsystem.

## Current validation

Head: `40d86ddab0853ffb292316653b22c3fd4f983f0c`.

- **Exact clean head:** live relay workflow-deletion E2E **4/4 passed**.
All applicable normal pre-push hooks passed: Rust unit suites, full
Desktop unit suite, Desktop lint/typecheck, Tauri clippy/tests, and
file-size/branch checks. No hook bypass.
- **Pre-commit implementation tree, then committed at this head:** full
discovered DB/relay PostgreSQL lane **351/351 passed**; built Desktop
deletion Playwright smoke **3/3 passed** (success, failure preserving
editor/draft, corrected confirmation copy).
- **Deterministic mutation evidence:** unchanged previous relay
`659b79282` fails both new regressions: concurrent identical requests
produce two normal/dispatch-owner responses, and rejected deletion
leaves one public event. Both controls pass. Fixed relay passes all
four.
- **Independent source review:** transaction/authorization/lock-order
and regression/cleanup lanes clear. Reviewers did not run tests
independently.

## Honest limits / merge gates

The broad pre-commit DB/relay/test-client selection passed
**1,215/1,216**; the untouched mesh-demo echo test returned HTTP 504
instead of 200. It remains outside this repair and is not counted green.
Initial hook attempts also exposed an untouched Desktop inventory
ordering assertion and unidentified ACP/agent test failures; standalone
package reruns and the final complete hooks passed without changing
those packages. Causes are not established.

The previous Desktop CI shard failed before tests on a Hermit `just`
download (HTTP 500). New-head CI/security review and reviewer
disposition remain required. No deployment, fresh interactive Desktop
session, full local `just ci`, or multi-replica runtime validation
claimed. Postgres/Redis validation used task-owned local containers,
fresh identities, and disposable fixtures.

## Compatibility

No automatic cleanup of existing workflows: repeating deletion repairs
definitions left by the old path. Canonical UUID coordinates and legacy
name fallback (`LIMIT 1` for duplicate names) are unchanged. Backdated
recreation remains intentional per [Tyler’s
decision](https://github.com/block/buzz/pull/7735#issuecomment-5729881295),
not an irreversible tombstone. Mention-index failure now rolls back the
complete deletion rather than best-effort accepting partial work.

Detailed reviewer disposition and evidence:
https://github.com/block/buzz/pull/7735#issuecomment-5737367825

---------

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz>
Co-authored-by: Mordecai <f314b6033b6e13df94c8e9e1187d3666b6b9ce54e66ee986e31cdaa909e8dcbc@buzz.block.builderlab.xyz>
 2026-09-21 08:37:14 -06:00
ravarora2
4c2086c084 feat(db): expose connection setup metrics (#7286) 
## Why

Current pool metrics show checkout outcomes and pool state after the
fact. They do not show when a checkout began or which writer-connection
setup step failed. During startup and pool growth, operators need to
distinguish pool saturation from a slow or unsafe connection setup.

## What

- Count every instrumented database checkout when it starts, with a
fixed operation label.
- Measure writer-pool creation and every physical writer connection
across `physical_connect`, `created_at_floor`, `session_timeouts`,
`isolation`, and `ready`.
- Publish fixed-cardinality start, terminal-outcome, duration, and
waiter metrics without database URLs, SQL, raw errors, connection
ordinals, or per-connection lifecycle logs.
- Document how to identify the active setup bottleneck during a rollout
or incident.

## How

The existing typed checkout wrapper records starts and current waiters.
The huddle-history path uses that wrapper instead of a raw pool
checkout.

The production SQLx `after_connect` hook records start, terminal
outcome, and duration for each sequential safety step. A drop guard
records cancellation once if setup exits before a terminal result. Fixed
enums keep every label bounded.

For a phase, `started_total - sum(attempts_total)` is the number of
in-progress attempts on that pod. Because setup is sequential, a later
phase starting also proves the earlier phases succeeded.

The database path emits metrics only. This PR intentionally does not add
per-connection lifecycle receipts or direct `stderr` writes.

## Risk

Medium. This changes the production writer-pool `after_connect` hook and
adds one counter update to instrumented checkout paths. It does not
change the database safety statements or their failure behavior. The
telemetry has fixed labels and no per-connection identifiers.

## Testing

At exact head `ffb5fcb114a9c986ffd0cb2cfc9d413a32d5b1aa`, a release
relay was started against isolated local PostgreSQL, Redis, and MinIO
services. The main health endpoint returned `ok`, readiness returned
`{"status":"ready"}`, the connection-step metrics were exported, and no
database lifecycle receipts were emitted. A release CLI then created a
channel, sent a message, and read the same event back successfully.

The earlier staging deployment used pre-rebase head
`f58e9480a4f068db0c591f604fd6800fdd4bfc45`. The deployed
multi-architecture image came from [GitHub Actions run
33780888255](https://github.com/block/buzz/actions/runs/33780888255),
manifest
`sha256:b9351fa644e08376cbe1999f9bee311d33d1799a68eadef7929c4f862a832fec`.

The [staging
deployment](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/317)
brought both pods in ReplicaSet `buzz-6c8758bd7d` to Ready with zero
restarts, and the connection metrics produced data in the [rollout
dashboard](https://app.datadoghq.com/dashboard/tm7-qxr-wt2/buzz-startup--rollout-safety).
That staging image predates the cleanup that removed per-connection
lifecycle logs; the metric schema and database safety statements are
unchanged.

## Verification

- `cargo test -p buzz-db -- --test-threads=1`: 128 tests passed across
the package and integration target; 255 opt-in tests remained ignored.
- Four opt-in production-path PostgreSQL regressions passed: initial
minimum connections, post-startup pool growth, isolation failure, and
session-timeout setup failure.
- All 35 relay media tests passed against the isolated PostgreSQL
database.
- Rust formatting, workspace clippy, desktop/Tauri clippy, web checks,
mobile analysis, security checks, and file-size policy checks passed.
- The repository-wide unit stage also exposed three unrelated existing
`buzz-acp` failures: one timing-sensitive keepalive test and two
environment-default tests.

## Bigger picture

This is the database-metrics part of the startup and rollout
observability work. The early-startup lifecycle foundation merged in
#7258, so the rebase removed that duplicate commit from this PR.

Originating discussion:
buzz://message?channel=6ac85131-70cd-4bda-a031-38d34114934e&id=fa2bed181c092697210a60bb6eedc55a665d5c1c6cabc1413a04686647011f71

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
 2026-09-17 11:26:22 -04:00
ravarora2
6b1e45d2f6 Isolate S3 storage metrics from the relay (#7543) 
## Why

**This PR moves the S3 storage scan out of the serving relay and into a
separate batch job.** The job saves a complete result in PostgreSQL. The
relay reads that result, exports the existing storage totals, and
reports the snapshot's freshness and database-read health.

```text
Today:         relay -> scan S3 -> calculate totals -> Datadog metrics
After rollout: CronJob -> scan S3 -> PostgreSQL snapshot -> relay -> Datadog metrics
```

The code in this PR is the new pipeline. It adds the worker, database
snapshot, relay read mode, and Helm CronJob template. It does **not**
enable the pipeline in production. The chart keeps the CronJob disabled,
and the relay keeps its current inline mode by default.
[squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244)
will set the production limits, enable the job, and switch the relay to
the saved snapshot after this PR merges.

The relay currently lists the media bucket once an hour and calculates
storage totals inside the serving process. The production bucket now
exceeds the relay's one million object cap, so each sweep fails and the
last good per-community values remain stale. The Datadog dashboard can
no longer show current storage use by community.

Raising that cap inside the serving relay would move a larger,
memory-heavy scan into the process that owns live WebSocket connections.
The scan retains state for unique blob hashes and community bindings, so
its memory use grows with the bucket. A large scan could raise relay
memory use, trigger restarts, and drop user connections.

The goal is to restore current metrics without putting a
production-scale bucket scan in the relay's failure domain. No media
objects move or change. Only the process that calculates the totals and
the handoff to the relay change.

## What

- Add `buzz-admin storage-snapshot`, a run-once worker that scans S3 and
saves one complete accounting snapshot in PostgreSQL.
- Add `external` relay mode to read the saved snapshot. Keep the current
`inline` mode as the default, and add `off` as a fail-closed option.
- Add an opt-in Helm CronJob for the worker. The 10 million object cap
exists only on this Job.
- Keep the current fleet and per-community gauges, and add snapshot age,
generation duration, configured cap, cap utilization, and database-load
health gauges.
- Add an additive migration for the latest complete snapshot and its run
metadata.

## Overall plan

1. Merge this PR to add the worker, database handoff, relay modes, and
disabled-by-default chart support. This step does not change production
behavior.
2. Publish the relay image and Helm chart `0.1.9` from `main`.
3. Update
[squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244)
to pin those final artifacts and rerender its manifests.
4. Use the infrastructure PR to deploy the CronJob with a 10 million
object cap, a one-hour deadline, and a 20 GiB memory limit.
5. Start one manual Job during a staffed window and verify its snapshot,
duration, memory use, relay stability, and Datadog output.
6. Keep the daily schedule and `external` relay mode only after that
gate passes, then add alerts for stale snapshots, snapshot-load failure,
failed jobs, cap pressure, duration, and worker memory.

The code and infrastructure remain separate so this PR can ship the
mechanism without silently enabling a production-scale scan. The
infrastructure PR owns the production settings and rollout gate.

## How

### Complete snapshot handoff

The worker takes a deployment-wide PostgreSQL advisory lock before it
lists S3. This prevents scheduled and manual workers from overlapping,
even if Kubernetes starts both.

The final database replacement runs through that same lock-owning
session. If the session dies during the scan, publication fails instead
of letting the stale worker overwrite its successor. A worker that no
longer owns the lock cannot publish through another pooled connection.

The worker reads S3 in pages of 1,000 objects and folds each page into
physical totals, logical per-community usage, and anomaly counts. It
stops before folding a page that would exceed `--max-objects` and emits
structured progress every 100,000 objects.

The worker replaces the singleton database row only after the full scan
succeeds. A listing error, malformed page, cap breach, serialization
error, lost lock session, or database error cannot publish a partial
snapshot. The last completed snapshot stays available. Each stored
result includes its completion time, duration, cap, and image or source
revision.

PostgreSQL is the handoff between the batch worker and the relay. The
worker writes one complete deployment-wide snapshot, and the relay reads
that snapshot on its normal leader metrics tick. No new network service
or worker endpoint is required.

### Relay behavior

`BUZZ_STORAGE_METRICS=inline` keeps the existing in-process sweep and
its one million object default. `external` loads the newest completed
worker snapshot on each leader metrics tick. `off` emits no
storage-family metrics. An unknown value fails closed to `off`.

External mode republishes the existing fleet, per-community, and anomaly
metric names, so those current queries remain valid. Snapshot handoff
health uses a separate family: `buzz_storage_snapshot_age_seconds`,
`buzz_storage_snapshot_duration_seconds`,
`buzz_storage_snapshot_max_objects`,
`buzz_storage_snapshot_cap_utilization`, and
`buzz_storage_snapshot_load_ok`.

The `buzz_storage_sweep_*` gauges remain inline-only because they
describe a scan attempted by the relay process. External mode does not
synthesize `sweep_ok=1` from the last successful database row: an old
row proves only that an earlier worker completed. Kubernetes CronJob/Job
status is authoritative for the latest worker result, including non-zero
exit, deadline expiry, and OOM termination.

The relay keeps the last complete snapshot available when a worker run
fails. Snapshot age shows that no replacement arrived on schedule, while
`buzz_storage_snapshot_load_ok` distinguishes staleness from a missing
row, database read failure, or invalid payload. If a community
disappears, changes host, or leaves the configured emission scope, the
relay zeroes the old labeled series instead of leaving stale values
behind.

Per-community bytes are logical referenced media usage. If two
communities reference the same blob, each community receives the blob's
bytes. Fleet physical totals still count each S3 object once. This PR
preserves those existing metric semantics.

### Deployment boundary

The CronJob is disabled by default. When enabled, it uses
`concurrencyPolicy: Forbid`, no retry, a configurable active deadline,
bounded history, explicit CPU and memory resources, a read-only root
filesystem, and no Istio sidecar by default.

The worker receives only `DATABASE_URL`, S3 settings, optional S3
credential references, and the code revision. It does not receive the
relay private key, Git hook HMAC secret, Redis URL, or the full relay
secret as environment variables. Operators may reuse the Buzz service
account or select a dedicated one.

### Non-goals

- This PR does not enable the worker or `external` mode in production.
- This PR does not change media uploads, object ownership, retention, or
deletion.
- This PR does not change the names or meaning of the existing fleet,
per-community, or anomaly storage gauges.
- This PR does not add a lease service or allow multiple workers to
publish concurrently.

## Risk

The application and chart defaults do not enable the new worker or raise
the serving relay's cap. The database change is an additive
deployment-global singleton table. The relay's default remains `inline`,
so merging this PR alone does not change the active metrics path.

The main rollout risk is worker memory use because the fold retains
state for distinct blob hashes and community bindings. The first
production-sized run must confirm peak memory and duration before the
schedule is trusted. The 10 million cap and one-hour deadline bound the
first deployment, but they do not replace measurement.

External mode also needs stale-snapshot, snapshot-load, and job-failure
monitors because the relay intentionally keeps publishing the last
complete result after a failed job or read error. This behavior avoids
gaps and partial data, while snapshot age and load health tell operators
when the result or PostgreSQL handoff needs attention. Job status
remains necessary for immediate failures and termination reasons that a
killed process cannot record itself.

## Testing

No production S3 scan was run from this PR.

Automated regressions verify that external mode keeps stale last-good
totals visible without emitting inline attempt-health gauges, reports
database-load failure while retaining the cache, and leaves inline
telemetry unchanged. Mutation checks also prove those tests fail if
external mode is routed through inline emission or a failed load is
reported as healthy.

After deployment, create one manual Job from the CronJob during a
staffed window. Require a completed Job, exactly one new snapshot, fewer
objects than the configured cap, acceptable duration and peak memory, no
relay restart regression, restored Datadog points, and no OpenMetrics
sample truncation.

## Next steps

1. Complete review and merge this PR.
2. Publish chart `0.1.9` and the final main-branch relay image.
3. Replace the provisional pins in
[squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244),
rerender, and require clean CI.
4. Run the staffed manual Job gate and record object count, peak memory,
duration, snapshot revision, and relay stability.
5. Confirm the relay reads the new snapshot in `external` mode and the
existing `block.buzz_relay.*` storage series update again.
6. Enable the daily 03:17 UTC schedule and add dashboard panels and
monitors for snapshot age, snapshot-load health, Job failures, cap
pressure, duration, and worker memory.

If production must roll back to an older relay image, set
`BUZZ_STORAGE_METRICS=off` first. Older images do not understand
`external` and would resume the failing inline scan.

## Bigger picture

This is the application half of the storage-metrics repair. The paired
infrastructure PR owns the production resource limits, schedule,
artifact pins, secret boundary, and monitored rollout. Keeping those
changes separate lets this PR remain safe by default while still
providing a complete path to current per-community S3 totals.

Related issue: N/A. No matching issue or pull request exists in
`block/buzz`.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-17 10:34:31 -04:00
Luke Tornquist
213092074c Instrument database pool roles (#7356) 
## What

Name the relay's four Postgres pool roles—writer, reader, audit, and
search—and export one fixed-cardinality utilization metric contract for
all of them. Existing writer and reader gauges remain unchanged.

## Why

Audit and search already use separate pools, but they are missing from
pool pressure telemetry. Operators cannot see all process-level
connection demand before setting a deployment-wide connection budget.

## How

A closed `DbPoolRole` vocabulary now labels
`buzz_db_pool_connections{pool_role,state}` and
`buzz_db_pool_configured{pool_role}`. The relay retains cheap pool
clones for statistics only; ownership, capacities, timeouts, query
routing, and failure behavior do not change. The docs also replace the
stale async search-index description with the generated-column behavior
used today.

This simplifies four ad hoc observability states into one bounded model
without adding a pool manager or changing service boundaries.

## Risk

Low to moderate. This changes relay metrics and pool construction
plumbing, but not SQL execution or routing. Legacy metric names remain
available for current dashboards.

## Testing

- Push gate at `3f5468a0463bdfc5879bcdd3878fa06b42ca84c2`: all 14 Rust
test lanes passed; desktop Tauri checks passed.
- Blox: `cargo test --workspace --all-targets` with repository-declared
Postgres services.
- Blox: `cargo clippy --workspace --all-targets -- -D warnings`.
- Blox: `cargo fmt --all -- --check`.

## Bigger picture

Aggregate deployment budget enforcement remains deferred. This PR
exposes the per-role facts that a later deployment-policy change can
consume.

Generated with Claude Code

---------

Signed-off-by: tornquist <tornquist@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-09-16 15:37:12 +00:00
Jarrod Sibbison
c507a4d488 fix(db): allow naturally expired push lease replacement (#7653) 
## Summary

Allow a naturally expired push lease to be replaced by a newer lease for
the same author and installation.

## Failure mode

Expiry cleanup marked the old lease inactive but left active-only fields
populated. The push_leases_check constraint requires those fields to be
NULL when a lease is inactive. When the client later registered a
replacement lease, the cleanup ran before the replacement and violated
the constraint with SQLSTATE 23514. The transaction rolled back, so the
replacement failed instead of taking over the expired lease.

## Fix

Clear all active-only fields as part of natural-expiry cleanup,
preserving the existing constraint and replacement flow. No schema
change is required.

## Testing

- Passed the focused Postgres regression test: cargo test -p buzz-db
replacing_naturally_expired_lease_replaces_old_state -- --ignored
- Passed scripts/test-postgres-test-discovery.sh
- Ran just ci: formatting, lint, security, file-size, desktop, web,
mobile, and 12 unit-test lanes passed. The overall run was red only
because existing timing-sensitive buzz-acp tests failed.
- The pre-push test-unit hook reproduced the same unrelated buzz-acp
failures, so the already-run verification was documented and the branch
was published with that hook bypassed.

## Related issue

No matching open issue or pull request was found.

## Scope

One file changed: crates/buzz-db/src/store/push.rs. The test uses the
production accept_lease_event path and identifies the replacement by its
cryptographic source event ID and generation.

Signed-off-by: Jarrod Sibbison <jsibbison@squareup.com>
 2026-09-16 12:58:30 +10:00
Tom Brow
76f7e5ef0c feat(push): make gateway origin configurable (#7261) 
Make the push gateway origin configurable across the relay, gateway,
mobile client, and deployment chart.

This is a fresh deployment. The gateway has only run on personal
development infrastructure, and push has no existing enabled users.
Legacy migration, configured-to-unconfigured mobile transitions, and
rolling upgrades from pre-launch gateway binaries are not supported.

Populated pre-launch databases are rejected with an informative error
before migrations run; their data is untouched. Mobile builds may omit
the gateway URL, leaving push unavailable. Once push is enabled,
subsequent builds must retain that configuration.

### Validation

- All applicable CI checks pass at
`5f546cfaf403f0386f146d213adfb4392f4ec29c`, including Rust,
mobile/Swift, desktop/integration, security, image builds, and Helm
checks. An unrelated agent cancellation test passed on retry.
- PostgreSQL CI passed 384 tests, including the production-entry-point
regression for refusing populated pre-launch databases before
migrations, fresh initialization, and subsequent initialized
deployments.
- Focused local validation passed all 10 gateway PostgreSQL tests under
the CI discovery filter and all 3 unconfigured-mobile tests.
Repository-wide PostgreSQL discovery validation and the complete
pre-push hooks passed.
- Physical-device APNs/App Attest behavior and deployment to a live
cluster were not verified in this work.

---------

Signed-off-by: Tom Brow <tomb@block.xyz>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-15 14:56:45 -07:00
klopez4212
c6ca9d94d9 Show status and huddle indicators beside names (#7112) 
## Summary

- show status and huddle emoji beside names in DMs and message rows
- provide accessible tooltips, fallback status emoji, and profile-menu
icon replacement
- add the desktop status editor with preset durations, a ShadCN
calendar, and a capped half-hour time menu

## Validation

- desktop checks, typecheck, and file-size guard
- 5,802 desktop tests
- focused Playwright coverage (3 passed)
- E2E build and native Builderlab staging verification

Updated visual snapshots are attached in the PR comments.

---------

Signed-off-by: kenny lopez <klopez4212@gmail.com>
Signed-off-by: Princess Donut <b238ea756dee4d98afa5883fc7f1de61eeabe65bf700e3a5a5a80db5e42e2c2b@buzz.block.builderlab.xyz>
Co-authored-by: Princess Donut <b238ea756dee4d98afa5883fc7f1de61eeabe65bf700e3a5a5a80db5e42e2c2b@buzz.block.builderlab.xyz>
 2026-09-03 09:39:37 +00:00
ravarora2
91ab9d31b8 Add operation-aware database pool acquisition metrics (#7195) 
## Why

Buzz already reports coarse writer/reader database checkout waits, but
those
signals cannot explain which startup or serving operation is blocked by
pool
pressure. That makes rollout diagnosis and postmortems ambiguous: a
readiness
probe, NIP-42 authentication, authorization check, reconnect history
repair,
event write, and background maintenance can all wait on the same pool
while
appearing identical.

This PR implements Package 2A of the pod-handoff plan: operation-aware
pool
borrow causality. It is observability-only; it does not change
configured pool
sizes, SQL semantics, transaction ordering, or timeout policy. Physical
DNS/TCP/TLS/Postgres authentication and session initialization remain
the
separate Package 2B boundary.

## Metric contract

The final contract separates three questions:

| Question | Metric |
|---|---|
| How long did checkout wait? |
`buzz_db_pool_acquire_duration_seconds{pool_role,operation}` |
| How did the attempt end? |
`buzz_db_pool_acquire_attempts_total{pool_role,operation,outcome}` |
| Who is waiting now for a tracked operation? |
`buzz_db_pool_waiters{pool_role,operation}` |

Outcomes are `success`, `timeout`, `error`, and `cancelled`. Operations
are
`bootstrap`, `readiness`, `tenant_resolution`, `authentication`,
`authorization`, `subscription_history`, `event_write`, and
`maintenance`.

Only these eleven pool/operation pairs are constructible:

```text
writer/bootstrap                 reader/bootstrap
writer/readiness
writer/tenant_resolution
writer/authentication
writer/authorization             reader/authorization
writer/subscription_history      reader/subscription_history
writer/event_write
writer/maintenance
```

The duration histogram intentionally does not carry `outcome`. Result
data
remains available on the terminal counter for historical counts and
rates,
without multiplying the expensive histogram family. Nine finite buckets
plus
`+Inf`, sum, and count produce 12 duration series per valid pair.
Together with
four outcome counters and one waiter gauge, the new-family ceiling is
exactly
187 raw Prometheus series per pod, asserted from the production
exporter.

The existing coarse acquisition families remain temporarily for
dashboard
compatibility while the new series are validated in staging.

The operation-specific waiter family covers the explicitly routed,
deployment-critical operations above; it is not a census of every
possible
SQLx checkout in the process. The dashboard pairs it with SQLx pool
active/idle/max gauges for whole-pool capacity context, and treats a
missing
operation series as unknown rather than healthy zero.

## What changed

### Cancellation-safe acquisition ownership

- Add writer- and reader-specific typed operation APIs so invalid label
pairs
  cannot be constructed and store modules cannot emit reader labels.
- Own every polled acquisition with one RAII terminal guard.
- Record exactly one duration and terminal outcome for success, timeout,
error,
  or cancellation.
- Emit nothing for a future that is created but never polled.
- Balance the operation-specific waiter count exactly once on every
terminal or
  dropped future.
- Periodically refresh every expected waiter pair, including healthy
zero, so
missing telemetry is not presented as zero. Reader pairs are emitted
only
when a distinct read pool is configured; a writer-only pod cannot
fabricate
  healthy reader-zero state.

### Production attribution

Route the deployment-critical acquisition paths through caller-owned
semantic
entry points, including:

- writer and reader bootstrap;
- the real post-#7149 readiness acquisition and deletion-catalog
validation;
- tenant resolution and community lifecycle checks;
- NIP-42 allowlist authentication;
- membership, moderation, invite, operator, Git, agent-owner, and policy
  authorization;
- operator community create/list/archive/unarchive, reverse host/channel
tenant
  resolution, and REQ row-community conformance lookups;
- writer/reader subscription history, feed, thread, and routed fallback
paths;
- primary and command event writes, replaceable events, mention
indexing,
  reaction/channel/member/archive side effects, and thread metadata;
- push matching, usage rollups/leadership, replica-fence startup and
recurring
probes, periodic reconciliation, channel/deletion reapers, partitions,
and
  other bounded maintenance/bootstrap paths.

Shared helpers now accept caller-owned intent or expose named semantic
variants
instead of assigning one misleading operation to every caller. No known
P0 path
uses `other`.

### Readiness and size-one-pool correctness

- Rebase on the post-#7149 readiness implementation and instrument the
actual
`Db::readiness_check` acquisition rather than the superseded ping-only
seam.
- Acquire once for deletion-catalog validation and run its queries on
that
  connection, preserving the shared readiness deadline.
- Scope the channel-roster catalog checkout before the behavior probe so
a
  size-one writer pool cannot self-deadlock during startup verification.

### Exporter, documentation, and CI

- Register metric HELP/type/unit metadata through the production
Prometheus
  builder.
- Configure dedicated checkout buckets at 1ms, 5ms, 10ms, 25ms, 50ms,
150ms,
  500ms, 1s, and 3s.
- Add a production scrape-contract test for exact names, labels,
buckets,
  valid pairs, sensitive-label exclusion, and the 187-series ceiling.
- Add source mutation guards for the P0 semantic entry points and
raw-checkout
  bypasses.
- Add an exact backend-integration CI selector for the production
attribution,
  cancellation, readiness, and size-one-pool PostgreSQL tests.
- Document the frozen label vocabulary, valid combinations, semantics,
and
  cardinality budget in the Helm chart README.

## Dashboard intent

The new Stage 2 row in **Buzz Startup & Rollout Safety** is
deployment-first:

- baseline-versus-candidate attempts, failure rates, cancellation rates,
and
  maximum wait by operation;
- outcome counts and percentages over time by SHA/ReplicaSet;
- acquisition wait heatmap, average, and maximum through the rollout;
- historical waiter pressure beside writer active/idle/max context;
- per-pod postmortem drilldown, including terminated pods;
- a smaller current-waiter table with explicit stale/missing semantics.

Percentile widgets remain disabled until Datadog metadata confirms
percentile
support for the new distribution. Current gauges use no fill,
interpolation, or
`default_zero`; missing means unknown.

## Risk assessment

Moderate. The patch touches many database acquisition call sites, but
preserves
the selected physical pool and executes the same SQL on the acquired
connection. The main risks are incorrect semantic attribution,
cancellation
double-counting, and a helper accidentally acquiring twice. Typed APIs,
production-method PostgreSQL tests, source guards, the raw scrape
contract, and
the size-one-pool regression cover those risks.

No tenant, community, user, pubkey, event, channel, SQL, URL, pod,
version,
ReplicaSet, or request-controlled value is emitted as an application
metric
label. Deployment identity is supplied by infrastructure enrichment.

## Verification

- `cargo fmt --all -- --check` — passed.
- `cargo clippy -p buzz-db -p buzz-relay --all-targets --all-features --
-D warnings`
  — passed.
- `cargo test -p buzz-db` — 122 passed, 0 failed, 263 ignored;
  source-contract integration test: 3 passed, 0 failed.
- Focused relay compatibility, metric-contract, and readiness tests —
passed.
- `scripts/test-postgres-test-discovery.sh` — passed.
- Full `buzz-relay` package run from the identical tree reached 1,015
passes;
the six media-test failures all stopped in their shared local PostgreSQL
setup with `Sqlx(PoolTimedOut)` because Docker/PostgreSQL was
unavailable.
The same six failed in isolation, while every changed exact test passed.
- Exact implementation head:
  `f92910b353086e9edf85918ca5f72190edbbe22f`.
- Exact multi-architecture staging image:
  `dev-sha-f92910b353086e9edf85918ca5f72190edbbe22f-run-33607968668-1`

(`sha256:161712c8ed2e265a15df9b63e02248d5973481f875ff129d7d2ae78a09d487a2`).
- Focused staging GitOps PR:

<https://github.com/squareup/builderbot-platform-core-infrastructure/pull/299>
— merged after renderer, inventory, infrastructure test, Kargo, Semgrep,
and
Intersect gates passed; the source/generated-artifact diff was exactly
two
  image lines.
- Exact GitHub head reports 47 terminal checks: 35 successful and 12
  intentionally skipped. PostgreSQL, unit, lint, security, both server
cross-compiles, backend integration, relay E2E, desktop, mobile, image,
  Helm, Semgrep, zizmor, and DCO gates are green.
- Datadog readback identifies two exact-image pods,
  `buzz-d79c8d8f7-ckv2l` and `buzz-d79c8d8f7-qzqdp`, in ReplicaSet
  `buzz-d79c8d8f7`; both report the full source SHA above.
- Both pods report all eleven allowed pool/operation waiter pairs at
current
zero, with no invalid pair. The acceptance window observed nonzero
success
  receipts for readiness, tenant resolution, authorization, subscription
history, event write, and maintenance, and no timeout, error, or
cancelled
outcome. Maximum observed wait was about 101 ms for maintenance and 50
ms
  for reader subscription history.

The main **Buzz Startup & Rollout Safety** dashboard now has a live
Stage 2
database row with eight widgets and nineteen fully scoped queries. Final
readback preserved all seven top-level groups, found zero under-scoped
Row 6
queries, and confirmed the tracked-operation waiter boundary in the
panel
descriptions.

Generated with Codex.

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
 2026-09-02 17:04:34 -04:00
Will Pfleger
04babf0265 chore(db): drop Phase-A NIP-FI relay-side authority ledger (#7221) 
Removes the dead relay-side authority ledger introduced by migrations
0041 and 0042, and the dead `require_attested_key` verifier knob from
`buzz-auth`. Both are unreachable by design under NIP-FI spec v2 (#7214,
squash `d4420eb47`), which makes OSS Buzz stateless for identity: the
relay neither stores nor verifies an authority chain.

## What changes

**`migrations/0044_drop_nip_fi_ledger.sql`**
Drops all fifteen NIP-FI ledger tables and their trigger functions using
`CASCADE` to resolve the circular deferred FK between
`identity_bindings` and `identity_lifecycle_history`. Drops proceed in
FK dependency order: selectors → history/bindings →
enrollment_policies/receipts → parallel drop of auth tables. Restores
`community_write_fence_excluded_table` to its pre-0041 body (removes
NIP-FI table names from the exclusion array).

**`schema/schema.sql`**
Removes the NIP-FI section (~1885 lines of tables, functions, and
triggers) and updates `community_write_fence_excluded_table` to match.

**`crates/buzz-db/src/runtime/migration.rs`**
- Updates the `embedded_migrator_contains_consolidated_initial_schema`
sanity check: count 43→44, adds 0044 assertion block (verifies `DROP
TABLE` statements and absence of NIP-FI names from `schema.sql`).
- Removes ~2580 lines of NIP-FI Postgres integration tests (all
`#[tokio::test] #[ignore = "requires Postgres"]` from the 0041/0042
behavioral coverage).
- Removes the `extract_excluded_table_array` drift check (0042 body no
longer matches `schema.sql` by design).
- Adds `migration_0044_drops_populated_nip_fi_ledger_cleanly`: runs
migrations to 0042, seeds rows in `authorization_operation_receipts` and
`authorization_invalidation_domains`, then runs to 0044 and verifies all
fifteen NIP-FI tables are absent.

**`crates/buzz-auth/src/nip_fi/config.rs`**
Removes `require_attested_key: bool` from `IssuerPolicy` — field,
constructor parameter, accessor, and its contribution to
`derive_assertion_policy_id`.

**`crates/buzz-auth/src/nip_fi/verifier.rs`**
`parse_nostr_pubkey_claim` no longer takes a `policy` parameter. The
`None` (absent claim) arm now returns
`Err(VerifierError::ClaimRejected)` unconditionally instead of
conditionally on `policy.require_attested_key()`.

**`crates/buzz-auth/src/nip_fi/verifier/tests.rs`**
- Removes `missing_nostr_pubkey_denies_under_attested_key_policy` (the
sole `require_attested_key: true` call site).
- Removes `false,` from all eleven `IssuerPolicy::new` call sites.
- Injects `nostr_pubkey` by default in `mint_signed_by` (spec v2
requires it unconditionally).
- Updates `valid_access_token_verifies` to assert
`asserted_key().is_some()`.

**`crates/buzz-auth/src/nip_fi/startup/tests.rs` + `jwks/tests.rs`**
Removes `false,` from all `IssuerPolicy::new` call sites and adds
`nostr_pubkey` to all token-minting helpers.

## Verification

- Fresh-DB migration run to head: all migrations apply cleanly in
sequence.
- Populated-0041/0042-DB migration through 0044: seeds rows in live
NIP-FI tables, verifies all fifteen are dropped without error.

Closes the dead-code inventory item from the spec-v2 cleanup plan
(channel `48374f48`). Follows #7214.

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
 2026-09-02 13:56:41 -04:00
ravarora2
beb76406c1 feat(relay): add detailed readiness metrics (#7149) 
## Summary

This PR makes relay readiness failures diagnosable without weakening the
existing fail-closed readiness contract. It distinguishes Postgres pool
acquisition from query execution, Redis pool acquisition,
deletion-catalog validation, and the overall two-second deadline;
exports a bounded Prometheus contract for rollout dashboards; and fixes
the concurrency, shutdown, and listener-boundary semantics needed for
those signals to be trustworthy.

## Why

The previous `/_readiness` implementation exposed only an aggregate
ready/not-ready result. During a rollout, operators could not tell
whether a pod was blocked on:

- acquiring a Postgres connection;
- executing the Postgres readiness query;
- acquiring a Redis connection;
- validating the deletion catalog; or
- the shared readiness deadline.

Adding metrics to the existing handler also exposed three correctness
hazards that this PR resolves:

1. the same handler is mounted on both the public application listener
and the private Kubernetes health listener, so public requests could
otherwise distort rollout telemetry;
2. concurrent probes can finish out of order, allowing an older result
to overwrite newer current-state gauges; and
3. a probe started before SIGTERM can finish afterward, incorrectly
return `200 ready`, and resurrect ready gauges while the process is
draining.

## Behavior

### Readiness evaluation

- Postgres, Redis, and deletion-catalog checks still run under one
shared two-second deadline.
- Postgres distinguishes pool acquisition timeout/error from query
timeout/error.
- Redis distinguishes pool acquisition timeout/error. This does not
claim a Redis command round trip.
- The deletion catalog distinguishes operation timeout/error.
- Multiple failures are reported as `multiple_dependencies_failed`;
exhaustion of the shared deadline is reported as `overall_timeout` when
no more specific completed outcome wins.
- Readiness remains fail-closed: every dependency must succeed for `200
{"status":"ready"}`.

### Ordered publication and shutdown

`ReadinessCoordinator` is process-owned and uses one mutex as the
linearization point for probe generations, current-state publication,
and terminal shutdown.

- Every completed dependency attempt may contribute its truthful counter
and duration observation.
- Only the newest admissible probe generation may publish current-state
gauges.
- An older, slower probe cannot overwrite a newer probe's gauges.
- `begin_shutdown()` and probe commit serialize through the same
coordinator.
- Once shutdown commits, an in-flight probe cannot return ready or
publish ready/current dependency gauges, even if its dependency work
later succeeds.

Shutdown without dependency evaluation records only:

- `buzz_readiness_checks_total{reason="shutting_down"}`; and
- `buzz_readiness_state{check="overall"} = 0`.

It does **not** fabricate dependency failures, dependency state changes,
or zero-duration latency samples. If shutdown wins after an in-flight
evaluation actually ran, those completed dependency attempts may remain
as attempt telemetry, but they cannot overwrite shutdown-dominant
current state.

### Listener and response contract

- The private health listener's `/_readiness` route is the sole
authority for rollout readiness telemetry.
- The public application listener retains `/_readiness` for
compatibility, evaluates the same dependencies, and preserves the
existing response shape, but it emits no `buzz_readiness_*` metrics.
- Ready responses remain `200 {"status":"ready"}`.
- Shutdown responses remain `503 {"status":"shutting_down"}`.
- Failed private-health responses include the bounded `reason` plus
`postgres`, `redis`, and `deletion_catalog` booleans.
- Failed public compatibility responses retain the dependency booleans
but omit the new detailed reason.
- No header, query parameter, path value, or other request-controlled
value becomes a metric label.

## Prometheus contract

The final schema is intentionally capped at **99 raw Prometheus series
per pod**.

| Metric | Type | Labels | Raw series/pod |
|---|---|---|---:|
| `buzz_readiness_checks_total` | counter | `reason` | 12 |
| `buzz_readiness_dependency_checks_total` | counter | `dependency`,
typed `outcome` | 11 |
| `buzz_readiness_check_duration_seconds` | histogram | `check` | 72 |
| `buzz_readiness_state` | gauge | `check` | 4 |
| **Total** |  |  | **99** |

### Closed label sets

`reason`:

```text
ready
shutting_down
postgres_pool_timeout
postgres_pool_error
postgres_query_timeout
postgres_query_error
redis_pool_timeout
redis_pool_error
deletion_catalog_timeout
deletion_catalog_error
overall_timeout
multiple_dependencies_failed
```

Valid `dependency` / `outcome` pairs are enforced by typed enums:

- `postgres`: `success | pool_timeout | pool_error | operation_timeout |
operation_error`
- `redis`: `success | pool_timeout | pool_error`
- `deletion_catalog`: `success | operation_timeout | operation_error`

`check` is `overall | postgres | redis | deletion_catalog`.

The readiness histogram has 15 configured buckets concentrated around
the two-second deadline:

```text
0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5,
0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.5, +Inf
```

The contract deliberately removes the redundant overall `result` label
and the histogram `outcome` label. Pod, ReplicaSet, version, rollout,
raw error, SQL, URL, tenant, user, community, pubkey, and
request-controlled values are prohibited application labels;
infrastructure enrichment can supply deployment identity outside the
application metric.

## Postgres production seam and CI

The real `Db::readiness_check` path now supports deterministic
production-seam testing while preserving the same acquisition/query
implementation used by the relay. The isolated PostgreSQL lane
automatically discovers and executes three ignored integration tests
covering:

- a held sole connection causing pool timeout, followed by recovery
after release;
- closed-pool acquisition error;
- acquisition success followed by query timeout;
- classified query error;
- cancellation while waiting for a connection;
- cancellation during an in-flight query; and
- eventual pool recovery with waiter/in-flight state balanced.

This prevents the central SQLx/Postgres behavior from being merely
compiled but never executed in CI.

## Tests and verification

- `cargo fmt --all -- --check`
- `./scripts/test-postgres-test-discovery.sh`
- complete `cargo test -p buzz-db`
- focused `cargo test -p buzz-relay readiness`
- real-router `cargo test -p buzz-relay real_health_route_`
- controlled PostgreSQL execution of all three ignored readiness tests
- production health router -> real `GET /_readiness` -> Prometheus
render assertions
- public-router requests proving zero rollout telemetry
- deterministic out-of-order A/B probe tests
- SIGTERM-during-probe tests proving shutdown dominance
- exported metric name/type/exact-label/bucket assertions
- exported raw-series allowlist and 99-series ceiling assertion

The route-to-scrape regression test uses the production Prometheus
builder and verifies the `2`, `2.5`, and `+Inf` readiness buckets. It
fails if the health recording call, health route, public/health
boundary, generation fence, shutdown fence, or bucket override is
removed.

## Risk assessment

**Medium.** This changes the live readiness publication path and adds
synchronization around probe commit/shutdown. The risk is bounded by:

- preserving the existing dependencies and shared two-second deadline;
- keeping readiness fail-closed;
- using a short, process-local mutex only at begin/commit linearization
points, not across dependency awaits;
- retaining the public compatibility endpoint while isolating its
telemetry;
- enforcing typed, low-cardinality labels and an exported series
ceiling; and
- covering the production router, Prometheus exposition, real PostgreSQL
seam, concurrency ordering, and shutdown races.

## Operational notes

- Dashboards should treat `buzz_readiness_state` as the latest sampled
current state, not as an event stream.
- `shutting_down` should be excluded from dependency-failure alerts
because no dependency failure is implied.
- Do not use `default_zero()` for missing current-state data;
missing/stale is unknown, not healthy.
- Use histogram buckets, heatmaps, max, or average until the Datadog
distribution metadata confirms percentiles are enabled; do not title a
widget p95 before that live readback.

## Non-goals

This PR does not add the broader process-startup lifecycle,
worker/listener supervision, shutdown coordinator, WebSocket/huddle
handoff, client recovery telemetry, dashboard mutations, Datadog
configuration changes, or deployment changes. Those remain separate
follow-up work.

## References

- [Staging dev relay image
runbook](https://github.com/block/buzz/blob/main/docs/staging-dev-relay-images.md)
- Readiness telemetry contract: `deploy/charts/buzz/README.md`

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
 2026-09-01 18:31:30 -04:00
Tom Brow
42b42447b0 feat(mobile): prepare buzz-push-gateway for deployment (#7158) 
A few things to prepare for deploying the Buzz push gateway as
implemented in #6269:
- Cut `buzz-push-gateway` chart `0.2.0` from the current config
(certificate auth, dogfood client)
- Datadog-compatible metrics
- Align the desired schema, metrics docs, and production ingress
guidance with the current dogfood deployment contract

## Links

- [BUZZ-17: Publish the MVP-compatible gateway
chart](https://linear.app/squareup/issue/BUZZ-17/publish-the-mvp-compatible-gateway-chart)
- [BUZZ-26: Add Datadog-compatible push-gateway metrics and correct
source
drift](https://linear.app/squareup/issue/BUZZ-26/add-datadog-compatible-push-gateway-metrics-and-correct-source-drift)

---------

Signed-off-by: Tom Brow <tomb@block.xyz>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-01 12:19:48 -07:00
Luke Tornquist
bd73490418 ci: run PostgreSQL tests in isolated lane (#6730) 
## Why

PR #6660 introduced four PostgreSQL-backed persistence tests and a
focused CI selector, but an exact test-name list does not automatically
cover future database tests. Broad ignored-test execution also exposed
shared-schema races and ambiguity between desired-state and
migration-applied schema expectations.

## What

- Establish a discoverable convention: PostgreSQL unit modules use
`postgres_tests`, PostgreSQL integration binaries use a `postgres_`
prefix, and unrelated external-infrastructure tests use an
`external_infra_` prefix.
- Add a dedicated nextest PostgreSQL profile and archive-backed CI job
covering the relevant crates without enumerating test names.
- Create a run-scoped desired-state source database and a unique
PostgreSQL database per test process for parallel-safe isolation, with
cleanup on success, failure, or interruption.
- Route destructive migration and migration-parity tests to clean
`template0` databases while desired-state tests clone the desired-state
database.
- Document discovery, schema modes, required role privileges, and a
bounded portable local runner.
- Keep the existing infrastructure-free unit-test jobs unchanged.

This is intentionally limited to test and CI harness behavior. It does
not move production database code, change persistence semantics, or
implement issue #20's broader shared test-utility refactor.

## Risk

Low production risk because all changes are confined to tests, CI
configuration, documentation, and test harness scripts.

Remaining operational risks:

- Cleanup retries `dropdb --force` five times across roughly four
seconds. Exhaustion warns with the database name but deliberately does
not mask the test result; individual PostgreSQL diagnostics are
suppressed.
- The seven-package boundary is duplicated between the runner and CI
archive and must remain synchronized if PostgreSQL tests move to a new
crate.
- Portable SHA fallback branches were exercised on Linux; no macOS Blox
workstation was needed for this Linux CI artifact.
- GitHub's unchanged generic Unit Tests job currently fails while
cold-linking `buzz-voice` because `sherpa-onnx-c-api` is absent. The
branch changes no voice/build/toolchain inputs; the exact parent
previously passed that job, and Blox passes the same infrastructure-free
suite (1,450/1,450). A prior-head one-job retry and the final-head run
both reproduced the hosted-runner failure.

## Verification

Author Blox workstation `2020088`:

- `cargo fmt --all -- --check`
- `just clippy`
- `just test-unit`: 1,450 passed; PostgreSQL tests remain skipped in
fast jobs
- PR #6660 focused tests: 4/4 passed through the final runner
- Full lane: 282/282 passed in 10.087s with a non-superuser role limited
to `CREATEDB`, `CREATEROLE`, and `pg_read_all_stats`
  - 272 desired-state tests
  - 10 migration-applied tests
  - 6 explicitly filtered external-infrastructure tests
- Cleanup fault injection: a deliberately underprivileged 282-test run
produced the expected 277 passes and 5 failures, and the post-run
catalog audit found zero lane databases after retry cleanup. The
restored successful run also left zero lane databases.

GitHub exact final head `92c231e48f299ea2af23817763f5ae80ac013d68`:

- [PostgreSQL
Tests](https://github.com/block/buzz/actions/runs/32789030203/job/97630827823):
282/282 passed across 10 binaries in 98.886s; 1,240 skipped, including 6
via the profile filter
- The shared relay/PostgreSQL archive predecessor completed successfully
- [Unit
Tests](https://github.com/block/buzz/actions/runs/32789030203/job/97626930420):
unrelated `sherpa-onnx-c-api` native-link failure described above
- Independent exact-head review on separate Blox workstation `2022762`:
no substantive findings after all initial findings were addressed. The
reviewer independently verified cleanup retry behavior, exit-status
preservation, shell portability, syntax, and a clean exact-head
worktree.

## References

- Stacked on #6660 at exact head
`561de54be4d9c2b622b7c2aa5b61bc3068f47e2a`
- https://github.com/TheSentinel454/buzz/issues/20
-
https://github.com/TheSentinel454/buzz/issues/20#issuecomment-5398526792

Generated with Codex.

## Update — August 24, 2026 review follow-up

- Centralized the repeated PostgreSQL test URL resolution in
crate-local, test-only helpers without introducing the broader shared
utility refactor from issue #20.
- Restored descriptive hybrid/Redis test function names. Structural
`external_infra_*_tests` modules now own exclusion, and the nextest
filter only recognizes module path segments.
- Added a three-second source guard that scans every Rust file and fails
CI when an ignored PostgreSQL test would be omitted or an
external-infrastructure test would be included. Fixture tests cover
accepted modules/binaries and both failure modes.
- Removed the two implementation-plan documents.

Performance profiling on Blox workstation `2027352` (same prebuilt
archive and cargo-nextest 0.9.143 for every comparison):

- Current per-test database model, 8 workers: 10.15–10.40s across three
runs. The 283 `createdb` calls and 284 `dropdb` calls consumed
28.85–29.46 aggregate seconds, about 39–40% of aggregate test-process
duration.
- Reusing one database per worker without cleanup: 6.55–6.84s, about 35%
faster, but all three runs failed because global matcher-queue tests
inherited a quiesced community from an earlier test; the failing test
varied with scheduling.
- Reusing workers with `TRUNCATE … RESTART IDENTITY CASCADE`: all 282
tests passed, but total time regressed to 10.47–10.67s. The 265 truncate
calls consumed 24.34–25.45 aggregate seconds, so truncation merely
replaced most clone/drop cost.
- Raising concurrency to 16 workers reduced one isolated-database run to
8.79s, but 24 workers exposed a cluster-global `pg_stat_activity` race.
The profile remains at the proven-safe 8 workers.

Decision: retain unique per-test databases. Harness cleanup uses `dropdb
--if-exists --force`; it drops the database rather than truncating
tables or deleting rows.

Verification at local branch head
`60086c1f6ee34a193c342255a5e8e6293b60e988`:

- Discovery inventory: exactly 282 intended tests; all 6 hybrid/Redis
tests present and structurally excluded.
- PR #6660 focused persistence tests: 4/4 passed.
- Full PostgreSQL lane: 282/282 passed three times; 1,240 skipped each
run; wall time 9.72–9.96s without profiling shims.
- Non-ignored tests with a desired-state database: 1,234/1,234 passed;
one mesh-demo timeout from the first run passed immediately in
isolation.
- `cargo fmt --all -- --check` and all-target/all-feature clippy for
`buzz-db`, `buzz-deletion`, and `buzz-relay` passed.

Final independent re-review at exact head
`42097c0136ab3fa8f0efe5720ae829c04575b9b6` on separate Blox workstation
`2028455`: no substantive residual findings. The reviewer independently
exercised ordinary strings, zero/one/three-hash raw strings, line/block
comments, and string-contained lookalikes, then reran the full 353-file
scan, shell syntax, Python AST parsing, and diff check.

Independent review found that the source guard's initial regular
expression could miss raw-string ignore reasons and treat a commented
attribute as real. Final head `42097c0136ab3fa8f0efe5720ae829c04575b9b6`
parses valid ordinary/raw Rust string literals only at attributes found
in comment-sanitized source; regression fixtures cover both cases. The
fixture suite, Python compilation, shell syntax, diff check, and full
353-file repository scan pass on Blox.

## Update — August 25, 2026 restack

PR #6660 merged, so this follow-up was rebased from its former exact
parent onto current `origin/main`
(`8d2d0ff5ad42733e9949442c4b6358d0ba87f9a8`). The final candidate head
is `05dcc2ab28948e3ab79bb44839a69f2ba44648a2`; the PR no longer carries
#6660's pre-squash history.

Fresh exact-head verification on Blox workstation `2028572`:

- Discovery guard passed across 353 Rust files; inventory remained
exactly 282 intended PostgreSQL tests.
- PR #6660 focused persistence tests: 4/4 passed.
- Full PostgreSQL lane: 282/282 passed three times; 1,243 skipped each
run; wall time 9.70–9.87s.
- `cargo fmt --all -- --check` and all-target/all-feature clippy for
`buzz-db`, `buzz-deletion`, and `buzz-relay` passed.

## Update — August 25, 2026 final hosted verification

- [GitHub PostgreSQL
Tests](https://github.com/block/buzz/actions/runs/32801679226/job/97667835080):
282/282 passed in 53.787s; 1,243 skipped.
- The final guard scanned all 353 Rust files successfully before the
hosted lane ran.
- The shared relay/archive prerequisite and infrastructure-free Unit
Tests job both passed at exact final head
`42097c0136ab3fa8f0efe5720ae829c04575b9b6`.

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Signed-off-by: tornquist <tornquist@squareup.com>
 2026-09-01 12:07:28 -04:00
Will Pfleger
9ab163190b feat(db): add NIP-FI identity and final-admission schema foundation (#6994) 
PR 2 of the NIP-FI plan: the schema foundation. Establishes the durable
server-side identity ledger and final-admission surface that the runtime
phases build on. All of Phase A's migrations live here; later phases own
their own deltas.

Depends on nothing — PR 1 (#6776, merged) owned zero migration files.
This PR's relations are shaped to store exactly what PR 1's verifier
produces: issuer-qualified identity and the four denial classes. They
meet in a later PR that writes a verified assertion into these tables in
one transaction.

## Two internally-ordered migrations

- `0041_nip_fi_identity_foundation.sql` (migration A) — core identity +
base-lifecycle relations (5 tables): issuer-qualified `(iss, sub)`
bindings, lifecycle history/selectors, enrollment policies, and
operation receipts. Applies cleanly to current `main`.
- `0042_nip_fi_authorization_foundation.sql` (migration B) — the
final-admission surface (10 tables): authorization events + capacity,
admission results, replay/receipt guards, audit, invalidation
domains/floors, protected-object authority, authority epochs, and
restore version deltas. Applies to A's resulting state.

Fifteen NIP-FI relations total, zero dangling foreign keys. Identity is
issuer-qualified throughout — no single-global-issuer assumption in any
relation, no `Block`-hardcoding. A single deployment may run one issuer;
that is config, not schema.

## Durable, immutable ledger posture

All 15 relations are append-only (immutable `no_delete`/`no_truncate`
triggers) and carry `community_id` as provenance, not ownership. Both
migrations widen the single SQL source of truth
`community_write_fence_excluded_table` so the relations are never
fence-attached, never purged on community deletion, and never counted as
tenant-scoped drift by the deletion control plane's exact-set catalog
check — the same posture main already applies to `product_feedback` and
`rate_limit_violations`. `schema/schema.sql` keeps one consolidated
definition of that function whose exclusion array byte-matches `0042`,
guarded by a parity assertion so a future consolidation cannot silently
drop NIP-FI relations from the ledger.

This makes a tenant's identity/authorization ledger survive community
deletion, per the spec's `FI-INV-02` (durable binding) and `FI-INV-03`
(tombstone monotonicity) and `NIP-FI.md`'s "durable server state"
ruling. `communities(id)` FK never dangles: community rows become
permanent tombstones, never hard-deleted.

## Authorization shape and cardinality contracts

Authenticated `OperatorDenied` events (`actor_kind` 1–3, non-null
`request_fingerprint`) carry a null `semantic_fingerprint` and commit
without a denial-attempt row. The denial-attempt cardinality and shape
guards are scoped to unresolved pre-auth kind-9 events (`actor_kind =
4`). Applied and no-op lifecycle receipts (`outcome_code IN (1, 3)`)
require exactly one mapped success-transition event; denied lifecycle
receipts (`outcome_code = 2`) require zero events from the complete core
lifecycle success-transition class (kinds 1, 2, 3, 6: enrolled, revoked,
rotated, retired) — any such event paired with a denied receipt would
record a transition that never occurred.

## Mined vs. new

Re-cut from Franco's #1476 (`0029`/`0030`) and Cea's #4772 committer
schema, re-cut along FK topology and renumbered above the live `main`
tip. The buzz-auth core of #1476 is Cea-authored; `Co-authored-by`
reflects verified per-commit authorship of the mined schema.

Zero Rust/`deletion.rs` edits — the migration-only exclusion widening
keeps `EXPECTED_SCOPED_TABLES` untouched.

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
Co-authored-by: Cea Stapleton Cordasco <261786559+cea@users.noreply.github.com>
Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
 2026-08-31 19:40:35 -04:00
Luke Tornquist
3ed623bb21 feat(db): configurable writer session timeouts (lock, idle-txn, statement) (#6229) 
## Why

A wedged relay boot pod holding a relation lock can park every other
writer in the fleet behind it: DB load pins at pool capacity in
`Lock:relation` waits while CPU stays flat, and nothing server-side
releases the lock until the holder dies. We hit exactly this in
production — ~1,400 sessions queued behind one crash-looping pod's boot
transaction for ~20 minutes until kubelet killed the container.

## What

Applies session-level Postgres timeouts to every **writer** connection
inside the existing single `after_connect` hook in `buzz-db`, all
env-tunable through the same `Config::from_env → DbConfig` path as the
existing pool-size knobs:

| Env var | GUC | Default | Effect |
|---|---|---|---|
| `BUZZ_DB_LOCK_TIMEOUT_MS` | `lock_timeout` | 5000 | statements waiting
on any lock fail fast instead of parking behind a wedged holder |
| `BUZZ_DB_IDLE_TXN_TIMEOUT_MS` | `idle_in_transaction_session_timeout`
| 60000 | reaps wedged clients idling inside an open transaction while
holding locks |
| `BUZZ_DB_STATEMENT_TIMEOUT_MS` | `statement_timeout` | 0 (off) |
opt-in runaway-statement cap; off by default because startup
migrations/backfills legitimately run long statements |

`0` disables a timeout (Postgres semantics) and deliberately passes
through the env parsing — unlike the pool-size knobs where `0` falls
back to the default. The reader pool is untouched: replica sessions
never take contended locks and already fail acquire in 150 ms.

Deployers tune these via plain env vars (`.env`, or `relay.extraEnv` in
the Helm chart) — no code changes needed.

## Behavior change to note

With the 5 s default `lock_timeout`, a boot-time migration or backfill
that waits >5 s on a lock now errors (surfacing in logs / crash-looping
the pod) instead of stalling silently. That is the intended
visible-failure-over-fleet-stall tradeoff; deployers with slow contended
migrations can set `BUZZ_DB_LOCK_TIMEOUT_MS=0`.

## Testing

- `cargo test -p buzz-db -p buzz-relay` — buzz-db green; buzz-relay has
9 failures that also fail on clean `main` in this environment
(api::admin/api::media/mesh_demo — unrelated, pre-existing).
- New config test covers override / `0`-passthrough / invalid-fallback
for all three env vars.
- Extended the existing `writer_pool_safety_hook_is_single_and_composed`
source-shape test so the timeouts can't drift out of the single
`after_connect` hook (SQLx replaces hooks — a second hook would silently
disarm the floor guard).
- `cargo fmt --check` and `cargo clippy --all-targets` clean for the
touched crates.

Closest existing PR/issue: none found.

---
**Update Aug 28, 17:06 EDT:** Rebased onto `main` at `a3730784fc` and
addressed the latest correctness review.

- Ported the timeout policy onto the refactored `buzz-db::runtime` pool
constructor and kept the shared env overlay for relay, admin, deletion,
and audit writers.
- Migration/schema-destruction connections now disable `lock_timeout`
and `statement_timeout` for their intentional long wait/DDL path. This
supersedes the earlier “Behavior change to note”: contended boot
migrations wait for the current migration owner rather than
crash-looping after five seconds.
- The audit worker now preserves and retries the same entry on
PostgreSQL `55P03` lock timeouts, using exponential backoff capped at
one second. Other database errors retain the existing terminal error
behavior, and retries emit `buzz_audit_log_lock_retries_total`.
- Added CI-backed PostgreSQL regressions for writer GUC
installation/migration exemption, audit-pool lock timeouts, and worker
recovery. The worker regression holds the real audit advisory lock past
`lock_timeout`, observes a retry, releases the lock, and proves the
original entry is appended exactly once.

Current verification supersedes the earlier testing notes: workspace
Rust clippy passed with warnings denied; all nine infrastructure-free
backend unit-test lanes passed; all three focused PostgreSQL regressions
passed against PostgreSQL 17; formatting, diff checks, file-size guards,
and desktop frontend checks passed. The Linux Blox workstation could not
run the unrelated Tauri native lane because `glib-2.0` is absent, so
that platform check is left to PR CI.


---
**Update Aug 31, 11:09 EDT:** Rebased onto current `main` at
`c3132c3ee9` and reran the requested audit-lock contention scenario on
Blox at head `896c3fe9ed`.

- `git range-diff` reports both PR commits unchanged by the rebase; the
branch remains two commits and the worktree is clean.
- `cargo fmt --all -- --check` and clippy with warnings denied passed
for `buzz-db`, `buzz-relay`, `buzz-admin`, and `buzz-deletion`.
- All three focused PostgreSQL 17 regressions passed: writer session
timeout/migration exemption, audit writer timeout bounds, and audit
worker recovery of the original entry exactly once.
- Live protocol verification used a head-built relay and CLI, native
PostgreSQL 17/Redis, `BUZZ_DB_LOCK_TIMEOUT_MS=300`, and an eight-second
hold on the community audit advisory lock. The real message was accepted
and persisted once while the lock was held; its audit-row count remained
zero during contention while retries accumulated. After release, exactly
one `event_created` audit row appeared and remained exactly one after an
additional two-second duplicate check. The run recorded nine
lock-timeout retries, zero audit failures, and event ID
`7f8c4ffae28e78555fcf2d56396d6e6c01b3712e5411288dc79e9a54af9d9444`.

Generated with Codex

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
 2026-08-31 11:59:13 -04:00
Tom Brow
c432a111ca feat(mobile): push notifications MVP (#6269) 
This PR implements MVP, iOS-only,
[NIP-PL](8d2d0ff5ad/docs/nips/NIP-PL.md)-compliant
push notifications.

A relay with `BUZZ_PUSH_ENABLED` will send a push notification for any
message that appears in the in-app Notifications tab.

## Enrollment flow
The first time the client first connects to a relay with
`BUZZ_PUSH_ENABLED`:
```mermaid
sequenceDiagram
    autonumber
    participant App as Buzz iOS app
    participant iOS
    participant Relay as Buzz relay
    participant Attest as Apple App Attest
    participant Gateway as Push gateway

    App->>Relay: Fetch NIP-11 push capability
    Relay-->>App: Push profile, current relay public key, and limits

    par
        App->>iOS: Request notification permission
        iOS-->>App: Permission result
    and
        App->>iOS: Register for remote notifications
        iOS-->>App: Device token
    end

    App->>Gateway: Request installation challenge
    Gateway-->>App: Single-use challenge
    App->>Attest: Attest installation transcript
    Attest-->>App: Attestation proof
    App->>Gateway: Enroll device token and proof
    Gateway-->>App: Installation handle

    App->>Gateway: Request delegation challenge
    Gateway-->>App: Single-use challenge
    App->>Attest: Assert relay-key delegation
    Attest-->>App: Assertion
    App->>Gateway: Create delegation
    Gateway-->>App: Opaque endpoint grant

    App->>Relay: Publish encrypted push lease and filters
    Relay-->>App: Lease acknowledged
```

## Push-time flow

When a notification-eligible event is received by the relay:

```mermaid
%%{init: {
  "sequence": {
    "actorMargin": 20,
    "width": 110,
    "messageMargin": 18,
    "diagramMarginX": 8,
    "wrap": true
  }
}}%%
sequenceDiagram
    autonumber
    participant Relay as Buzz relay
    participant Gateway as Push gateway
    participant APNs as Apple Push<br/>Notification service
    participant iOS
    participant NSE as Notification service<br/>extension

    Relay->>Gateway: POST /v1/deliveries/apns<br/>opaque endpoint grant, request ID, expiry, NIP-98 authorization

    Gateway->>APNs: POST /3/device/{device-token}<br/>topic, request ID, expiry, constant mutable-content payload
    APNs-->>Gateway: 200 OK: request accepted
    Gateway-->>Relay: 200 OK: accepted status

    APNs-->>iOS: Notification: constant reconnect alert<br/>mutable-content = 1
    iOS->>NSE: Invoke extension<br/>original notification content

    NSE->>Relay: POST /query: subscription filters, limit 10<br/>NIP-98 authorization
    Relay-->>NSE: 200 OK: signed Nostr events<br/>kinds 9, 40002, 45001, or 45003

    NSE->>iOS: Complete notification: title, body, subtitle<br/>thread ID, exact-message target
```

relay → push gateway → APNs -> NSE -> Notification Center

## Known limitations

The APNs wake payload is intentionally constant and opaque: it contains
no originating community or message identifier, in keeping with the
implemented NIP-PL privacy design.

The Notification Service Extension must therefore reconnect to the relay
and resolve eligible messages after each wake. Around overlapping wakes,
timing boundaries, or resolution windows, notification presentation may
occasionally omit an expected message or display a message more than
once.

This best-effort behavior is deliberately accepted for the current
implementation and will be measured during the internal rollout to
determine whether the user experience is acceptable before any broader
deployment; the implementation does not claim exactly-once presentation.

## Validation

Live end-to-end hardware validation used an internal remotely hosted
development relay and push gateway, the APNs sandbox, and a physical
iPhone 12 mini:

- A second real Buzz client published a uniquely marked message through
the hosted relay.
- The relay matched the message and sent the constant opaque wake
through the hosted gateway. The gateway made an actual APNs request; no
`simctl push` or simulated notification was used.
- The iPhone received the notification on its lock screen. The
Notification Service Extension reconnected to the relay, fetched the
event, verified its ID and signature, and replaced the placeholder
content with the real notification title and body.
- After the app populated its shared presentation cache, a final marked
notification visibly showed the sender display name, sender avatar, and
hashtag-prefixed channel name.
- Tapping a lock-screen notification opened Buzz and exercised the
notification-response path and navigated to the corresponding message.

Final validation with a dogfood-signed artifact and production App
Attest/APNs configuration remains a release step.

## Independent pre-reviews

- **First pass:**
[Carl](buzz://message?channel=18882f4c-289f-41db-942f-81f6f8066da1&id=74ab9a93bb227f3e762568f1cf9fee66d7495b0edc3918735ff787238b9cc585)
found missing transient retries, executor-key rotation suppression,
duplicate installation renewal, and an unauthenticated challenge write
amplifier. These were resolved by [retry-safe
bootstrap](https://github.com/block/buzz/pull/6269/changes/12c66ea62)
and [authenticated renewal plus a cross-replica
quota](https://github.com/block/buzz/pull/6269/changes/8e5ece0bd).
[sol-max](buzz://message?channel=ad83385f-8e9e-4461-9a35-c1bf2e208532&id=d26d53daa4684669e2ed354638241f13f36c3a97027fe8b4dd738aff09038962)
found delegation generation burning and an edited applied migration,
resolved by [exact-generation
revocation](https://github.com/block/buzz/pull/6269/changes/c26d2159d)
and a [forward-only
migration](https://github.com/block/buzz/pull/6269/changes/956c1d099).
[k3-max](buzz://message?channel=5e46055d-a766-4065-ae25-05d1e4aaa6b2&id=d43139138a0b15f806cbdbeeedd8f69d992cadf2e805876db6fdde6a34c7eda1)
found no blockers.
- **Exact-head re-review:**
[Carl](buzz://message?channel=18882f4c-289f-41db-942f-81f6f8066da1&id=a897721673459301b0cf26e8b85a1478d7ebbb56a4621f93d774c98d395b8f68),
[sol-max](buzz://message?channel=ad83385f-8e9e-4461-9a35-c1bf2e208532&id=fb2159f709ec68f74f7b21459acd76da0e8a7c5c0f3d469f99826b0cc2380849),
and
[k3-max](buzz://message?channel=5e46055d-a766-4065-ae25-05d1e4aaa6b2&id=2ce2842910435f562e9d9cc718595848f281b122c94605e523a4b964254b8bfb)
independently returned **NO BLOCKERS** at `7eb3a650b`; k3-max also
revalidated every remediation and the endpoint-specific App Attest
enrollment bound.

---------

Signed-off-by: Tom Brow <tomb@squareup.com>
Signed-off-by: Tom Brow <tomb@block.xyz>
Co-authored-by: Tom Brow <tomb@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Jordan Mecom <jm@squareup.com>
 2026-08-28 15:15:09 -06:00
Luke Tornquist
a3730784fc refactor(db): extract domain stores from database runtime (#6987) 
## Summary

Finish the remaining database-store extraction tracked by
[TheSentinel454/buzz#2](https://github.com/TheSentinel454/buzz/issues/2)
in one reviewable PR.

This consolidates the previously stacked domain slices after #6782
merged. It preserves the runtime/store boundary established by #6660,
#6668, #6700, and #6782 while separating database runtime infrastructure
from domain-owned persistence:

- `runtime/` owns pool construction and sizing, writer/reader routing,
read sessions and route proofs, transaction infrastructure,
observability primitives, replica fencing, health support, migrations,
and cross-cutting runtime tests.
- `store/` owns domain records, SQL, row parsing, locks and invariants,
`Db` domain methods, focused tests, and logical-operation datastore
spans.
- `lib.rs` remains a 57-line compatibility facade that preserves
existing crate-root paths and `Db` method signatures through re-exports.

Domain coverage includes API tokens, authentication allowlists,
reminders, event queries, threads, reactions, feeds, users and DMs,
push, workflows/runs/approvals, relay membership and invites, product
feedback, moderation/admin moderation, relay admin actions/operators,
git repositories, archived identities, usage, partition maintenance,
deletion, channel membership inherited from merged #6782, and the final
runtime/store layout.

The branch has been rebased onto current `main`. Database changes that
landed there were incorporated rather than overwritten:
`relay_admin_actions.rs` and `relay_operators.rs` now live under
`store/`, their 27 public `Db` wrappers and existing behavior remain
intact, and every wrapper has exactly one fixed-name datastore span.
Concurrent changes to migration, moderation, admin moderation, and error
handling are also retained.

### Exact base and head

- Base: `main` at `ed11c8d8bf0a17402be5cf243724f89471530d2f`
- Head: `codex/issue-2-store-extraction` at
`be24430472d1a87ac5c0d6026c620cd6caea3537`

### Related issue

- Structural tracker:
[TheSentinel454/buzz#2](https://github.com/TheSentinel454/buzz/issues/2)
- Domain trackers:
[#6](https://github.com/TheSentinel454/buzz/issues/6),
[#7](https://github.com/TheSentinel454/buzz/issues/7),
[#12](https://github.com/TheSentinel454/buzz/issues/12),
[#13](https://github.com/TheSentinel454/buzz/issues/13)
- Acceptance trackers:
[#17](https://github.com/TheSentinel454/buzz/issues/17),
[#19](https://github.com/TheSentinel454/buzz/issues/19)

This supersedes #6783, #6784, #6787, #6788, #6789, #6792, #6820, #6794,
#6796, #6797, #6798, #6799, #6804, #6805, #6806, #6808, #6809, #6811,
#6812, #6813, #6814, #6815, and #6890. Their discussions remain
available for review history.

### #17 / #19 acceptance

- Preserves the metric names, fixed labels, transaction/lock timing
boundaries, and privacy/cardinality constraints introduced by #6700.
- Keeps exactly one datastore span per public logical operation,
including the 27 relay-admin wrappers added on `main`.
- Removes `store_ownership.rs`; physical ownership and focused source
guards now enforce the boundary directly.
- Leaves no `impl Db`, domain SQL, focused domain test group, or
datastore span in `lib.rs`.
- Preserves existing public paths such as `buzz_db::channel`,
`buzz_db::event`, and `buzz_db::workflow` through crate-root re-exports
while keeping internal `runtime` and `store` namespaces private.

### Non-goals

- No SQL, schema, locking, transaction, retry, timeout, or
client-visible behavior changes.
- No generic store traits, domain handles, broad `PgExecutor` migration,
new store crate, raw pool accessor, or broader directory reorganization.
- No tracker issues are closed by this PR.

### Risk

The cumulative diff is large but structural. Risk is primarily
module-path, ownership, or conflict-resolution drift. It is mitigated by
preserving public re-exports, comparing the newly moved `main`
implementations to their upstream source, source guards, touched-crate
compilation, PostgreSQL-backed test coverage, and an independent
exact-head review on a separate clean Blox workstation.

### Testing

Author workstation `buzz-tornquist-pr-6987-rebase`, rebased branch
ending at exact head `be24430472d1a87ac5c0d6026c620cd6caea3537`:

- `cargo fmt --all --check`
- `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings`
- `cargo test -p buzz-db --lib` — 113 passed, 240 PostgreSQL tests
intentionally ignored
- `cargo test -p buzz-db --test observability_source` — 2 passed
- PostgreSQL-backed `buzz-db` coverage under native PostgreSQL — 235
passed in the shared serial run; the five shared-state/config-sensitive
cases passed as isolated reruns against fresh schemas, including the two
owner-limit tests with their fixture's
`BUZZ_MAX_COMMUNITIES_PER_OWNER=3`
- `cargo test -p buzz-relay --lib -- --test-threads=1` under native
PostgreSQL/Redis — 991 passed; the three current-month
partition-sensitive identity-archive cases passed after provisioning the
August 2026 test partition; 87 infrastructure-marked tests remained
ignored
- Source/diff guards — relay-admin implementation bodies match current
`main`; all 27 public wrapper signatures are retained; exactly one
datastore span wraps each wrapper; `lib.rs` has zero `impl Db` blocks
and zero datastore spans; no duplicate top-level relay-admin modules or
`store_ownership.rs`; `error.rs` matches current `main`

Independent clean review workstation `buzz-tornquist-pr-6987-review`,
detached at exact head `be24430472d1a87ac5c0d6026c620cd6caea3537`:

- `cargo fmt --all --check`
- `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings`
- `cargo test -p buzz-db --lib` — 113 passed, 240 ignored
- `cargo test -p buzz-db --test observability_source` — 2 passed
- Exact-head ownership/re-export/instrumentation audit — no remaining
actionable findings

---------

Signed-off-by: OpenAI Codex <codex@openai.com>
Signed-off-by: tornquist <tornquist@squareup.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-28 15:24:32 -04:00
Will Pfleger
86b9142a09 refactor(relay): NIP-98 admin auth with Operator/Moderator roles and NIP-11 discovery (#3777) 
Adds authenticated, role-based moderation to the relay admin API. On
`main` the admin API is read-only and gated only by `Host`/`Origin`
matching; this branch adds NIP-98 authentication, a two-tier
Operator/Moderator principal model, mutation and staffing endpoints, and
NIP-11 auto-discovery so clients never type the admin URL by hand.

## Authentication (`BUZZ_ADMIN_AUTH`)

`BUZZ_ADMIN_AUTH` accepts `nip98` or `disabled`. Leaving it unset
defaults to `nip98` (fail-secure). Configuration fails closed: any other
value aborts startup, while a lingering `BUZZ_ADMIN_TOKEN` is ignored
with a startup warning — token (bearer) authentication is not supported.
`Host`/`Origin` matching is retained in every mode as defense-in-depth.

- **`nip98`** (default) — per-request signed NIP-98 (kind 27235) events,
resolved to an Operator or Moderator principal with per-person
attribution and individual revocability. Read-write per resolved
principal.
- **`disabled`** — no credential; relies entirely on network-layer
controls (reverse proxy, VPN, firewall) and logs a `WARN` on every boot.
Always read-only: `authorize()` resolves no principal, so mutation and
staffing routes always `403`.

## Roles

Buzz has two independent authority axes after this change.
**Relay-level** roles (new here) are deployment-global: they act across
every community on the relay, through the admin API. **Community-level**
roles (pre-existing, unchanged by this PR) are tenant-scoped: they act
inside one community, through signed Nostr moderation commands.

### Relay level (new)

| Role | Description |
|---|---|
| **Operator** | Full control of the deployment's moderation surface:
read all reports, feedback, and attachments across every community;
resolve reports with enforcement (`delete`/`kick`/`ban`/`timeout`) or
decisions (`dismiss`/`escalate`); reopen and cancel; update feedback
status; and manage the Operator/Moderator roster via the staffing
endpoints. |
| **Moderator** | Day-to-day triage: everything an Operator can do
except staffing — cannot view or change the roster. |

How a pubkey acquires a relay role (resolution order; config always
outranks DB):

1. Listed in `RELAY_OPERATOR_PUBKEYS` → **Operator** (source `config`)
2. Equals `RELAY_OWNER_PUBKEY` while `RELAY_OPERATOR_PUBKEYS` is empty →
**Operator** (source `owner_fallback`, a break-glass grant for
self-hosters that deactivates once any operator is configured)
3. Row in the `relay_operators` table → **Operator** or **Moderator**
(source `db`, managed via the staffing endpoints)
4. No match → `403`

### Community level (pre-existing, unchanged)

| Role | Description |
|---|---|
| **Owner** (community) | Full authority within their community: every
moderation action (delete, kick, ban/unban, timeout/untimeout, resolve
reports, view queue) plus member, role, and invite management. No guard
rails. |
| **Admin** (community) | Same community-wide moderation capabilities as
owner, except an admin cannot ban or time out the owner or a fellow
admin — only the owner may action an admin. Manages members and invites;
only the owner grants the admin role. |
| **Member** (community) | Standard participant; no moderation
capability. |
| **Owner / Admin** (channel) | Channel-local authority only: delete
messages and kick users within their own channel. |
| **Member / Guest / Bot** (channel) | No moderation authority. |

There is no community-level Moderator tier in v1; relay-level Moderator
is the only role by that name.

## Escalation scoping

The operator report queue is an escalation backstop, not the community's
day-to-day triage surface (per `VISION_MODERATION`, the severe class is
the platform's to review rather than the community's). Two rules enforce
that:

- **Escalated-by-default listing.** `GET /reports` with no `status`
parameter returns only `escalated` reports. An explicit
`status=<open|resolved|dismissed|escalated>` filter is always honored as
given, and full visibility across every status stays available for
platform-safety and legal review via `scope=all` (which lists reports
regardless of status). `scope` accepts only `all` and is ignored when an
explicit `status` is present.
- **Auto-escalated `illegal` reports.** Member reports whose category is
`illegal` are ingested with `status=escalated` rather than `open`, so
the severe class reaches the operator backstop without waiting for a
community admin to forward it. Every other category still lands `open`.
Auto-escalation only sets the queue status — it records no moderator
decision and stamps no resolver, so an auto-escalated report is
indistinguishable downstream from an admin-escalated one: the reopen
route returns it to `open` on the same terms, keyed only on status,
never on how the report became escalated.

## Principal resolution and NIP-98 admission

`resolve_admin_principal()` returns `AdminPrincipal { pubkey, role,
source }` per the resolution order above; `None` never falls through as
a role. Admission is ordered so the replay guard is a privilege, not a
public surface: signature/URL/method/payload-hash verification first,
roster check second, and only then is the deployment-scoped replay id
atomically consumed — a validly-signing but unrostered key never
allocates a replay slot. Redis failure fails closed.

## Report resolution, recovery, and enforcement provenance

`POST /reports/{id}/resolve` is a crash-safe enforcement state machine:
decision-only outcomes (`dismiss`/`escalate`) are a single
CAS-plus-audit transaction; enforcement
(`delete`/`kick`/`ban`/`timeout`) claims the report
(`open`→`processing`), runs the durable mutation, then finalizes — a
re-drive resumes at the step marker and converges to exactly-one
enforcement, fenced by a lease and an outbox claim token.

Person-directed enforcement on an `event`-kind report derives its target
from the stored event's author (server-owned truth, never the reporter's
`p` tag) via a single `derive_enforcement_target` shared by the HTTP
driver and the recovery worker. If the reported event was purged before
its author could be read, person-directed actions are rejected pre-claim
and the report stays `open`; `delete` needs only the event id and is
exempt.

`GET /reports/{id}` and the resolve response carry an `activeAction`
field surfacing the enforcement that actually executed — a report
dismissed after a reopen still reports the ban that ran. `POST
/reports/{id}/reopen` returns a terminal report to `open` (idempotent on
`requestId`). `POST /reports/{id}/cancel` is the sole recovery path for
a pre-mutation `failed` action, attributed via
`relay_admin_actions.cancelled_by`.

## Feedback

`GET /feedback` and `/feedback/{id}` survive a tenant purge: provenance
columns are severed to `NULL` rather than cascade-deleted, and the
attachment path fails closed to `404` on a severed row. `PATCH
/feedback/{id}` updates lifecycle `status`
(`new`/`reviewed`/`archived`).

## Staffing and probe

`GET/PUT/DELETE /operators/{pubkey}` are Operator-only; mutating a
config-backed pubkey returns `409 Conflict`. `GET /operators` returns
the union of config and DB principals with per-entry `source`. `GET
/probe` reports auth mode, role, source, `canAct`, and `canStaff` for
the desktop console.

## NIP-11 auto-discovery

The NIP-11 relay-information document gains an optional `admin_api`
field carrying the canonical admin origin (`scheme://host[:port]`, no
path), present iff `BUZZ_ADMIN_HOST` is set and omitted otherwise. The
scheme follows the same loopback rule as NIP-98 `u`-tag verification via
a shared `scheme_for_host` helper, so the advertised origin and the
origin the relay verifies against can never diverge.

## Operator API origin decoupling

`RELAY_OPERATOR_API_ORIGIN` is no longer required at boot when
`RELAY_OPERATOR_PUBKEYS` is set — it is used only by the
community-provisioning endpoints, which fail closed at request time
(with a boot-time `WARN`) until it is set. The admin console needs no
origin.

## Admin-web adaptation

The standalone `admin-web` dashboard signs each request as a NIP-98
event via a NIP-07 browser extension, discovers the auth mode with a
single unauthenticated probe (`200` → `disabled`, anything else →
`nip98`, fail-secure), and carries no token entry surface. Playwright
coverage exercises the NIP-98 and CSP paths.

## Security hardening

Three findings from security review are folded in:

- **Append-only roster audit.** `PUT`/`DELETE /operators/{pubkey}`
mutate the deployment-wide root of trust, but the upsert overwrites
`role`/`added_by` in place and the delete removes the only row — so a
grant→revoke sequence left no trace of who was ever granted or by whom.
Each mutation now writes an `relay_operator_audit` row (actor, target,
`grant`/`revoke`, pre-image `prev_role`, `new_role`, timestamp) inside
the same transaction as the mutation. A per-target transaction-scoped
advisory lock serializes concurrent mutations of the same pubkey before
the pre-image read, so the recorded `prev_role` is always the true
predecessor even under a concurrent-grant race. Chronology is keyed on a
`BIGINT GENERATED ALWAYS AS IDENTITY` `seq` column, not the wall clock:
the serializing lock guarantees insertion order and `seq` captures it,
so ordered reads (`ORDER BY seq`) follow the true privilege chain even
across a backward NTP step that a `clock_timestamp()` ordering would
invert. `created_at` (`clock_timestamp()`) is retained as informational
occurrence time only. Append-only by construction — no `UPDATE`/`DELETE`
path and no API surface. A no-op delete writes nothing.
- **`expirationSecs` overflow.** The timeout path built `Utc::now() +
Duration::seconds(secs as i64)` from an attacker-controlled `u64`:
`i64::MAX` panicked the handler, and a wrapped-negative magnitude minted
a *past* expiry that still passed validation. `compute_timeout_until`
now rejects zero, rejects magnitudes above a documented
`MAX_TIMEOUT_SECS` (365 days), and uses checked
`try_seconds`/`checked_add_signed` so no input can panic or produce a
past expiry — over-cap, zero, `i64::MAX`, and wrapping-negative inputs
all return a clean `4xx`.
- **Uppercase-hex config-backed bypass.** Config pubkeys are lowercased
at parse, but the `409` immutability check raw-string-compared the path
param while `decode_hex_pubkey` accepted uppercase — so `PUT
/operators/{UPPERCASE}` skipped the guard and wrote a shadow row for the
same 32 bytes. The validated param is now canonicalized (lowercased)
before the `409` check, DB write, `DELETE`, and response body.

## Migrations

- `0035_relay_operators.sql` — `relay_operators` roster table
(deployment-global), `actor_authority` on `moderation_actions`,
`processing` status plus `active_action_id` on `moderation_reports`,
`status` on `product_feedback`.
- `0036_relay_admin_actions.sql` — enforcement-action table with a
`request_id` idempotency key, a `step_marker` for crash recovery, and a
`cancelled_by` attribution column.
- `0037_relay_admin_action_lease.sql` — lease fencing for the action
worker.
- `0038_relay_admin_outbox_claim_token.sql` — fenced claim token on the
outbox worker.
- `0039_relay_operator_audit.sql` — append-only `relay_operator_audit`
trail for roster mutations (see Security hardening).

`docs/admin/README.md` documents the full principal model, NIP-98 event
requirements, capabilities by role, the startup error matrix, and the
discovery field.

## Production blast radius

A relay without `BUZZ_ADMIN_HOST` is completely unaffected — the admin
surface stays disabled and `BUZZ_ADMIN_AUTH` is ignored; a lingering
`BUZZ_ADMIN_TOKEN` logs a startup warning and must be removed. Where
`BUZZ_ADMIN_HOST` **is** set, unset `BUZZ_ADMIN_AUTH` defaults to
`nip98` (per-person signed auth); `BUZZ_ADMIN_AUTH=disabled` reproduces
`main`'s prior `Host`/`Origin`-only gating but is read-only (mutation
routes `403`). The five migrations add tables and columns without
touching existing data.

---

Related: [block/buzz#4768](https://github.com/block/buzz/pull/4768)
(desktop admin console consuming the `admin_api` field),
[squareup/bb-public#339](https://github.com/squareup/bb-public/pull/339)
(Phase 4 rollout config)

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
 2026-08-28 11:43:05 -04:00
Luke Tornquist
e76c81968b refactor(db): split channel membership store (#6782) 
## Current reconstructed head

Exact base: `codex/issue-7-roster-test-fixtures` at
`5df440c411be9705eb29a57f0c41f7239767e007`
Exact head: `codex/issue-7-channel-membership-store` at
`8ad0782ee311f5f51b714494ce750c5937f127cc`

This current head removes `crates/buzz-db/tests/store_ownership.rs`; no
replacement path-sensitive ownership test is introduced. Apart from
removing that complete test-file diff, the production patch is
byte-for-byte identical to the previously reviewed slice. This remains
part of tracker #2 and the #17/#19 acceptance work.

Independent exact-head review from a separate clean Blox workstation
found no issues. Current-head evidence passed formatting, strict
`buzz-db` clippy, 111 non-PostgreSQL library tests with 200 PostgreSQL
tests ignored, the observability source test, relay consumer
compilation, exact ownership/unique-span review checks, and 3 channel
and 19 membership PostgreSQL tests on native PostgreSQL where
applicable.

## Why
Complete the channel ownership slice of [tracker
#2](https://github.com/TheSentinel454/buzz/issues/2) and [domain issue
#7](https://github.com/TheSentinel454/buzz/issues/7) while preserving
the runtime/store boundary established by #6660 and #6668. This child
stacks on the test-only fixture prerequisite #6819 above #6777 and
carries forward PR #6700's membership/replacement lock timing without
changing lock or transaction behavior.

## What
- Keep channel lifecycle, metadata, TTL advisory locking, and lifecycle
tests in `channel.rs`
- Move membership/roster records, SQL, advisory-lock helpers, `Db`
methods, focused tests, and datastore spans to a dedicated
`channel_members.rs`
- Preserve existing `buzz_db::channel::*` paths with compatibility
re-exports while exposing the dedicated module
- Move the four roster-fence PostgreSQL tests out of `lib.rs`

## Stack
- Exact base: codex/issue-7-roster-test-fixtures at
21d1b265c133292e6707e766cd4204e6a43f08af
([#6819](https://github.com/block/buzz/pull/6819))
- Exact head: codex/issue-7-channel-membership-store at
25138bfd6588e046170dbdbc4ed953bdc3cf7ed1
- Tracker: https://github.com/TheSentinel454/buzz/issues/2
- Domain: https://github.com/TheSentinel454/buzz/issues/7
- Test/span acceptance: https://github.com/TheSentinel454/buzz/issues/17
and https://github.com/TheSentinel454/buzz/issues/19

## Non-goals
- No SQL, schema, retry, timeout, lock ordering, transaction boundary,
or client-visible behavior changes
- No change to channel TTL lifecycle ownership merely because lifecycle
bootstrap writes an owner membership row
- No store traits, domain-handle redesign, broad `PgExecutor` migration,
raw pool accessor, new crate, or directory-wide reorganization
- No changes to, retargeting of, or merge action on PR #6700 or #6777

## Risk Assessment
Moderate review surface, low semantic risk. The file split is large, but
method signatures, SQL, bind order, membership and replacement lock
namespaces, transaction boundaries, and span names remain unchanged.
Compatibility re-exports preserve existing `buzz_db::channel::*`
consumers.

## Blox Verification
Author workstation: `buzz-tornquist-issue-2-store-stack` (`2046520`),
exact head `8376e19d0da3ec77550590cd91cc3dfe284d95d6`.

- `cargo fmt --all --check` — passed
- `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` —
passed
- Native PostgreSQL channel lifecycle suite — 3 passed
- Native PostgreSQL membership/roster suite — 17 passed; two
pre-existing ignored-test fixture failures reproduced identically on the
untouched parent `2de5444`:
`large_roster_reconciliation_candidates_respect_snapshot_count_and_signer`
and `locked_member_snapshot_blocks_post_capture_membership_mutation`
both receive the migration-0032 `23514` invalid-`p`-tag rejection. This
extraction intentionally does not fold a test-behavior fix into the
move.
- `cargo test -p buzz-relay --lib -- --test-threads=1` — 908 passed, 48
ignored; the existing load-sensitive mesh demo test returned 504,
matching the #6700/parent baseline
- `cargo test -p buzz-relay --lib
api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo --
--exact --test-threads=1` — passed

Independent exact-head review: `buzz-tornquist-pr-6782-review`
(`2048397`) found no remaining critical, important, or minor issues. The
full implementation review also independently reproduced both stated
PostgreSQL fixture failures on the exact parent and passed the relay
library suite (909 passed, 48 ignored).

Generated with Codex

## Superseded pre-comment restack verification

PR #6700 merged before publication completed. This layer was restacked
onto current main through the exact parent named above; the final
cumulative tip is 2ddcc8a29f95c8c8c9cb87bb6cf59335f2190fae. Cumulative
author gates passed: formatting and diff checks; buzz-db and buzz-relay
all-target clippy with -D warnings; DB lib 111 passed / 200 ignored;
ownership 22/22; observability 1/1; the full isolated PostgreSQL domain
matrix; and relay lib 910 passed / 49 ignored.

- Workstation: `buzz-tornquist-pr-6782-final-review` (`2057620`), fresh
shallow checkout
- Base: `c60e793eadde79d9eab9f48bbb2ede0ad4831f9b`
- Head: `fa09b6c81c4db3b3e1940a2117a97ab2186e49f7`
- Findings: none

Reviewed both commits in `base..head`. Channel lifecycle/metadata, TTL
transitions, and their lock rationale remain in `channel.rs`; membership
authorization, roster fencing/snapshots, membership advisory locking,
membership records, and focused tests move together to
`channel_members.rs`. SQL, transaction, and lock sequences are
preserved.

Verification: format and diff checks passed; `buzz-db --all-targets`
clippy passed with `-D warnings`; DB lib tests passed (111 passed, 200
PostgreSQL tests ignored); ownership (2/2) and observability (1/1)
guards passed; native PostgreSQL 17 passed 3 channel lifecycle tests
plus 19 membership/roster tests with migrations 1-32 successful; relay
lib test target compiled successfully. Final worktree was detached at
the exact head and clean.

Complete evidence archive SHA-256:
`81ae374095f649ca7a25d8b9a4fc864257b7925d1b44657a69ca111523adf36e`.

## Comment-addressed restack

Review follow-up on #6777 removed only the low-value replaceable
ownership source test. This PR was restacked onto its rewritten parent;
its production patch is unchanged.

- Exact base: `21d1b265c133292e6707e766cd4204e6a43f08af`
- Exact head: `25138bfd6588e046170dbdbc4ed953bdc3cf7ed1`
- Final cumulative tip: `6fa2f104d42c6ba85bdf62e7ccb74ceaf4a84f67`
- Per-layer patch-ID and tree audits confirm this PR’s production diff
is unchanged from its pre-comment head.
- Cumulative Blox gate: formatting and diff checks; strict
`buzz-db`/`buzz-relay` Clippy; DB lib 111 passed / 200 ignored;
ownership 21/21; observability 1/1; every moved PostgreSQL test; relay
lib 910 passed / 49 ignored.
- Independent re-review at this exact head: no findings; fresh
exact-parent/head Blox review passed fmt/diff, strict Clippy, DB lib 111
passed / 200 ignored, current ownership/observability guards, 3 channel
plus 19 membership PostgreSQL tests, and relay compilation.

Signed-off-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-28 10:55:48 -04:00
Luke Tornquist
57216c942f test(db): use canonical channel roster fixtures (#6819) 
## Summary

Fix two PostgreSQL channel-roster test fixtures so they satisfy
migration 0032's canonical kind-39002 fence before the
channel/membership store extraction moves them.

This is a test-only prerequisite in the issue #2 extraction stack. It
emits the canonical four-field `p` tags with authoritative roles, keeps
the stale-history mutation after insertion, and gives the colliding
other-tenant fixture its own matching roster.

## Stack

- Exact base: codex/issue-6-finish-replaceable-store at
da018405cc
([#6777](https://github.com/block/buzz/pull/6777))
- Exact head: codex/issue-7-roster-test-fixtures at
21d1b265c133292e6707e766cd4204e6a43f08af
- Structural tracker: TheSentinel454/buzz#2
- Domain issue: TheSentinel454/buzz#7
- Acceptance: TheSentinel454/buzz#17 and TheSentinel454/buzz#19
- Next: `codex/issue-7-channel-membership-store`

## Why separate

The existing ignored tests are red against a freshly migrated current
`main`: migration 0032 rejects their legacy two-field roster tags before
the assertions run. The extraction PR is intended to be a pure move, so
this fixture correction is isolated here rather than mixed into the
channel-membership ownership diff.

## Non-goals

- No production Rust or SQL changes.
- No schema, lock, transaction, timeout, retry, API, or client-visible
behavior changes.
- No change to migration 0032's canonical roster rules.
- No store ownership movement; that remains in the child PR.

## Risk

Low and test-only. The main risk is accidentally changing the scenario
rather than only its representation. The tests retain the same member
counts, stale-versus-complete distinction, tenant collision, signer
isolation, and lock-freshness assertions.

## Blox verification

Author workstation: `buzz-tornquist-issue-2-store-stack` (`2046520`),
native PostgreSQL 17.11.

- Red before the change on a freshly migrated database: both focused
tests failed with migration 0032's `kind 39002 roster contains an
invalid p tag` constraint.
- `cargo fmt --all --check`: pass
- `git diff --check`: pass
- `cargo clippy -p buzz-db --all-targets -- -D warnings`: pass
-
`channel::tests::large_roster_reconciliation_candidates_respect_snapshot_count_and_signer`:
pass
-
`channel::tests::locked_member_snapshot_blocks_post_capture_membership_mutation`:
pass
- Cumulative exact-tip PostgreSQL matrix: pass

## Superseded pre-comment restack verification

PR #6700 merged before publication completed. This layer was restacked
onto current main through the exact parent named above; the final
cumulative tip is 2ddcc8a29f95c8c8c9cb87bb6cf59335f2190fae. Cumulative
author gates passed: formatting and diff checks; buzz-db and buzz-relay
all-target clippy with -D warnings; DB lib 111 passed / 200 ignored;
ownership 22/22; observability 1/1; the full isolated PostgreSQL domain
matrix; and relay lib 910 passed / 49 ignored.

- Workstation: `buzz-tornquist-pr-6819-final-review` (`2057618`), fresh
shallow checkout
- Base: `ffbeaaf00810aa359ab85818ff4820f92263e45f`
- Head: `c60e793eadde79d9eab9f48bbb2ede0ad4831f9b`
- Findings: none

Reviewed the test-only roster-fixture correction. The affected
large-roster and owner fixture events now use canonical four-field `p`
tags and preserve the intended roster cardinalities. The distinct
other-community fixture remains isolated and the production
implementation, SQL, spans, and API are untouched.

Verification: format and diff checks passed; `buzz-db --all-targets`
clippy passed with `-D warnings`; DB lib tests passed (111 passed, 200
PostgreSQL tests ignored); ownership (1/1) and observability (1/1)
guards passed; all 18 channel PostgreSQL tests passed on native
PostgreSQL 17 with migrations 1-32 successful. Final worktree was
detached at the exact head and clean.

Complete evidence archive SHA-256:
`da8379f4f0ae3989eb3573162f0f19cbde733400000756ab0f63af3322244d4b`.

## Comment-addressed restack

Review follow-up on #6777 removed only the low-value replaceable
ownership source test. This PR was restacked onto its rewritten parent;
its production patch is unchanged.

- Exact base: `da018405cc83605362125c2d5e5a3f91492431ac`
- Exact head: `21d1b265c133292e6707e766cd4204e6a43f08af`
- Final cumulative tip: `6fa2f104d42c6ba85bdf62e7ccb74ceaf4a84f67`
- Per-layer patch-ID and tree audits confirm this PR’s production diff
is unchanged from its pre-comment head.
- Cumulative Blox gate: formatting and diff checks; strict
`buzz-db`/`buzz-relay` Clippy; DB lib 111 passed / 200 ignored;
ownership 21/21; observability 1/1; every moved PostgreSQL test; relay
lib 910 passed / 49 ignored.
- Independent re-review at this exact head: no findings; fresh
exact-parent/head Blox review passed fmt/diff, strict `buzz-db` Clippy,
DB lib 111 passed / 200 ignored, observability, and all 18 channel
PostgreSQL tests; relay compilation was not applicable to this test-only
layer.

Signed-off-by: tornquist <tornquist@squareup.com>
 2026-08-27 16:07:00 -04:00
Luke Tornquist
0ccf934b88 fix(db): disable heartbeat vacuum truncation (#6898) 
## Why
`replica_heartbeat` is updated continuously, but autovacuum heap
truncation can take an `ACCESS EXCLUSIVE` lock whose replay cancels
concurrent hot-standby reads. Truncating a fixed, single-row table
provides no meaningful space benefit.

## What
- Add migration 0034 to set `vacuum_truncate = false` on
`replica_heartbeat`
- Keep the desired-state schema aligned for fresh installations
- Extend migration coverage so both paths retain the reloption

## Risk Assessment
Low — this disables only autovacuum's heap-truncation phase for a
single-row table; normal vacuum cleanup remains enabled. The tradeoff is
retaining at most the table's otherwise-truncatable tail pages.

## References
- [Investigation
thread](https://sq-block.slack.com/archives/C0B2WT43BDH/p1787760831375989)
- [PostgreSQL table storage
parameters](https://www.postgresql.org/docs/current/sql-createtable.html#SQL-CREATETABLE-STORAGE-PARAMETERS)

Generated with Codex

---

**Update Aug 27, 2026:** Hardened fresh `pgschema` bootstraps after
review.

- Renamed the shared post-apply script to
`reconcile-schema-after-pgschema.sql` and made it restore the heartbeat
reloption and singleton row idempotently.
- Added live catalog and row assertions that stop bootstrap when the
database does not match the required state.
- Updated every repository `pgschema apply` caller, including the
desktop release smoke path.
- Added a regression test that requires future `pgschema apply` callers
to run reconciliation.
- Documented the `pgschema` DML and storage-parameter limitation in
`AGENTS.md` and at the reconciliation entry point.
- Merged current `main` and renumbered this migration to 0034 after
`main` added migration 0033.

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
 2026-08-27 13:01:13 -04:00
Tyler
b3baa56ba8 fix(db): exclude kind:30179 ciphertext from brownfield FTS (#6822) 
## Problem

Kind `30179` (NIP-PMA private managed agent — NIP-44 ciphertext carrying
the agent nsec, env vars, prompt) joined `AUTHOR_ONLY_KINDS` in #4593,
but only **fresh** installs stopped indexing it:

- `migrations/0008` installs the positive FTS allowlist only when
`events` is empty;
- `migrations/0014` wraps the retained brownfield expression for `30350`
alone;
- `schema/schema.sql` still carries the legacy negative skip-set without
`30179`.

So a relay upgraded in place keeps tokenizing `30179` ciphertext into
`events.search_tsv`. Found by Wren while reviewing #4999 (which
activates publication of 30179 from Desktop). No readable leak: `/query`
applies `search_hit_accepted` + `event_visible_to_reader` before
serialization and live foreign searches against builderlab returned `[]`
— this is the storage-layer privacy invariant (`docs/nips/NIP-PMA.md`
deployment step 2) plus wasted FTS work.

## Fix

- `migrations/0033_private_managed_agent_fts.sql` — same shape as 0014:
capture the current generated expression via `pg_get_expr`, drop/re-add
`search_tsv` wrapped with `kind = 30179 → NULL`. Every other kind keeps
whatever policy the database already had (fresh allowlist or brownfield
skip-set). Rebuilds the GIN index; no heap-wide policy rewrite.
- `schema/schema.sql` — add `30179` to the desired-state skip-set.

## Tests

- `populated_upgrade_preserves_search_policy_except_for_private_kinds`
(renamed from `…_push_leases`): provisions the **legacy negative
expression** (migrations 1–7), inserts kind 1 / 30179 / 30350 rows,
checkpoints after 0014 (30179 still searchable, 30350 not), then runs to
head and asserts 30179 is NULL. **Proven red** with the 0033 file
removed: `left: [(1, Some(true)), (30179, Some(true)), (30350, None)]`.
- `embedded_migrator_contains_consolidated_initial_schema`: count 32→33,
asserts 0033 shape and `schema.sql` parity.
- `buzz-search/tests/fts_integration.rs` setup applies 0033 so the FTS
tripwires run against the real chain.

## Verification (HEAD 6154cc33d, same shell)

- `cargo test -p buzz-db -- --include-ignored --test-threads=1`: 305
passed / 3 failed — the 3
(`create_community_with_owner_enforces_per_owner_limit`,
`insert_mentions_indexes_rosters_past_bind_parameter_cap`,
`transfer_ownership_returns_limit_reached_for_maxed_transferee`) fail
identically on pristine `main` f24971033 with a fresh DB; unrelated to
this diff.
- `cargo test -p buzz-search -- --include-ignored`: 19 passed (incl.
`author_only_kinds_are_storage_level_unsearchable`,
`p_gated_persistent_kinds_have_storage_null_tsvector`).
- `desired_state_schema_bootstrap_progresses_beyond_fencing` +
`run_migrations_applies_consolidated_initial_schema_on_fresh_database`:
pass.
- `cargo fmt --all --check` clean; `cargo clippy -p buzz-db -p
buzz-search --all-targets -D warnings` clean.

Independent of #4999 — both main and #4999 are brownfield-exposed today;
this lands either order.

---------

Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
 2026-08-27 10:47:35 -04:00
thomaspblock
de188ebf06 feat(projects): add agent and CLI project-home support (#6590) 
## Summary
- inject bounded project-home identity and repository context into
managed agent sessions
- add project-aware CLI flows for creating projects, repositories,
issues, and related channels
- match project homes through the existing relay query surface, then
filter channel metadata client-side

This is Part 1 of the channel-first Projects stack. Part 2 contains
project creation and model foundations.

## Testing
- `cargo fmt --all -- --check`
- `cargo clippy -p buzz-cli -p buzz-acp --all-targets -- -D warnings`
- `cargo test -p buzz-cli -p buzz-acp` — 1,184 tests passed, 1 doc test
ignored
- full pre-push gate passed

## Post-Deploy Monitoring & Validation
- validate project-home agent context and project-aware CLI commands
against a staging relay
- healthy signals: project context matches the active channel, explicit
repo coordinates remain stable, and normal channels receive no project
block
- failure signals: cross-channel project context, duplicate project
creation, or commands targeting an unrelated repository; mitigate by
reverting this PR

---------

Signed-off-by: Thomas Petersen <thomasp@squareup.com>
Signed-off-by: Wrench <0eabe6ea5758c1e4c5b68cea4ac42b32c479072883cb28da8110b4a47c32b9a1@buzz.block.builderlab.xyz>
Co-authored-by: Wrench <0eabe6ea5758c1e4c5b68cea4ac42b32c479072883cb28da8110b4a47c32b9a1@buzz.block.builderlab.xyz>
 2026-08-26 12:09:06 -07:00
Luke Tornquist
ef0d202568 refactor(db): finish replaceable event store extraction (#6777) 
## Why
Finish the replaceable-event slice of [tracker
#2](https://github.com/TheSentinel454/buzz/issues/2) and [domain issue
#6](https://github.com/TheSentinel454/buzz/issues/6) without disturbing
the runtime/store boundary established by #6660 and #6668. PR #6700 has
merged; this PR now targets current main containing its replacement lock
and transaction observability.

## What
- Move `Db::replace_addressable_event`, its SQL, lock/transaction
instrumentation, and focused addressable/parameterized replacement tests
from `lib.rs` to `replaceable.rs`
- Preserve the transaction-required parameterized API, replacement
ordering, rollback semantics, mention indexing, and exactly one
datastore span per public operation
- Keep implementation and focused PostgreSQL tests co-located; follow-up
review removed the dedicated replaceable ownership source guard as low
value

## Stack
- Exact base: main at f249710331
- Exact head: codex/issue-6-finish-replaceable-store at
da018405cc
- Tracker: https://github.com/TheSentinel454/buzz/issues/2
- Domain: https://github.com/TheSentinel454/buzz/issues/6
- Test/span acceptance: https://github.com/TheSentinel454/buzz/issues/17
and https://github.com/TheSentinel454/buzz/issues/19

## Non-goals
- No SQL, schema, retry, timeout, lock ordering, transaction boundary,
or client-visible behavior changes
- No store traits, domain-handle redesign, `PgExecutor` migration, raw
pool accessor, new crate, or directory reorganization
- No changes to, retargeting of, or merge action on PR #6700

## Risk Assessment
Low. This is a mechanical ownership move with unchanged signatures and
SQL. The primary review risk is losing or nesting instrumentation,
covered by exact moved PostgreSQL tests and the cumulative
production-diff audit.

## Blox Verification
Author workstation: `buzz-tornquist-issue-2-store-stack` (`2046520`),
exact head `2de5444e14606ce6912d1bb932bc88bffa379a77`.

- `cargo fmt --all --check` — passed
- `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` —
passed
- `cargo test -p buzz-db --quiet` — 108 passed, 200 ignored; both
integration source guards passed
- Native PostgreSQL: `cargo test -p buzz-db replaceable::tests:: --
--ignored --test-threads=1` — 11 passed
- `cargo test -p buzz-relay --lib
api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo --
--exact --test-threads=1` — passed
- Full serial relay suite was also exercised: 908 tests passed but the
existing load-sensitive mesh demo test returned 504 under suite load,
matching PR #6700's documented baseline; it passed in isolation and this
diff does not touch that subsystem

Independent exact-head Blox review: `buzz-tornquist-pr-6777-review`
(`2047422`) at `2de5444e14606ce6912d1bb932bc88bffa379a77`.

- Review findings — none (critical, important, or minor)
- `cargo fmt --all --check` and the ownership guard — passed
- `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` —
passed
- `cargo test -p buzz-db --quiet` — 108 passed, 200 ignored; source
guards passed
- Native PostgreSQL moved suite — 11 passed
- `cargo test -p buzz-relay --lib` — 909 passed, 48 ignored

Generated with Codex

## Superseded pre-comment restack verification

PR #6700 merged before publication completed. This layer was restacked
onto current main through the exact parent named above; the final
cumulative tip is 2ddcc8a29f95c8c8c9cb87bb6cf59335f2190fae. Cumulative
author gates passed: formatting and diff checks; buzz-db and buzz-relay
all-target clippy with -D warnings; DB lib 111 passed / 200 ignored;
ownership 22/22; observability 1/1; the full isolated PostgreSQL domain
matrix; and relay lib 910 passed / 49 ignored.

- Workstation: `buzz-tornquist-pr-6777-final-review` (`2057617`), fresh
shallow checkout
- Base: `f24971033178926153b49d320bd876d15d9cb2bf`
- Head: `ffbeaaf00810aa359ab85818ff4820f92263e45f`
- Findings: none

Reviewed `base..head` for the replaceable-store extraction. The
addressable replacement SQL, stale/duplicate outcomes, advisory-lock
ordering, transaction rollback/commit boundaries, mention-index
atomicity, and public `Db` signature are retained in `replaceable.rs`.
The PR #6700 transaction timer remains around the same logical
transaction through `TransactionTimer::observe`, and the public
operation has exactly one datastore span. Focused addressable and
parameterized tests moved with the implementation; the ownership guard
excludes duplicates from `lib.rs`.

Verification: format and diff checks passed; `buzz-db --all-targets`
clippy passed with `-D warnings`; DB lib tests passed (111 passed, 200
PostgreSQL tests ignored); ownership (1/1) and observability (1/1)
guards passed; all 11 replaceable PostgreSQL tests passed on native
PostgreSQL 17 with migrations 1-32 successful; relay lib test target
compiled successfully. Final worktree was detached at the exact head and
clean.

Complete evidence archive SHA-256:
`e37f92bbde4e2343196085095b3093b1bc78196be58bc86826325e0e1639d7ea`.

## Comment-addressed restack

Review follow-up removed the dedicated
`replaceable_store_has_single_ownership` source test as requested. No
production code changed.

- Exact base: `f24971033178926153b49d320bd876d15d9cb2bf`
- Exact head: `da018405cc83605362125c2d5e5a3f91492431ac`
- Final cumulative tip: `6fa2f104d42c6ba85bdf62e7ccb74ceaf4a84f67`
- Focused PR verification: formatting and diff checks passed; strict
`buzz-db`/`buzz-relay` Clippy passed; DB lib 111 passed / 200 ignored;
all 11 replaceable PostgreSQL tests passed.
- Cumulative Blox gate: formatting and diff checks; strict
`buzz-db`/`buzz-relay` Clippy; DB lib 111 passed / 200 ignored;
ownership 21/21; observability 1/1; every moved PostgreSQL test; relay
lib 910 passed / 49 ignored.
- Independent re-review at this exact head: no findings; fresh
exact-parent/head Blox review passed fmt/diff, strict `buzz-db` Clippy,
DB lib 111 passed / 200 ignored, manual one-owner/one-span replacement
checks, observability, 11 replaceable PostgreSQL tests, and relay
compilation.

---------

Signed-off-by: tornquist <tornquist@squareup.com>
 2026-08-26 10:33:09 -04:00
Luke Tornquist
113a33b7e4 Add database pressure observability (#6700) 
## Why

Database pressure currently collapses several distinct delays into one
symptom. This adds the evidence layer needed to distinguish pool
acquisition wait, logical database operation time, advisory-lock wait,
and selected transaction duration before changing timeout or retry
policy.

This is the phase 2 Lane A observability bundle for
[#26](https://github.com/TheSentinel454/buzz/issues/26),
[#28](https://github.com/TheSentinel454/buzz/issues/28), and
[#33](https://github.com/TheSentinel454/buzz/issues/33). It is stacked
on #6668.

## What

- Record explicit reader/writer checkout wait and acquisition outcomes
with `buzz_db_pool_acquire_wait_seconds` and
`buzz_db_pool_acquisitions_total`.
- Extend the compile-time `#[datastore_span(name = "...")]` seam with
`buzz_db_operation_duration_seconds`, so operation labels remain static
source literals instead of request data.
- Route correctness-critical replacement, membership, push-gate,
deletion, and migration/schema-safety advisory locks through one
observer without changing their SQL, order, scope, or blocking behavior.
- Measure six internally owned transaction lifetimes with
`buzz_db_transaction_duration_seconds`, starting after `BEGIN` succeeds
and ending after explicit commit/rollback or scope exit.
- Emit root slow-operation warnings at 500 ms, logging the first slow
completion and then 1/100 per call site with only `operation`,
`outcome`, and `elapsed_ms`.
- Document names, units, fixed label vocabularies, measurement
boundaries, and blind spots in this PR description.

Fixed labels are deliberately small:

- `pool_role`: `writer`, `reader`
- `lock_type`: `replacement`, `membership`, `push_gate`, `deletion`,
`migration_schema_safety`
- `outcome`: `success`, `error`, `timeout` where SQLx/PostgreSQL can
distinguish it accurately
- `operation`: compile-time datastore names plus the six closed
transaction operation names documented in the runbook

No metric or slow warning contains community IDs, event IDs, event
kinds, coordinates, d-tags, SQL/query text, query IDs, returned errors,
or event content.

## Coverage boundaries

- Operation duration is the complete annotated logical function body,
not pure SQL execution; it may include implicit checkout, lock wait,
nested operations, and application work. Cancelled futures do not reach
its completion hook.
- Pool timing covers explicit helper checkouts, including proved-reader
routing and selected writer-owned transactions. Implicit SQLx checkout
through `&PgPool` remains folded into operation duration.
- Lock timing covers application-side blocking locks in the five named
families. Trigger/stored-procedure locks, channel-TTL locking, the usage
try-lock, and the audit service session lock remain outside this slice.
- Transaction timing covers only the six wholly owned boundaries
documented in the runbook. It excludes pool wait, `BEGIN`, asynchronous
rollback cleanup after an early return, and caller-owned
`Db::begin_transaction` lifetime.

## Relationship to #6229

#6229 is the incident-driven timeout precursor. This PR does not add or
change `statement_timeout`, `lock_timeout`,
`idle_in_transaction_session_timeout`, retries, audit durability, or
client-visible conflicts. It provides the missing distributions needed
to evaluate those policies later and intentionally leaves #6229's open
audit retry/durability finding untouched.

The branches overlap in `crates/buzz-db/src/lib.rs` and
`crates/buzz-db/src/migration.rs`, so a later rebase may need textual
conflict resolution, but the behavior is complementary rather than
duplicated.

## Risk assessment

Moderate-low. The primary risk is instrumentation overhead and added
static series. Cardinality is source-bounded, slow logs are
sampled/redacted root events, and the lock/transaction changes wrap
existing awaits without changing policy or ordering.

## Verification

Author workstation: `buzz-tornquist-db-pressure-observability`
(`2010927`), exact head `d7cf833e26c528adfcde3917ded80daf6f4ddac9`,
parent `6f50e6b2b2a996349149af61d35bdd6a355f77fd`.

- `cargo fmt --all --check` — passed
- `cargo clippy -p buzz-datastore-tracing -p buzz-db -p buzz-audit -p
buzz-search -p buzz-relay --all-targets -- -D warnings` — passed
- `cargo test -p buzz-datastore-tracing --quiet` — 4 passed
- `cargo test -p buzz-db --quiet` — 109 passed, 200 ignored
- `cargo test -p buzz-audit -p buzz-search --quiet` — 16 passed, 25
ignored
- `cargo test -p buzz-relay --lib --quiet -- --test-threads=1` — 906
passed, 48 ignored
- Native PostgreSQL focused tests for pool success/timeout/error, lock
success/contention/timeout/error, replacement, membership serialization,
push ordering, deletion fencing, migration/schema exclusion, and reader
fallback — 8 passed

The default-parallel relay run passed once; subsequent runs exposed the
existing load-sensitive
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` 504 at
the end of the suite. That test passes in isolation and the full relay
suite passes serially.

Independent exact-head review workstation:
`buzz-tornquist-db-pressure-observability-review` (`2013067`).
Formatting, the same all-target clippy command, datastore
instrumentation tests, DB unit tests, source privacy guards, and
diff/non-goal audits passed; no review findings.

Generated with Codex

---------

Signed-off-by: tornquist <tornquist@squareup.com>
 2026-08-25 13:00:22 -04:00
Brad Seiler
d12dea4e67 Support community deletion in versioned media buckets (#6738) 2026-08-25 09:02:18 -07:00
Luke Tornquist
9d1e4b2576 Extract community persistence (#6668) 
## Why

Community persistence is the next incremental `buzz-db` store
extraction, keeping tenant lifecycle SQL, records, tests, and
instrumentation out of the database runtime module without changing
behavior.

## What

- Move community records and the existing `impl Db` operations into
`community.rs` while preserving crate-root re-exports.
- Move focused PostgreSQL tests with the implementation and enforce
single ownership for each method and datastore span.

## Risk Assessment

Low — this is a structural move of the existing records, SQL, method
bodies, and focused tests; database runtime concerns, schema, and
behavior remain unchanged.

## References

- Architecture guardrail:
https://github.com/TheSentinel454/buzz/issues/34
- Incremental tracker: https://github.com/TheSentinel454/buzz/issues/2
- Primary task: https://github.com/TheSentinel454/buzz/issues/5
- Stacked on: https://github.com/block/buzz/pull/6660

Generated with Codex

Signed-off-by: tornquist <tornquist@squareup.com>
 2026-08-25 10:05:32 -04:00
Luke Tornquist
8d2d0ff5ad Centralize replaceable event persistence (#6660) 
## Why
Command persistence duplicated NIP-33 replacement SQL in the relay and
obscured the boundary between database runtime concerns and domain-store
behavior. This proving slice establishes that boundary inside the
existing `buzz-db` crate.

## What
- Centralize parameterized-replaceable coordinate locking, ordering,
replacement, watermark, and mention-indexing behavior in `buzz-db`
- Expose transaction-required replacement and ordinary-insertion seams
while keeping transaction ownership and workflow conflict messages in
the relay
- Cover concurrency, same-second ties, stale writes, replay, caller
rollback, nested rollback recovery, and mention-index atomicity with
focused PostgreSQL tests

## Risk Assessment
Medium — this changes core event persistence and command idempotency
paths, while intentionally preserving NIP-33, NIP-RS, mesh, and workflow
conflict semantics.

## References
- Architecture guardrail:
https://github.com/TheSentinel454/buzz/issues/34
- Proving slice: https://github.com/TheSentinel454/buzz/issues/3,
https://github.com/TheSentinel454/buzz/issues/4,
https://github.com/TheSentinel454/buzz/issues/6

Generated with Codex

---------

Signed-off-by: tornquist <tornquist@squareup.com>
 2026-08-24 20:42:15 -04:00
Wes
24ec6a468e Repair stale large channel roster snapshots (#6251) 
## Summary

- detect relay-authored NIP-29 kind 39002 roster snapshots truncated by
the former 1,000-member query cap and repair stale large rosters during
relay startup
- serialize canonical roster capture and replacement with membership
writes, preserving tenant, channel, signer, pubkey, and role boundaries
through mixed-version deployments
- install migration 0032's fail-closed roster fence on the partitioned
events table and verify its catalog shape plus behavior before opening
relay listeners

## Rollout

Migration 0032 is a hard schema-before-code compatibility boundary.
Apply migrations before rolling this relay version. Startup refuses to
open listeners when the parent/partition triggers are missing, disabled,
mis-shaped, or behaviorally inert. For large installations, prefer
`buzz-admin migrate` and monitor lock acquisition as documented in the
chart README.

## Validation

Exact head: `bcbba271f54bc0046a6683007e5a2b70403a11d5`

- rebased onto `569308c23c9c2bf620dd3a9a5e4baecbcfa22e16`; the nine-file
feature patch is byte-identical to pre-rebase head
`be8ea0084f4d4c78c7c2550baad4399e4df8ce73`
- pre-push hook passed at exact head: branch-skew, file-size, full Rust
unit suite, Desktop Tauri clippy, and Desktop Tauri tests
- `cargo fmt --all -- --check`
- `cargo test -p buzz-relay
group_members_snapshot_keeps_members_past_one_thousand -- --nocapture`
- focused CI-mode Playwright regression: `selected relay agents revoked
after the invite prompt cause no side effects` passed at exact head
- prior exact-patch validation:
`large_roster_reconciliation_candidates_respect_snapshot_count_and_signer`,
mixed-writer locking/rollback, migration admission, partition trigger
coverage, and desired-schema parity regressions

## Review

Independent DB/relay review found no blocking issues in the exact
feature patch. The concurrency fence holds the established replacement
and membership locks on one transaction/connection through replacement;
failures roll back both soft-delete and insert. Reconciliation remains
tenant/channel/signer scoped and validates exact normalized
pubkey-plus-role membership.

The prior red Desktop shard was unrelated to this backend-only diff: its
mocked mention test exercises no relay, database, or migration path. It
reproduced as a timing flake on the old head, passed on retry/base, and
now passes locally after rebasing onto current main.

---------

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz>
 2026-08-20 21:22:41 -06:00
Taylor Ho
f7a01bda7b fix(workflows): preserve multi-channel listing semantics (#6009) 
**Category:** fix
**User Impact:** Workflow listings reliably include every accessible
channel, including for users with more than 128 memberships and when
connected to older relays.
**Problem:** Multi-value `#h` filters could lose live delivery, apply
channel scoping after SQL limits, mishandle partial authorization or
revocation, and permit unbounded membership work. Desktop also submitted
every channel in one request, exceeding the relay's new 128-value safety
bound.
**Solution:** Preserve NIP-01 OR semantics across relay query, count,
and live-subscription paths while enforcing authorization and bounded
explicit-channel work before database or Redis operations. Desktop keeps
the older-relay-compatible one-channel-per-filter shape, sends filters
in bounded batches, combines responses, and deduplicates signed events
by event ID.

<details>
<summary>File changes</summary>

**crates/buzz-db/src/event.rs**
Distinguishes authorization channel scopes from explicit `#h` scopes in
list and count SQL so requested channels are applied before limits
without implicitly including global rows.

**crates/buzz-relay/src/handlers/req.rs**
Shares explicit-channel scope extraction and limits, preserves valid OR
siblings when malformed branches cannot match, repairs request-local
membership misses, and registers authorized live subscriptions per
channel.

**crates/buzz-relay/src/handlers/count.rs**
Applies the same bounded explicit-channel authorization to COUNT and
preserves channel scope when a multi-channel request narrows to one
authorized channel.

**crates/buzz-relay/src/api/bridge.rs**
Brings HTTP query and count behavior in line with WebSocket semantics
before SQL execution and rejects over-limit explicit-channel requests
before membership I/O.

**crates/buzz-relay/src/subscription.rs**
Indexes multi-channel subscriptions by every authorized channel and
shrinks, rather than destroys, their scope when one channel is revoked.

**crates/buzz-relay/src/handlers/side_effects.rs**
Releases only revoked channel topics and sends terminal closure only
when no authorized channel remains.

**crates/buzz-test-client/tests/e2e_relay.rs**
Adds ignored relay integration coverage for multi-channel delivery and
valid historical/live behavior with malformed or empty OR siblings.

**desktop/src-tauri/src/commands/workflows.rs**
Builds one single-channel filter per membership, submits at most 128 per
relay request, combines batches, and deduplicates by immutable signed
event ID.

**desktop/src-tauri/src/commands/workflows_tests.rs**
Covers filter compatibility, malformed input, 129-channel batching, and
cross-batch event-ID deduplication.

</details>

## Reproduction steps

1. Join multiple channels containing workflows, open **Workflows**, and
confirm workflows from every accessible channel appear.
2. Repeat with more than 128 memberships and confirm the listing remains
complete rather than failing the relay request.
3. Send a multi-value `#h` query/count and confirm only requested
authorized channels affect SQL limits and counts.
4. Subscribe to channels A and B, revoke A, and confirm B continues
delivering live events.
5. Subscribe with a valid channel branch plus a malformed or empty `#h`
sibling and confirm valid history, EOSE, and post-EOSE live delivery
still occur.

## Validation

At pushed head `c419a923f05e483ab26c006a0b3a80cfb3c73844`:

- Relay request tests: 53 passed.
- Desktop full Rust unit suite: 2,468 passed, 17 ignored.
- Relay E2E target compiled with `--no-run`.
- Strict relay clippy passed.
- Desktop Tauri clippy/check passed.
- Pre-push Rust tests and Desktop Tauri checks passed.
- Rust formatting and `git diff --check` passed.

---------

Signed-off-by: Taylor Ho <taylorkmho@gmail.com>
 2026-08-17 15:16:14 -07:00
Wes
e0940927ff fix(channels): return complete member rosters (#5765) 
## Summary

- return complete channel rosters instead of truncating at 1,000 members
- chunk `event_mentions` inserts inside one transaction so large kind
`39002` snapshots remain discoverable by every `p` tag
- add a targeted `buzz-admin reconcile-channels --channel <uuid>`
force-republish path for stale discovery snapshots
- cover a 1,501-member roster, 11,000-tag mention index, and kind
`39002` tag construction past member 1,000

## Why

The relay builds NIP-29 discovery and several authorization decisions
from `get_members()`, but that helper silently returned only the first
1,000 active members. Desktop then counted the truncated kind `39002`
event, while late members could be rejected by roster-scanning member
actions.

Removing the roster cap exposes PostgreSQL's 65,535 bind-parameter
ceiling in mention indexing, so the insert is chunked transactionally to
preserve all-or-nothing indexing.

The existing reconcilers only fill missing discovery events. The
targeted admin option bypasses the separately known 1,000-channel
reconciliation-list ceiling and replaces an existing channel snapshot
using the configured production relay key.

## Attribution

This supersedes and builds on #3166 by @LordMelkor. Thank you for
identifying the roster boundary and contributing the original
complete-roster and mention-index patch. The production roster/query
changes and the two PostgreSQL regressions retain that work's shape;
this PR rebases it onto current `main`, adds relay coverage, and adds
the targeted repair operation requested for rollout.

## Validation

Exact pushed head: `24d02e4f3824150ed84913c9d230e675502e5b12`

- `cargo check -p buzz-db -p buzz-admin`
- `cargo test -p buzz-db
channel::tests::get_members_returns_full_roster_beyond_1000 -- --ignored
--exact --nocapture`
- `cargo test -p buzz-db
feed::tests::insert_mentions_indexes_rosters_past_bind_parameter_cap --
--ignored --exact --nocapture`
- `cargo test -p buzz-relay --lib
handlers::side_effects::tests::group_members_snapshot_keeps_members_past_one_thousand
-- --exact`
- `cargo run -q -p buzz-admin -- reconcile-channels --help`
- mandatory pre-push hook: branch-skew, desktop checks/typecheck/tests,
mobile tests, Rust tests, and desktop Tauri checks all passed on the
pushed head

## Rollout

1. Deploy the relay/backend build.
2. Run `buzz-admin reconcile-channels --channel <general-channel-uuid>`
with `BUZZ_RELAY_PRIVATE_KEY` configured.
3. Verify the replacement kind `39002` roster count matches the active
database membership count.

No schema migration or desktop release is required.

Fixes #3156
Supersedes #3166

---------

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz>
 2026-08-13 14:48:04 -07:00
Wes
2693e0db1f Make workflow run history authoritative in Desktop (#5780) 
## Summary

- persist stable workflow run `error_code` values separately from human
diagnostics
- expose NIP-98 authenticated, channel-authorized run history and
approval reads with stable keyset pagination
- connect Desktop to those authoritative reads and return the
relay-created run ID on trigger
- show truthful loading, failure, and pending-trace states, and do not
render approval actions from non-actionable stored hashes

## Validation

- pre-push `branch-skew`, `desktop-typecheck`, `desktop-test`,
`rust-tests`, `desktop-tauri-checks`, and `desktop-check` all passed on
`a097dbe5f`
- Desktop tests: 4,761 passed, 0 failed
- `cargo check -p buzz-relay`
- `git diff --check`

## Remaining gate

This does not claim a relay-backed Playwright workflow journey. The
browser relay bridge still routes workflow invokes through in-memory
handlers; that production-shaped acceptance gate remains follow-up work
before Workflows can leave preview.

---------

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz>
Co-authored-by: Mongo <5c25403eab7271f9f94ddd4f2b270e8cac2c92e2c830c51877cca6ec974ffb3f@buzz.block.builderlab.xyz>
Co-authored-by: Princess Donut <68157ebd23b3897c1991015c3038658ea916200c67d3a54620b0754d1b92f6e0@buzz.block.builderlab.xyz>
 2026-08-13 12:56:37 -07:00
Kalvin C
8a2c9af2db feat(deletion): add durable whole-community deletion (#4425) 
## Summary

Adds a durable, operator-controlled V1 for deleting an entire Buzz
community without deleting another tenant's data.

The workflow is exposed through `buzz-admin deletions`:

- `sweep` records independent fleet storage-taxonomy observations
- `submit`, `list`, `inspect`, and `approve` manage a deletion request
- `unblock` resumes a fail-closed request after an operator records
remediation identity and reason
- `run` and `drain` execute bounded work

Requests advance through a PostgreSQL-backed state machine and stop at
`retention_pending` after logical deletion has been independently
verified across PostgreSQL, object storage, and Redis.

This PR ships the engine and CLI, not a continuously running worker or
Kubernetes packaging. For V1, a cluster/VM administrator invokes
`/usr/local/bin/buzz-admin` from the existing relay image, for example
with `kubectl exec` or an equivalent container/VM exec path.

## What whole-community V1 removes

For the target community, V1 removes:

- rows from the allowlisted community-scoped PostgreSQL catalog,
including members, profiles, authored events and bodies, DMs, reactions,
mentions, memberships, tokens, workflows, moderation, audit, feedback,
and rate-limit state
- media sidecars and upload-attribution records under
`_meta/<community>/` and `_uploads/<community>/`
- Git repository pointers under `repos/<community>/`
- Redis keys under `buzz:<community>:*`

The community row survives as a permanent tombstone, and deletion
control-plane records remain as evidence of the request, approval,
execution, and result.

## Safety model

Deletion is not a broad `DELETE CASCADE` followed by optimistic cleanup.
The destructive boundaries are durable and fail closed.

### 1. Inventory and approval

- `submit` resolves the target and freezes the schema plus summary-only
storage inventory.
- Approval is bound to the exact request, community, and frozen
inventory digest.
- Unsupported manifest versions, malformed keys inside the target's
owned prefixes, live scoped-table/write-fence coverage drift,
frozen-inventory mismatch, and approval mismatch block execution rather
than guessing. Migration and catalog revision numbers are not
authorization gates; the executor validates the live safety shape
instead.
- Storage inventory is server-side prefix scoped to exactly:
  - `_meta/<community>/`
  - `_uploads/<community>/`
  - `repos/<community>/`
- The deletion path never lists the whole shared bucket and has no
arbitrary per-community object cap. Its listing work is proportional to
the target community's bindings, not total fleet storage.
- Fleet-wide taxonomy sweeps remain independent observability. They
report unknown writer shapes but do not gate deletion submission,
fencing, or destructive progress. Maintainers must add deletion taxonomy
coverage whenever a new community-owned object-key class is introduced;
writer-coverage tests bind the current media and Git writers to that
contract.

### 2. Quiesce, fence, and destructive freeze

- Writes continue through submission, inventory, and approval. They stop
when execution moves the target into `quiescing` and then establishes
the durable fence.
- Already-admitted external effects finish under heartbeated
serving-write leases; the exact admitted lease may renew while the
community is quiescing, but new lease acquisition is rejected. The
executor drains admitted leases before destructive work.
- Invite minting after quiescing begins fails as typed `AccessDenied`
(HTTP 503 at the relay boundary) before an invite can be persisted.
- Database triggers enforce the community write fence across the
complete catalog of community-scoped tables. Startup/readiness and
destructive execution validate that catalog so a newly added but
unfenced table cannot silently escape.
- **Named isolation assumption — fresh write snapshot.** Every writer
transaction that can reach a community-fenced relation must use
PostgreSQL `READ COMMITTED`; each guarded write therefore observes a
statement snapshot no older than acquisition of the community deletion
lock. `REPEATABLE READ` and `SERIALIZABLE` can retain a pre-fence
snapshot and are unsupported for writers. The writer pool refuses
non-`READ COMMITTED` sessions at connection setup, and both SQL fence
functions reject an explicit per-transaction isolation override with
SQLSTATE `25000`. Configuration-delivered bad isolation can surface
through SQLx as a pool-acquire timeout because every `after_connect`
attempt is rejected; the precise `community writes require READ
COMMITTED isolation` reason remains observable when the SQL guard is
reached. Read-only replica transactions are outside this assumption.
- Holding the shared advisory lock until the guarded write executes is a
separate liveness condition: under `READ COMMITTED`, releasing it early
does not permit resurrection because the trigger rechecks the fence, but
it can turn a fleet sweep into a statement-wide SQLSTATE `55000` abort.
- After the fence closes writers, storage is re-enumerated into chunked
side-table rows. Per-prefix counts and digests bind those concrete keys
to the destructive manifest.
- Manifest chunk insertion, update, and deletion are protected after
freeze. This closes the race where an unbound key could otherwise appear
after the manifest was committed.

### 3. Checkpointed destruction

- Target-owned object bindings are deleted from the frozen destructive
manifest in bounded batches with durable progress.
- The concrete key list lives in chunked side-table rows rather than one
request-row JSON value. It supports large communities, resumable
execution, and terminal cleanup.
- Missing objects are accepted as idempotent crash-window outcomes;
malformed ownership, changed evidence, and unexplained target-prefix
drift fail closed.
- PostgreSQL purging remains scoped by `community_id`, including the
guarded NIP-RS hard-delete path discovered with real Desktop kind
`30078` read-state data.
- Redis cleanup explicitly scans and `UNLINK`s only
`buzz:<community_id>:*`. Natural expiry is insufficient because some
keys, including tunnel generation counters used as fencing state, are
deliberately persistent.

### 4. Independent verification

- PostgreSQL logical absence is checked after purge.
- The three target-owned storage prefixes are freshly inventoried again
and must be empty.
- Redis requires two complete empty namespace scans.
- Only after all three stores pass does the request advance through
`logically_verified` to `retention_pending`.

## What V1 deliberately does not erase

### Shared content-addressed storage

Per-community deletion removes bindings, metadata, attribution records,
and Git pointers. It does **not** physically delete fleet-shared CAS
bytes that another community may still reference:

- media blobs and thumbnails
- Git manifests, packs, and indexes (`manifests/`, `packs/`, and `idx/`)

Safe reclamation requires a separate fleet-wide reachability and
retention GC. Unknown keys elsewhere in the shared bucket do not block
one community's deletion; malformed or unrecognized keys inside that
community's three owned prefixes still fail closed.

### External retained copies

The online logical-deletion proof does not erase object
versions/replicas, database backups/WAL, CDN copies, provider retention
copies, or observability exports. Those require their own retention and
purge controls.

### Member-only erasure

This PR erases a whole community. It does not implement the different
operation "erase one npub while preserving the community."

Removing membership or accepting NIP-09 is not member erasure. A
member-only workflow would need to find and selectively remove or redact
authored event content and pubkeys, profile data, DMs, reactions,
mentions, memberships/roles, tokens, workflows/subscriptions, upload
attribution, moderation/audit history, repository attribution, and
identity embedded in tags or JSON. It would also need explicit rules for
ownership transfer, surviving replies and thread metadata, audit-chain
integrity, immutable Git history, and shared-CAS reachability. That
requires a pubkey-level fence and selective graph rewrite; it is a
separate deletion product, not a safe extension of this whole-tenant
worker.

## In scope

- migration `0029_community_deletion.sql`: requests, approvals, leases,
manifest chunks, checkpoints, tombstones, and the universal write-fence
catalog
- durable executor leases, generations, heartbeats, retry/block state,
and resumable stage transitions
- operator-driven `sweep`, `submit`, `list`, `inspect`, `approve`,
`unblock`, `run`, and `drain` commands
- serving-path fences for database writes and external effects across
event ingest, media, Git, workflow, push, invites, mesh/tunnel, and
related paths
- target-prefix-only storage inventory, summary manifests, post-fence
destructive chunks, and bounded batch deletion
- exact community Redis namespace purge and two-pass absence
verification
- cross-community isolation, crash/resume, manifest-integrity,
writer-taxonomy, and schema/migration regressions
- desired-state `schema/schema.sql` support without requiring a SQLx
migration ledger

## Deferred / not covered

- dedicated Helm/chart worker Deployment, service account, secrets,
probes, resources, and network policy
- autonomous `buzz-admin deletions worker` poll loop and worker-only
health server
- least-privilege separation among migration, relay-serving, and
destructive execution roles
- fleet-wide shared-CAS physical GC
- backup/provider/CDN/observability retention completion
- member-only erasure
- provider-native conditional-delete improvements
- a general force-continue escape hatch; permanent safety failures
remain fail closed unless an operator remediates the cause and records
an audited `unblock`

The removed continuous-worker implementation remains deferred; no remote
follow-up branch is claimed by this PR.

## Validation

### Current PR head and repository state

Current pushed head: `359d8402ee15f049768f54156f67b953c7a7e2ed`, rebased
onto `cc9a2f783375e51a6e8d1f2f9d01d5f7e22813d1` (`origin/main` at push
time). The complete PR diff is now 47 files, 9,834 additions, and 517
deletions.

The bespoke source-scanner stack was removed to keep this PR scoped to
community deletion. Tyler/team requested the underlying fenced-write
safety behavior, not `ast-grep`,
`crates/buzz-db/tests/community_fenced_writes.rs`, its 27 fixtures, or
the new `scripts/lints/community_*.yml` rules. Those scanner-specific
files, dependencies, Hermit links, and runner wiring are absent from the
current tree. The production database write fence, startup/destructive
live-catalog validation, and deletion behavior remain.

Source validation on this exact SHA passed:

- `cargo fmt --all -- --check`
- `bash -n scripts/run-tests.sh`
- `cargo nextest run -p buzz-db --all-targets`: 102 passed, 173 skipped,
0 failed
- `cargo nextest run -p buzz-deletion --all-targets`: 10 passed, 9
skipped, 0 failed
- `cargo nextest run -p buzz-admin --all-targets`: 1 passed, 0 failed
- affected-package/all-target Clippy with warnings denied
- lockfile consistency
- Helm 3.16.4 lint and all 44 chart unit tests
- Helm region controls using that fixture: default
`BUZZ_S3_REGION=us-east-1`, explicit `eu-west-2` override, and
blank-region schema rejection

The prior Kubernetes battery below was run against
`928992237358a3294621ac0280830b77155abc04`. It remains useful evidence
for the patch-equivalent production deletion implementation, but it is
**not** claimed as exact-SHA evidence for current head
`359d8402ee15f049768f54156f67b953c7a7e2ed`; the current cleanup removes
only scanner/test/tooling infrastructure. CI restarted for the new head
after the rebase and is pending. Human review remains
`CHANGES_REQUESTED`.

### Prior-head live Kubernetes deletion and safety gates

The full program used one immutable image, real PostgreSQL, Redis,
MinIO, and a three-relay Kubernetes release:

- source: `928992237358a3294621ac0280830b77155abc04` (**prior head**)
- image: `buzz-e2e:sha-928992237358`
- immutable image digest:
`sha256:a1a204f4618ac22d9e210be5e5290645a15d79831ae30b0e44379357c8e4a895`
- evidence root:
`/tmp/buzz-e2e/20260807T033025Z-928992237358-full-gates/`
- evidence-manifest digest:
`82875c5bc9bea7370b796a7aef3457b3a1c8306c84c59e0f7388bbb5ad30e865`

Passed gates at that prior head:

- **Chart/operator region:** default `us-east-1`, explicit nondefault
propagation, blank-region schema rejection, live in-pod environment, and
an in-pod taxonomy sweep over 18 objects with zero unknown.
- **Fenced writers and lifecycle:** open-write/fence ordering;
100-attempt anti-starvation; invite, push matcher, and exhausted-reaper
bystander isolation; non-`READ-COMMITTED` rejection; manifest/tombstone
contracts; eight-failure stage block and audited `unblock`.
- **Destructive lifecycle:** submit → approve → run →
`retention_pending`; PostgreSQL tombstone and Redis/S3 verification
true; zero retries/errors; terminal reruns rejected with exit 5.
- **Fresh 10,001-object crash boundary:** exactly two chunks (10,000 +
1). The executor deleted chunk 0 from MinIO while its PostgreSQL stamp
was row-lock-blocked, was killed with `SIGKILL`, left one object and
both stamps absent, then resumed the same request under generation 2 to
zero objects and terminal state.
- **Independent dead-owner recovery:** a dedicated executor claimed
generation 1, blocked before effects, and was killed through containerd
with `SIGKILL` (no TERM cleanup). The request remained owned and
unreclaimable before lease expiry; a successor claimed generation 2
after 60 seconds and completed with two attempts and zero retries.
- **Three-pod socket isolation:** ordinary NIP-42 and joined
huddle-audio target witnesses on every replica received exact `1008 /
community deleted`; healthy-tenant witnesses on those pods remained
live; deleted-host reconnect returned HTTP 404.
- **Health/provenance:** all replicas independently returned ready and
retained the exact image digest before/after destructive runs and an
audio-enabled rolling restart; PostgreSQL, Redis, and MinIO were healthy
at close.

Instrument corrections were retained as evidence rather than counted as
product failures: a foreground PostgreSQL forward caused an initial
`PoolTimedOut`; Kubernetes pod deletion exercised graceful TERM rather
than dead-owner recovery; shell-background socket witnesses died with
their parent; and the first image build hit the corporate TLS proxy.
Detached forwarding/witnesses, containerd `SIGKILL`, and the configured
internal CA/Artifactory mirror produced the discriminating runs without
weakening product security.

### Prior-head cleanup

For the prior-head Kubernetes run, the Helm release was removed,
namespace absence was verified, run-owned Screen sessions were absent,
and that source worktree remained clean. The evidence manifest was
independently recomputed and every indexed artifact passed `shasum -a
256 -c`. The current `359d8402` source worktree is also clean after the
scanner-only cleanup and push.

---------

Signed-off-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Signed-off-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Signed-off-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
Co-authored-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Co-authored-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Co-authored-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
 2026-08-12 09:25:58 -07:00
Dave Grochowski
397796c5f3 feat(tracing): add PostgreSQL tracing spans (#3678) 
## Why
Expose PostgreSQL datastore latency within existing request traces so
slow logical database operations can be identified without recording
tenant data or query arguments.

## What
- Add client spans around logical PostgreSQL operations across the
database facade, search, audit, replica fencing, and command persistence
- Use a dedicated `buzz_datastore` target and `db.system.name =
"postgresql"` for filtering and backend classification
- Exclude health-check database calls and scrub raw identifiers and
errors from newly traced paths

## Risk Assessment
Medium — this instruments frequently used datastore paths and increases
trace volume when enabled, but does not change SQL execution or
datastore behavior. Existing OpenTelemetry filtering controls export.

## References
- Pre-push clippy and fast unit-test hooks passed

Generated with Amp

---------

Signed-off-by: David Grochowski <dgrochowski@squareup.com>
Co-authored-by: Amp <amp@ampcode.com>
 2026-08-12 08:04:54 +10:00
Wes
2777189d96 fix(channels): restore member invitations to private channels (#5493) 
## Summary

- restore private-channel invitations for every active member
- keep owner/admin-only enforcement for elevated role grants, active
role changes, and removals
- preserve #4612's unrelated Desktop/mobile failure handling and
hardening
- add relay coverage for the ordinary actor/target role matrix
(`member`, `guest`, `bot`)

## Validation

- pre-push hook passed on `7de700e17642ad7e10155f9537033168d9249268`:
branch skew, Desktop checks/typecheck/tests/Tauri checks, mobile tests,
and Rust tests
- `cargo test -p buzz-test-client --test e2e_relay --no-run`
- `cargo fmt --all -- --check`
- `git diff --check`
- Donut and Mongo independently reviewed the cross-layer authorization
behavior; Donut's role-matrix coverage finding is addressed in this
revision

---------

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz>
 2026-08-10 10:59:22 -07:00
Taylor Ho
2ea9385015 fix(reactions): support max-length custom emoji (#3833) 
**Category:** fix
**User Impact:** Custom emoji with valid 64-character names can now be
used as reactions without errors.

**Problem:** Buzz accepted 64-character custom emoji names during
registration, but rejected them as reactions after the required
surrounding colons made the payload 66 characters. Validation also
differed between desktop, SDK, relay, and storage boundaries.

<img width="554" height="47" alt="image"
src="https://github.com/user-attachments/assets/4013452f-210e-4dd3-9003-f45ff3b28dc8"
/>

**Solution:** Keep the product limit at 64 ASCII characters for custom
emoji names, enforce it consistently when emoji sets are registered, and
allow only valid matching custom reaction payloads up to 66 characters.
Widen the reaction projection to preserve the wrapped payload while
retaining the existing 64-character limit for ordinary reactions.

<details>
<summary>File changes</summary>

**crates/buzz-sdk/src/builders.rs**
Defines the shared custom emoji boundaries and covers accepted
64-character and rejected 65-character shortcodes.

**crates/buzz-relay/src/handlers/ingest.rs**
Validates emoji-set shortcodes and permits 66-character reactions only
when they are valid colon-wrapped custom emoji with a matching tag.

**crates/buzz-db/src/event.rs**
Adds storage regression coverage for maximum-length custom emoji
reactions.

**crates/buzz-db/src/migration.rs**
Verifies the reaction column migration is applied correctly.

**desktop/src/shared/api/customEmoji.ts**
Enforces the existing 64-character shortcode maximum during desktop
normalization and registration/import.

**desktop/src/shared/api/customEmoji.test.mjs**
Covers the desktop shortcode boundary.

**migrations/0027_long_reaction_payloads.sql**
Widens stored reaction payloads to 66 characters for the two required
surrounding colons.

**schema/schema.sql**
Keeps the desired schema aligned with the migration.

</details>

## Reproduction Steps

1. Register or import a custom emoji whose ASCII shortcode is exactly 64
characters.
2. Select that emoji as a reaction to a message.
3. Confirm the reaction publishes, persists, and renders without an
error.
4. Attempt to register a 65-character shortcode and confirm it is
rejected.
5. Publish an ordinary or malformed reaction over 64 characters and
confirm the relay rejects it.

## Verification

- `cargo test -p buzz-sdk`: 243 passed
- `cargo test -p buzz-db`: 94 passed, 152 Postgres-required tests
ignored
- `pnpm test` in `desktop`: 3,859 passed
- `cargo test -p buzz-relay`: 795 passed, 9 existing
Postgres-unavailable failures, 35 ignored; new reaction boundary tests
pass directly
- `cargo fmt --all -- --check`
- `git diff --check`

Originating Buzz channel: `f2ec9671-d78e-4cde-894c-9f4c458c7f1f`

---------

Signed-off-by: Taylor Ho <taylorkmho@gmail.com>
Co-authored-by: npub1223z34hd7vtwc6qj4s7flsxkj644nlre2nthu7lrrmkumhu3xddsrx9r6w <52a228d6edf316ec6812ac3c9fc0d696ab59fc7954d77e7be31eedcddf91335b@buzz.block.builderlab.xyz>
 2026-08-05 21:02:57 +00:00
Jordan Mecom
efe1893dd3 fix(channels): restrict private-channel invitations (#4612) 
This change requires an active owner or administrator for third-party
additions to private channels. The relay validator and transactional
database authority enforce the same rule, including removed-member
reactivation and role-change paths.

Idempotent self-target behavior remains available, while ordinary
members can no longer extend private-channel access to another identity.

## Testing

- `git diff --check
origin/main...codex/security-private-channel-invite-authority`
- Rebased onto `origin/main` at `5c98932`
- Full CI pending

Originating Buzz thread:
`buzz://message?channel=3928fe05-df61-4b5d-b9c7-d623b9b10ea1&id=3c6c02312f763fbe0d2bfc33a6c1a362f91d0354f3d18b039cf7a0558c1439d1`

---------

Signed-off-by: Jordan Mecom <jm@squareup.com>
Signed-off-by: Eli Foster <efoster@squareup.com>
Co-authored-by: Eli Foster <efoster@squareup.com>
 2026-08-05 10:47:00 -07:00
Jemiah Westerman
bc9e6528a7 perf(relay): index channel-id lookups and skip trace-only reads (#4647) 
## Problem

`SELECT id, community_id FROM channels WHERE id = ANY($1) AND deleted_at
IS NULL` is the top **Load by waits (AAS)** on the Buzz Postgres writer.
Two independent causes compound, and both are fixed here.

### 1. No index can serve it

`channels` is `PRIMARY KEY (community_id, id)`, and every secondary
index leads with `community_id`:

| Index | Columns |
|---|---|
| *(primary key)* | `(community_id, id)` |
| `idx_channels_nip29_group` | `(community_id, nip29_group_id)` |
| `idx_channels_dm_hash` | `(community_id, participant_hash)` |
| `idx_channels_community_type` | `(community_id, channel_type)` |
| `idx_channels_community_visibility` | `(community_id, visibility)` |
| `idx_channels_created_by` | `(community_id, created_by)` |
| `idx_channels_ttl_expiry` | `(ttl_deadline)` *(partial)* |

The two tenant-independent lookups carry **no `community_id` predicate**
— deliberately:

- `Db::communities_of_channels` — `WHERE id = ANY($1) AND deleted_at IS
NULL`
- `Db::community_of_channel` — `WHERE id = $1 AND deleted_at IS NULL`

That independence is load-bearing, not an oversight: projecting a row's
*true* owning community regardless of the fetch query's `WHERE` clause
is what makes `Inv_NonInterference` non-vacuous. If the fetch ever
dropped its tenant scoping, this lookup would still report the real
label and the checker would catch the mismatch.

But a composite btree is only usable when its leading column is
constrained, so neither query can use the primary key, and nothing else
leads with `id`. **Both sequentially scan `channels` on every call.**

### 2. In production the result is discarded

Both call sites feed `record_read_message_rows` /
`record_read_by_id_rows`, which call `tracer.record(...)`. Production
binds `NoopTracer` (`crates/buzz-relay/src/state.rs`), whose `record`
body is empty.

The existing guard tests `trace_state`, which is `Some` for every
well-formed request — it only goes `None` on malformed pubkey bytes. So
the scan ran on the hot read path and its output was dropped. This is
the classic eager-argument bug: `log.debug("..." + expensiveCall())`
with no `isDebugEnabled()` check.

### 3. Multiplied per filter

The non-search call site sits **inside the phase-3 per-filter loop**, so
a `REQ` carrying N filters performed N sequential scans of `channels`
before responding.

## Changes

**`Tracer::enabled()`** — a capability check on the trait (the
`isDebugEnabled()` of this seam), defaulting to `true`. `NoopTracer`
overrides it to `false`, and both emitters in `req.rs` now gate on it,
skipping the trace-only DB read entirely in production.

**`migrations/0027_channels_id_lookup_index.sql`**

```sql
CREATE INDEX IF NOT EXISTS idx_channels_id_live
    ON channels (id) INCLUDE (community_id)
    WHERE deleted_at IS NULL;
```

- `INCLUDE (community_id)` — both queries select exactly `(id,
community_id)`, so this is covering and can be served index-only.
- Partial on `deleted_at IS NULL` — matches both predicates exactly,
excludes soft-deleted history, and lets Postgres skip the recheck.
- **Not `UNIQUE`.** `id` alone is *not* unique in this table —
`command_executor.rs` documents that `community_of_channel(channel_id)`
is ambiguous because the same channel id can appear under more than one
community. A unique index would encode a false constraint and fail to
build on any database already holding such a pair.

Worth keeping the index even though fix #1 removes the production
caller: it still runs under conformance, and `community_of_channel` has
the same problem on its own paths.

**`schema/schema.sql`** — mirrored, since a test asserts desired-state
parity.

## Conformance is unchanged

This is the part worth reviewing closely. Under a real tracer
`enabled()` returns `true` and **every emit happens exactly as before**
— the gate only skips *building* emit inputs when nothing observes them,
never an emit that would otherwise have been made. The coverage-breach
guard stays non-vacuous.

`CountingTracer` forwards `enabled()` to its inner tracer rather than
inheriting the `true` default. Both directions matter and both fail
silently:

- inheriting `true` over a `NoopTracer` would keep the overhead this PR
removes;
- hardcoding `false` over a live tracer would suppress the emits whose
absence `EmitGuard` reports as `ImplBug` — masking real breaches behind
expected ones.

Covered by a new regression test,
`counting_tracer_delegates_enabled_to_inner`, which asserts delegation
in both directions.

## Verification

- `cargo check -p buzz-conformance -p buzz-relay` — clean
- `cargo clippy --all-targets` — clean, zero warnings
- `cargo test -p buzz-conformance` — 6/6
- `cargo test -p buzz-relay --lib conformance` — 11/11
- `cargo test -p buzz-db --lib migration` — 7/7
- `just test-unit` (pre-push) — green

Migration-count assertions in `crates/buzz-db/src/migration.rs` were
bumped 26 → 27, with content assertions for 0027 following the existing
per-migration pattern (including a guard that it never becomes
`UNIQUE`).

## Open questions for reviewers

1. **Lock strategy.** Built *without* `CONCURRENTLY`, following
migration 0004's precedent, because sqlx runs each migration inside a
transaction and `CREATE INDEX CONCURRENTLY` cannot run in one. This
takes a brief `SHARE` lock on `channels` (blocks writes, not reads) —
small relative to `events`, but an operator preferring zero
write-blocking can pre-build it by hand and `IF NOT EXISTS` makes the
migration a no-op. I could not confirm whether sqlx 0.9 supports a `--
no-transaction` directive; if it does, that may be preferable.

2. **Diagnosis is static.** This comes from reading the source, not from
`EXPLAIN` against the live database. Worth confirming with `EXPLAIN
(ANALYZE, BUFFERS)` on the writer before/after — that also sizes the win
by revealing the real table size and row counts.

3. **Expected impact** scales with average filters-per-`REQ`, which I
did not measure. `pg_stat_statements` ordered by `total_exec_time` would
confirm this query drops off the top and show whether anything else is
scanning the same way.

Signed-off-by: Jemiah Westerman <jemiah@squareup.com>
 2026-08-04 13:59:54 -04:00
Tyler
ac4fa13b8e perf(relay): serve relay-membership checks from the read replica (#4124) 
## Summary

Route `Db::is_relay_member` — the membership check that runs on every
authenticated HTTP request and WS AUTH — through the standard
`route_read` machinery on the bounded arm, instead of adding a bespoke
cache (replaces #3844).

- `crates/buzz-db/src/relay_members.rs`: add `is_relay_member_on(&mut
PgConnection, ...)` executor seam; the pool version delegates to it.
- `crates/buzz-db/src/lib.rs`: `Db::is_relay_member` now routes via
`route_read("relay_membership", RoutePredicate::Bounded)` — replica only
on a proved fresh session, writer on any route rejection, writer re-run
on replica query error. Exactly the shape of every other routed read.

This is the one permission read served from the replica, by explicit
product decision (Tyler accepted ≤1s bounded staleness on reads we
choose): the fleet-wide fence guarantee (`BUZZ_REPLICA_READ_MAX_AGE_MS`,
deploy target 1s) is an order of magnitude tighter than the 10s TTL
proposed in #3844 and needs no invalidation machinery. Staleness is
symmetric for admits and revokes. `BUZZ_REPLICA_READ_MAX_AGE_MS` unset =
writer-only = kill switch. It is not precedent for routing other
permission reads.

## Validation

At this exact commit (`git rev-parse HEAD` confirmed in the same shell,
rustc 1.95):

- `cargo test -p buzz-db` — 94 passed, 0 failed
- PG-gated suite single-threaded — **151 passed, 2 failed**; the 2
failures are the per-owner-limit tests broken on main by #3829 (limit
3→5, tests still seed 3) — they fail identically at base `19d57b0d4` in
a pristine control checkout; separate trivial fix to follow
- New PG-gated test `is_relay_member_is_bounded_routed_and_fails_closed`
— divergent writer/replica fixtures prove: budget unset ⇒ writer; budget
set + fresh proof ⇒ replica; over-budget entry ⇒ writer
- clippy `-D warnings` + fmt clean; pre-push hooks green (desktop
check/test, rust tests, tauri checks)
- **Live-local pass** (TESTING.md, release binary,
`BUZZ_REQUIRE_RELAY_MEMBERSHIP=true`, fresh DB):
- writer-only (no `READ_DATABASE_URL`): member accepted, outsider 403
`relay_membership_required`; metrics
`route_decision{path="relay_membership",decision="writer",reason="disabled"}`
- replica configured + `BUZZ_REPLICA_READ_MAX_AGE_MS=1000`: member
accepted / outsider denied via `decision="replica",reason="fresh"`;
admit visible to the routed check within ~1.2s; revoke enforced within
~1.2s
- reader outage mid-flight (TCP proxy killed): member send still
succeeds in <200ms via
`decision="writer",reason="reader_acquire_timeout"`; outsider still
denied — fails closed, no availability loss

Reviewed by Wren: 9/10 minimalness, 9/10 elegance, 9.5/10 correctness at
this SHA.

Supersedes the 10s-cache approach in PR 3844, which should be closed
unmerged once this lands.

Signed-off-by: Tyler <109685178+tlongwell-block@users.noreply.github.com>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
 2026-08-01 13:32:16 -04:00
Tyler
5765fc74b7 fix(relay): allow open relays to set their NIP-11 workspace icon (kind:9033) (#3998) 
## Problem

The desktop deliberately shows the workspace icon editor on open relays
(#2640, gate: `canEditIcon` in
`desktop/src/features/communities/ui/EditCommunityDialog.tsx`) and
defers to the relay-side kind:9033 check — which required an admin/owner
row in `relay_members`. For a community with **no admin/owner row at
all** (the `ensure_configured_community` path, which never writes an
owner), every 9033 was refused and the icon was permanently unsettable.

**Correction from review (thanks @Dawn):** the original version of this
PR claimed nobody holds a role on an open relay. That's false —
`main.rs` bootstraps `RELAY_OWNER_PUBKEY` as owner regardless of
`BUZZ_REQUIRE_RELAY_MEMBERSHIP`, so a production open relay like
bb-block *does* have an owner row, and the old gate was refusing
everyone except that owner. The first revision of this diff would have
silently widened that owner-only control to any NIP-42-authenticated
sender.

## Fix — steward-wins

`may_set_workspace_profile(sender_role, membership_enforced,
community_has_steward)`:

| Relay mode | Community has admin/owner row? | Who may set the icon |
|---|---|---|
| Closed (`require_relay_membership=true`) | any | admin or owner
(unchanged) |
| Open | yes (e.g. bb-block) | admin or owner (unchanged posture) |
| Open | no (genuinely rosterless) | any NIP-42-authenticated sender |

- New DB helper `has_admin_or_owner(community)`
(`crates/buzz-db/src/relay_members.rs`); the call site only queries it
on open relays.
- The rosterless admit logs a `warn!` with the sender pubkey — 9033
writes no audit row and publishes no announcement event (unlike
9030/9031), so this is the only durable attribution.
- Kinds 9030–9032, NIP-42 auth, `AdminUsers` scope, ban gate, and icon
validation are all untouched.
- Doc comment fixed: cited nonexistent `canEditCommunityProfile`; real
symbol is `canEditIcon`.

## Test coverage — closing the mutation gap

Dawn's mutation testing showed the original unit tests pinned only the
helper's truth table: inverting the flag at the call site or deleting
the gate entirely survived the full suite.

- Unit tests now cover the 3-arg truth table (closed
steward-independent, open-with-steward stays steward-only,
rosterless-open admits).
- Two `#[ignore]`d Postgres integration tests drive
`handle_relay_admin_event` with a real `AppState` (open rosterless admit
→ steward appears → roleless refused again; closed relay member
refused). Wired into the Backend Integration CI job as a dedicated
nextest step.
- **Both of Dawn's mutants verified killed** at this head: flag
inversion fails 1 unit test; gate deletion fails both integration tests
(`Ok(())` where `Rejected` expected).

## CI wrinkle found and fixed: pre-existing schema drift

The first Backend Integration run of the new 9033 tests failed with
`column "icon" of relation "communities" does not exist` — migration
`0003_community_icon.sql` added the column, but `schema/schema.sql` (the
desired-state file that CI job applies via pgschema) was never updated.
Pre-existing drift, invisible until a test in that job actually wrote
the column. Fixed in `297148f62` (3-line addition to
`schema/schema.sql`).

## Receipts (at `1b4b52db8` code / `297148f62` head)

- `cargo test -p buzz-relay`: 835 pass, 1 fail —
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`,
pre-existing (fails identically at the old base and on clean main);
`telemetry::trace_context_lookup_does_not_enable_callsites` is a known
order-dependent flake, passes in isolation.
- `cargo test -p buzz-db`: 94 pass.
- Both ignored integration tests pass live against local Postgres.
- `cargo fmt --all -- --check`: clean.
- Live-local pass per TESTING.md at this head (release build, relay on
:3199, real WS + NIP-42 via nak):
- open rosterless: roleless key sets icon → NIP-11 serves it; `warn!`
with sender pubkey in the relay log
- open + owner row inserted: fresh roleless key refused ("must be admin
or owner"); owner sets icon
- closed relay (owner bootstrapped, `BUZZ_RELAY_PRIVATE_KEY` set): plain
member refused, owner sets icon, `javascript:` URL rejected, empty icon
clears (NIP-11 → null)

---------

Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
 2026-08-01 12:03:31 -04:00
Will Pfleger
cb9701cd30 feat(relay): accept kind:30621 multi-repo projects at ingest (#3171) 
Buzz renders one card per `kind:30617`, so a project spanning several
repositories has no representation.
[NIP-MP](https://github.com/block/buzz/pull/3163) defines `kind:30621`
as an addressable container holding a group's name, description, channel
binding, and member coordinates. This adds the kind to `buzz-core` and
its structural validation to the relay ingest path.

## Event shape

```json
{
  "kind": 30621,
  "tags": [
    ["d", "platform"],
    ["name", "Platform"],
    ["description", "Relay, desktop, and mobile."],
    ["a", "30617:<owner-a-hex>:buzz"],
    ["a", "30617:<owner-b-hex>:buzz-infra"],
    ["buzz-channel", "<channel-uuid>"],
    ["buzz-visibility", "listed"]
  ]
}
```

## Validation at ingest

| Rule | Behavior |
|------|----------|
| `d` tag | exactly one, non-empty (length already bounded by the
generic `D_TAG_MAX_LEN` check) |
| member `a` tag arity | exactly 2 or 3 elements per NIP-01's `a` tag
grammar; a 4th element has no defined meaning and is rejected |
| member `a` tag coordinate | must parse as
`30617:<lowercase-64-hex-owner>:<non-empty-d>` |
| duplicate members | rejected on exact string match of the canonical
coordinate |
| member cap | 64, counted over raw `a` tags |
| metadata cardinality | at most one each of `name`, `description`,
`buzz-channel`, `buzz-visibility` |
| metadata length | `name` ≤ 256 bytes, `description` ≤ 2048 bytes,
`buzz-channel` ≤ 256 bytes, `buzz-visibility` ≤ 256 bytes |
| zero members | valid |
| unknown tags | ignored |

Rejection order is normative so a client can predict which rule fires:
`d`-cardinality → `d`-empty → member-cap → member-arity → coordinate
parse → member-duplicate → metadata cardinality → metadata length.

## Design notes

**No membership authorization.** Members are `a` tags, so one project
may name repositories owned by different pubkeys — the entire point of
the kind. That is safe because membership grants nothing: push policy
reads a repository's own `kind:30617` (`api/git/policy.rs`) and never a
project. `buzz-channel` is a metadata reference, not a routing
directive, so projects are classified global-only.

**Owner-only editing is free.** NIP-33 addressing keys replacement on
`(pubkey, kind, d)`, so one signer can never overwrite another's
project. No relay-side permission check exists or is needed, and
`test_project_same_d_under_two_authors_are_independent` pins it.

**Duplicates are rejected, not deduped.** A relay cannot rewrite tags
inside a signed event without invalidating its id and signature, so the
alternative to rejection is a stored duplicate-member head that every
consumer must apply a first-wins rule to.

**The cap is checked before the duplicate set is built.** Counting raw
`a` tags rather than distinct coordinates means an event naming one
coordinate thousands of times is refused on count, instead of being
bounded only by the relay frame limit.

**No side-effect handler.** Generic NIP-33 replacement and generic
NIP-09 coordinate soft-delete already cover replacement and deletion;
`kind:30621` needs no entry in `is_side_effect_kind`.

## Generic NIP-09 fix carried along

`soft_delete_by_coordinate` (`crates/buzz-db/src/event.rs`) previously
deleted the live coordinate head regardless of the tombstone's own
`created_at`, so a delayed or replayed `a`-tag deletion signed between
two versions destroyed the newer replacement. NIP-09 scopes an `a`-tag
deletion to versions at or before the deletion request, so the `UPDATE`
now carries `created_at <= $5` and `handle_a_tag_deletion` threads the
deletion event's `created_at` through.

The bug predates `kind:30621` and affected every
parameterized-replaceable kind on the generic path — `kind:30617`
repository announcements included — so the fix lands there rather than
as a project special case. `events.created_at` is immutable per row, so
the predicate guarantees a tombstone can never erase a version newer
than itself; the UPDATE re-evaluates its WHERE clause after any lock
wait. Under READ COMMITTED, a same-coordinate replacement racing the
deletion may cause the deletion to evaluate before the new head lands,
returning `Ok(false)` — but that outcome is state-identical to the
deletion having arrived first, a valid Nostr ordering Nostr never fixes.
The return value feeds only a debug log. No coordinate-level lock is
needed.

## Coverage

32 unit tests in `crates/buzz-relay/src/handlers/ingest.rs` pin the
envelope contract (accept: minimal, cross-owner, zero-member, same repo
`d` under two owners, colon-bearing repo `d`, cap boundary, unknown
tags, relay hint on member `a` tag, max-length metadata, stranger-owned
member, uninterpreted metadata values, non-empty content; reject: every
rule above plus valueless `d`/`a` tags). A fixture-driven test
(`project_envelope_validates_all_shared_fixtures`) runs every case in
the shared `NIP-MP.fixtures.json` oracle (11 accept + 20 reject) against
`validate_project_envelope`, so any future change that breaks a case
turns the test suite red.

6 `#[ignore]`d e2e tests in
`crates/buzz-test-client/tests/e2e_project.rs` cover behavior that only
exists past storage — coordinate round-trip, newer-wins replacement, two
authors sharing a `d`, an `a`-tag tombstone that removes the project
while leaving referenced `kind:30617`s intact, and a tombstone
timestamped between V1 and V2 that must leave V2 live. The negative e2e
case asserts on the rejection message so a refusal for an unrelated
reason cannot satisfy it; that is what proves the validator is reachable
from the live write path rather than merely correct in isolation. The
new e2e binary is wired into the Relay E2E job.

The timestamp predicate is additionally pinned at the storage layer by
`coordinate_delete_spares_head_newer_than_the_deletion` in
`crates/buzz-db/src/lib.rs`, which asserts both directions: a stale
tombstone deletes nothing and leaves the newer head readable, and a
tombstone at the head's own timestamp still deletes it. This test is
wired into the Backend Integration job.

Related: #3163 (the NIP-MP spec and shared conformance fixtures).
Independent — either can merge first.

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
 2026-07-31 16:22:57 -04:00
Kalvin C
10d5a26414 feat(relay): raise hosted community limit to five (#3829) 
## Summary
- raise the relay authoritative default community ownership limit from 3
to 5
- raise the desktop hosted-community treatment from 3 to 5
- preserve `BUZZ_MAX_COMMUNITIES_PER_OWNER` as a deployment override

## Validation
- `pnpm -r check`
- `cargo fmt --all -- --check`
- `cargo test -p buzz-db` (94 passed, 151 Postgres-dependent tests
ignored)
- pre-push hooks: desktop checks/tests, Rust tests, Tauri checks (all
passed; 1,995 desktop Rust tests passed)

Signed-off-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Co-authored-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
 2026-07-30 18:20:58 -07:00
Will Pfleger
23f0c26b1c fix(relay): align NIP-11 max_limit with REQ ceiling (#3635) 
Buzz's NIP-11 document advertised `limitation.max_limit: 10_000`, but
the effective websocket REQ page ceiling was `1_000` — a 10x lie.

The websocket REQ path never sets `EventQuery::max_limit`, so
`query_events` applied its own `unwrap_or(1000)` clamp to every
historical query. Only the COUNT fallback (`apply_count_fallback_limit`)
ever raises that clamp. A client that trusts the advertised value asks
for 10,000 events, silently receives 1,000, and — with no error and no
continuation signal — reads that short page as exhaustion. Up to 9,000
events are dropped without anyone noticing.

`MAX_HISTORICAL_LIMIT = 2_000` in `handlers/req.rs` was dead weight for
the same reason: nothing clamped to 2,000 could survive the DB's 1,000
clamp one layer down.

## Change

`buzz_db::DEFAULT_MAX_PAGE_LIMIT` (`1_000`) is now the single source of
truth. It is the `query_events` clamp default, the value both REQ clamp
sites use, and the value advertised as NIP-11 `max_limit`.
`MAX_HISTORICAL_LIMIT` is removed rather than re-pointed — an alias for
a constant used four lines away adds a name without adding meaning.

The NIP-50 search path carries a second, independent bound. It clamps
its emission target to the shared ceiling like any other REQ, but how
many FTS candidates it will scan was bounded separately, by a bare
10-page loop over 100-hit pages. That product only coincidentally
equalled the ceiling, so raising the ceiling — or shrinking a page —
would shrink the scan relative to what clients may now request,
degrading search quality while nothing in the code registered the
change. The page count is now ceiling-divided from
`DEFAULT_MAX_PAGE_LIMIT` over a named `SEARCH_PAGE_SIZE`, so the scan
budget tracks the advertised ceiling by construction.

That budget is a resource policy, not a delivery promise. It bounds
candidates *scanned*, not events *emitted*: post-filtering (NIP-01
match, channel access, reader visibility, dedup) discards an
unpredictable share of every page, so a search result smaller than the
requested limit remains possible. This is not a NIP-11 violation —
`max_limit` is defined as a clamp the relay applies to a requested
`limit`, not a guaranteed count in the response.

Two guards hold the pair together:

- `req_filter_limit_clamps_to_advertised_nip11_max_limit` reads
`max_limit` back out of a built `RelayInfo` and asserts the REQ path
clamps to exactly that number.
- `search_scan_capacity_covers_advertised_nip11_max_limit` asserts the
scan budget covers exactly one advertised ceiling's worth of candidates
— no less, and with no spare page of slack, so the derivation can't be
quietly replaced by a hand-tuned constant that happens to pass today.

## Behavior

Websocket behavior is unchanged: 1,000 was already the real ceiling on
every path, including NIP-50. The advertisement now tells the truth
about it. Raising the effective limit is a capacity decision and is
deliberately not made here.

The generic HTTP bridge's page-2+ offsets do change, as a consequence of
the corrected clamp. `extract_page_offset` sizes a page from
`query.limit` *before* the DB clamp applies, so an absent limit
previously produced an offset of 2,000 and a requested 1,500 produced
1,500 — while the page actually returned held at most 1,000 rows. Both
now produce 1,000. This corrects paging that had been skipping rows the
previous page never returned;
`extract_page_offset_sizes_pages_from_clamped_limit` locks it down.

## Scope note

The bridge's per-endpoint ceilings — `BRIDGE_WINDOW_MAX_LIMIT` (200) for
channel windows and `BRIDGE_THREAD_MAX_LIMIT` (500) for thread reads —
are endpoint contracts on a non-NIP-01 transport, not values NIP-11
speaks for, and are unchanged.


Fixes #3757

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
 2026-07-30 18:42:18 -04:00
Will Pfleger
114d40d9d3 feat(relay): gate kind 30178 team-catalog reads behind the shared tag (#3358) 
Team catalog projections (`kind:30178`) embed every member's system
prompt, so they need the same read gate personas already have: only the
author sees an unshared event. The gate was hardcoded to `kind:30175` at
six read surfaces plus the SQL pushdown, so rather than adding a second
special case it becomes kind-generic over `SHARED_GATED_KINDS = {30175,
30178}`.

## Kind 30178

New parameterized-replaceable kind, addressed by `(pubkey_o, 30178,
team_id)`. It embeds sanitized member projections instead of referencing
`kind:30175` heads — a foreign reader of a shared team could not
otherwise hydrate members whose own persona events are unshared or, for
built-ins, absent entirely. `kind:30176`'s wire body is untouched, so
device sync keeps its contract.

## Kind-generic shared gate

`buzz_core::kind` replaces `is_persona_shared_kind` /
`is_unshared_persona_event` / `persona_event_is_shared` with
`SHARED_GATED_KINDS` and the kind-agnostic `is_shared_gated_kind` /
`is_unshared_gated_event` / `event_is_shared`. Every read surface
consults the set:

| Surface | File |
|---|---|
| REQ historical delivery + `ids` lookup |
`crates/buzz-relay/src/handlers/req.rs` |
| Live fan-out | `crates/buzz-relay/src/handlers/event.rs` |
| COUNT fallback | `crates/buzz-relay/src/handlers/count.rs` |
| NIP-98 HTTP `/query`, `/count`, `/search` |
`crates/buzz-relay/src/api/bridge.rs` |
| Pre-`LIMIT` SQL pushdown | `crates/buzz-db/src/event.rs` |

The SQL clause generalizes from `kind != 30175` to `kind NOT IN (...)`
bound from `SHARED_GATED_KINDS`, still applied before `ORDER BY … LIMIT`
so a page of newer private events cannot starve an older shared one off
the candidate set. `EventQuery::persona_reader` is renamed
`shared_gated_reader` and `needs_persona_filtering` to
`needs_shared_gate_filtering` to match.

Because the `buzz-core` rename has consumers outside the relay, the four
desktop call sites of `persona_event_is_shared` travel with it:
`desktop/src-tauri/src/commands/personas/pending.rs`,
`desktop/src-tauri/src/event_sync.rs`, and two in
`desktop/src-tauri/src/managed_agents/persona_events.rs`. Each call is
unchanged apart from the name — the persona `shared` projection behaves
exactly as before.

## Ingest validation

`validate_persona_envelope` splits into two reusable pieces —
`validate_shared_tag` (exactly-two-element `["shared","true"]`, at most
one occurrence) and `single_bounded_d_tag` (exactly one `d` tag,
non-empty, `<=64` chars, no ASCII control characters or whitespace).
`validate_team_catalog_envelope` composes both; personas additionally
keep the slug grammar `^[a-z0-9][a-z0-9_-]{0,63}$`.

`kind:30178` deliberately does **not** get the slug grammar. Team ids
are UUIDs or built-in identifiers such as `builtin-team:welcome`, and
the colon is not slug-legal; rewriting ids to fit would break NIP-33
addressing against the team's own `kind:30176` head. The non-empty and
exactly-one checks are load-bearing regardless — without them generic
NIP-33 storage maps a missing `d` onto `(pubkey_o, 30178, "")` and every
team overwrites its predecessor.

The exact two-element `shared` shape is enforced because the SQL
visibility clause is JSONB containment (`tags @>
'[["shared","true"]]'`), which would match a three-element superset such
as `["shared","true","extra"]`.

`kind:30178` is also added to the `Scope::UsersWrite` allowlist and to
`is_global_only_kind`, so a stray `h` tag cannot channel-scope an
owner-authored definition.

## Deferred

`kind:30176` is deliberately not a gate member. Its writers never emit
`shared`, so catalog opt-in semantics do not describe it — it needs
owner-private reads driven by an authenticated principal set, tracked as
a separate follow-up.

## Tests

- 19 new `ingest.rs` unit tests covering the 30178 envelope (UUID and
colon `d` tags, 64-char boundary, non-ASCII bound,
empty/valueless/duplicate/missing `d`, embedded newline, `shared`
false/three-element/duplicate, scope and global-only membership).
- Persona regressions for the valueless `["d"]` shapes, since the
`d`-tag helper is shared by both validators.
- Existing `kind.rs` gate tests generalized and extended to assert the
gate applies to 30178 as it does to 30175.
- New `crates/buzz-test-client/tests/e2e_team_catalog.rs`: 9 WS-level
tests over a live relay covering author reads of unshared heads, foreign
omission from REQ, `ids`-lookup denial, COUNT existence-leak, share and
unshare transitions, and the mixed-kind filter case.
- `.github/workflows/ci.yml` adds `--test e2e_team_catalog` to the Relay
E2E job so the new suite runs.

## Docs

`docs/nips/NIP-AP.md` gains a "Team catalog projection: kind:30178"
section and an "Ingest validation: kind:30178" subsection, records the
gate as kind-generic, documents 30178 deletion vs. unshare semantics,
and adds a security note that sharing a team exposes every member's
instructions even when that member's own `kind:30175` head is unshared.

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
 2026-07-30 17:53:30 -04:00