mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
a7c7414a7f
Implements the Rust/backend half of 30178 team catalog sharing on the community catalog. No TS callers yet — this is PR 1 of 2; [#3995](https://github.com/block/buzz/pull/3995) (stacked here) adds the parse layer, hooks, `CommunityCatalogDialog`, and e2e tests. ## What this adds **Projection builder** (`team_catalog.rs`): `build_team_catalog_event/content` produces a 30178 event from a team + member definitions. Size contracts: 192 KiB total ceiling, per-field bounds (name 256 B, text 4 KiB, system-prompt 16 KiB, avatar URL 32 KiB for projection fields; https URLs additionally validated at 2,048 bytes). Avatar handling: oversized raster data URLs downscaled to fit; oversized built-in avatars silently omitted; oversized https URLs rejected with a named error. **Share/unshare/tombstone** (`commands/teams/pending.rs`): `prepare_team_publication_at` retains a signed 30178 head for the flush loop; share state is relay-scoped (one community can share while another does not). `refresh_or_retract_shared_head_at` rebuilds or tombstones the head immediately on team/member edit — no stale publication until the next boot. Both the 30178 catalog and 30176 team tombstones are signed with a `created_at` that strictly dominates the retained head's (read inside the delete transaction), so a future-dated head cannot survive its own deletion under the relay's `created_at <=` soft-delete gate. **Serialized-publisher share command** (`commands/teams/sharing.rs`, `managed_agents/persona_events.rs`): `set_team_shared` publishes the retained head through `flush_pending_events_at` rather than submitting the prepared event directly. A direct submit ran outside `managed_agents_store_lock` and raced `delete_team`: the delete atomically purges the head's retained row and enqueues a newer 30178 tombstone in one transaction, and a delayed direct submit could land the old shared head *after* that tombstone. Because 30178 replacement has no deletion watermark, the deleted team would go publicly live again. Routing through the flush is necessary but not sufficient, because the flush is not itself a single publisher: several call sites (the 30s sweep, this share toggle, managed-policy updates) invoke it concurrently, and each invocation has an await gap between its per-row re-read and its relay POST. A second flush could publish the tombstone in that gap while an earlier flush's delayed POST lands the just-purged head after it. The flush now acquires a per-scope publisher mutex — keyed by the canonical retention `db_path`, which already *is* the durable scope identity (hashed normalized relay URL + owner pubkey) — and holds it across its entire invocation: snapshot, per-row re-read, POST, and `mark_synced`. Serialized-per-scope flush ⟹ within one scope the only interleavings are head-before-tombstone (the head lands first, then is dominated by the later tombstone) or purged-row-skip (the delete committed first, so the re-read skips the head) — a purged head can never publish after its tombstone. The lock is a `LazyLock<Mutex<HashMap<PathBuf, Arc<tokio::sync::Mutex<()>>>>>` static rather than an `AppState` field: it keeps the invariant at its acquisition site and out of the size-ratcheted `app_state.rs` (precedent: `agent_models_databricks.rs`'s `AUTH_GATE`), and the std-mutex map guard is released before the async guard is awaited so it never spans an await point. Keying per scope — rather than one process-global lock — means a stalled or hostile relay in one community can no longer block publication in every other community, and each per-row submit is additionally wrapped in a `tokio::time::timeout(PUBLISH_TIMEOUT = 60s)`: `submit_signed_event_at_with_keys` first waits on the process-wide admission gate (up to 300s on a 429) and then POSTs on the app-wide `http_client`, which leaves reqwest's connect/read/total timeouts unset, so a relay that accepts the connection and never finishes the response would otherwise pin the lock forever. A timeout takes the same `Err`/`continue` path as a relay rejection — the row stays pending for the next 30s sweep and a timed-out tombstone keeps its replacement deferred this pass — so a live admission gate now surfaces as timeout-pending rather than a held lock, the correct durable behavior since the sweep retries. `publicationStatus` (Published/Queued) is derived by re-reading the retained row's pending flag after the flush. The previously-unused `relayMessage` field is dropped from `SetTeamSharedResult`: the flush loop swallows every per-event relay rejection to its own log and only surfaces local DB faults (which the status re-read already propagates), so the field was permanently `null` on this path. #3995's `tauriTeams.ts` mapping and `useTeamActions.ts` log branch drop with it. **Domination-aware flush** (`managed_agents/persona_events.rs`): a strictly-dominating tombstone can be signed past the relay's ingest acceptance window (`MAX_TIMESTAMP_DRIFT_SECS`, ±900s from server time). Republishing such a byte-frozen event verbatim from the pending queue lets it age out of the window and be rejected forever, stranding the head live. The flush loop is now domination-aware for every retained kind:5 (covering both 30176 and 30178). For a pending tombstone with floor `f` (= the retained row's own `created_at`): - `f <= now` → re-date and re-sign at `now` (mirrors the existing archive-request re-sign branch; `mark_synced` stays keyed to the untouched retained row, so a re-date can't mask a concurrent edit), - `now < f <= now + 900` → publish verbatim at `f`, inside the window, - `f > now + 900` → skip the sweep; the event stays pending and its replacement keeps deferring (via `failed_tombstones`) so out-of-order retraction remains impossible, converging as the wall clock advances. No path emits an event the gate rejects, and a boundary reject self-heals through the submit-error requeue. Durable across offline gaps of any length. **Atomic adopt** (`commands/teams/adopt/`): `add_team_from_catalog` re-fetches and signature-verifies the head from the relay, then plans and commits a multi-entity add across two store writes with byte-exact rollback on any failure (crash window between writes explicitly retained). `plan_add` resolves full member provenance (owner, d-tag, member-key, projection-hash), reuses a recipient's own local built-in only when the published slug matches and the reuse hint's `projection_hash` — recomputed from the member's own embedded fields at the parse boundary and rejected on mismatch — equals (case-insensitively, matching the boundary's hex tolerance) the recipient's local built-in hash, and reactivates deactivated copies on re-add. Because the boundary already proves the hint hash describes the reviewed projection, a publisher cannot pair a real built-in's slug + hash with arbitrary reviewed fields to make adoption install the recipient's built-in in place of what was shown. `commit_stores` snapshots both stores before writing and restores them on failure. The commit and the retention enqueue are sequenced inside `commit_and_enqueue`, the sole route to a durable adoption commit: once the store write succeeds it enqueues a pending 30175 for every member copy the add wrote or reactivated and a pending 30176 for the team, so a crash before the next boot reconcile cannot lose the only adopted copy. A provenance match on an already-active copy is retained too (not just a reactivation): a recovery retry after a crash between the persona write and post-commit retention finds the copy active with no 30175 row, and `plan_add` short-circuits once the team row exists, so this reuse branch is the only place that retry can re-enqueue the orphaned member head. Retaining unconditionally is conservative, not exact — an active copy still referenced by a standalone managed agent can already hold a live head, and re-retaining only bumps it monotonically; reused built-ins are handled separately and never reach this branch. A byte-identical reused built-in and an idempotent replay write nothing and enqueue nothing; a failed commit propagates and enqueues nothing. Enqueue is best-effort per row (the boot reconcile is the backstop). The frontend refreshes via the `useAddTeamFromCatalogMutation` query invalidation in #3995, so no `agents-data-changed` emit is needed here. **Startup reconcile** (`event_sync.rs`): `reconcile_team_catalog_heads_at` walks all retained 30178 heads at boot: republishes heads whose content changed, tombstones heads whose team or member was deleted, skips unshared heads and unchanged content. Multi-team continuation — all shared teams processed in one pass. **Cross-device catalog retention** (`commands/personas/inbound.rs`): both recovery paths above — the boot reconcile worklist and the interactive `refresh_or_retract_shared_head_at` — key off a retained 30178 row and guard-return without one. A second device therefore never retained the owner's own catalog head published from another device, so its later edit or delete could never supersede or retract that discoverable head. The inbound reconcile now retains an inbound 30178 head as this device's publication witness through `retain_inbound_catalog_witness`, a self-gating dispatcher invoked unconditionally on the production non-deletion path: newest-wins via `retain_inbound_event`, arrival-scoped, no local JSON store, and deliberately **no** refresh or republish on arrival — a 30178 arrival is either this device's own echo or the other device's publication, and rebuilding on either would make two devices ping-pong identical heads. Retention advances the witness and stops. The tombstone router accepts a kind:5 covering a 30178 coordinate, so an inbound deletion purges the retained head on the receiving device (the covered-head purge already happens inside `commit_inbound_tombstone_with_store`; a 30178 head has no local record to remove). After a successful inbound persona/team upsert, this device refreshes the affected shared heads so the community catalog tracks the inbound edit (persona edit → every team whose resolved members include it, resolving the local persona `id` by d-tag; team edit → that team's head); after an inbound tombstone, a team deletion retracts its 30178 coordinate and a persona deletion refreshes the teams that listed it. The refresh is idempotent across devices: `refresh_or_retract_shared_head_at` skips the publish when the rebuilt projection is byte-identical to the retained head and still shared, so the editing device's own published head triggers no churn republish on the receiving device. **Executable-text concealment gate** (`team_catalog.rs`, `definition_validation.rs`): `validate_team_catalog_content` — the single chokepoint both the publish builder (`build_team_catalog_content`) and the adopt parser (`team_catalog_content_from_event`) funnel through — now rejects invisible, default-ignorable, and bidirectional-override characters (e.g. U+200B, U+2066, U+202E) in every field delivered verbatim to the ACP harness or rendered as reviewed identity in the catalog UI. This is the same invariant the persona catalog already enforces at its own parse boundary (`persona_catalog::parse_agent`); the 30178 boundary was the outlier. Member `display_name` + `system_prompt` go through `validate_agent_definition_text` per member (exact parity with `parse_agent`: display-name rule with no layout controls, prompt rule allowing `\n`/`\t`); `name_pool` entries take the display-name rule since they are minted verbatim as instance display names; team `instructions` take the visible-text rule with layout controls allowed, since they reach `BUZZ_ACP_TEAM_INSTRUCTIONS` multiline. The team `name` takes the display-name rule (no layout controls) and the `description` takes the visible-text rule with layout controls allowed, since both are rendered verbatim in the catalog UI as reviewed identity. `validate_visible_text` is exposed `pub(crate)` from `definition_validation.rs` and re-exported via `managed_agents`. A signed, shared, current head can no longer smuggle concealed control characters into executable configuration or reviewed catalog text through either the publish or the adopt path; emoji (VS16/ZWJ) names and multiline instructions still pass. **Types**: `TeamRecord` and `AgentDefinition` extended with `shared`, `catalog_source`, `team_catalog_source` fields. All commands registered in `lib.rs`. ## Tests - `team_catalog/tests.rs`: projection, size contracts, member-key stability, tombstone rollback, fixture matrix - `adopt/tests.rs`: head verification, store planning, provenance, rollback - `team_catalog/tests/concealment.rs`: the chokepoint rejects default-ignorable (U+200B) and bidi controls (U+2066, U+202E) in member `display_name`, `system_prompt`, `name_pool`, team `instructions`, and the team `name`/`description`, on both the publish and adopt paths; an emoji-bearing display name and multiline instructions still pass, so no legitimate team becomes unshareable - `team_catalog/tests/reuse_hint.rs`: a member pairing a real built-in's slug with that built-in's genuine `projection_hash` but carrying unrelated reviewed fields is rejected at the parse boundary, so adoption can never substitute the recipient's built-in for the reviewed projection; an honestly-stamped built-in reuse hint (including an uppercase form of its true hash) still passes the boundary. Removing the boundary recompute lets the tampered member validate, proving the test discriminates the substitution class - `adopt/tests/reuse.rs`: `reusable_builtin` reuses a local built-in for an exact-match hint (one record, no copy), reuses it just the same when the genuine hash is uppercased (case-insensitive, matching the boundary — one record, not two), and falls through to an authoritative embedded copy when the hash does not match. Comparing the hash case-sensitively turns the uppercase case red (two records instead of one reused) - `adopt/tests/concealment.rs`: a signed, shared, current head carrying a bidi override drives the `add_verified_team` sequence (verify+parse → `plan_add` → `commit_and_enqueue`) through real temp stores and a real retention scope, asserting the head is rejected AND the personas store, teams store, and retention rows are all left byte-unchanged. Stripping the concealment call at the chokepoint turns it red — the parse then succeeds and both stores are written, proving the test discriminates a validate-after-write regression, not just an error return - `adopt/tests/retention.rs`: adoption drives `commit_and_enqueue` through a spy commit + a real temp-dir retention scope and asserts persisted pending rows — commits-then-enqueues (30175 per minted member + 30176 team), a failed commit enqueues nothing, an idempotent replay skips both the commit and the enqueue, a reused built-in retains only the team, a reactivated copy is re-retained, and a partial-commit crash recovery (active member copy with no retention row, team row absent) re-enqueues the orphaned member's 30175. Deleting the enqueue inside the seam turns these red — the wiring, not just the helper, is protected - `sharing/tests.rs`: publish/queue lifecycle plus three concurrency gate tests. (a) `concurrent_flushes_never_land_the_head_after_its_tombstone` prepares a share, runs a concurrent delete's purge+tombstone, flushes the tombstone to a recording relay, then releases the delayed share and asserts the purged 30178 head is never published after its tombstone and no pending row survives (removing the lock turns it red — the relay sees the resurrected head after the tombstone). (b) `a_stalled_scope_does_not_block_publication_in_another_scope` pins one scope's flush mid-POST on a stalled relay and runs a second scope's flush to completion, asserting it publishes without waiting (re-globalizing the key turns it red). (c) `a_stalled_relay_releases_the_publisher_lock_within_the_bound` proves a never-completing POST returns within `PUBLISH_TIMEOUT`, leaves the row pending, and releases the lock so a subsequent same-scope flush proceeds (removing the timeout turns it red). `test_relay_rejection_stays_durably_queued` asserts queued + still-pending rather than a relay-message string, matching the flush-routed contract where the rejection text is no longer surfaced - `pending/tests.rs`: share/unshare/tombstone lifecycle, edit refresh/retract, tombstone timestamp domination, typed outcomes, cross-device catalog convergence — device B retains device A's inbound head then supersedes it on a member edit and tombstones the coordinate on a delete, a byte-identical rebuild is a no-op (`Noop`, `created_at` untouched), and an inbound retention alone queues no outbound publish (the no-ping-pong guard). Neutralizing the inbound retention leg turns the supersede/tombstone regressions red — B stays blind (`Noop`, no dominating tombstone) — proving they discriminate the load-bearing leg - `catalog_reconcile_tests.rs`: a signed kind:30178 head is driven through the real production entrypoint `reconcile_inbound_persona_event_blocking` over a `MockRuntime` `AppHandle` (retention scope resolved from the handle's `app_data_dir` under an overridden `$HOME`/`$XDG_DATA_HOME`), asserting the arrival witness is retained at the owner coordinate with `pending_sync=false`, stored verbatim, and no outbound publish is queued. An early return for `KIND_TEAM_CATALOG` immediately before the production `retain_inbound_catalog_witness` invocation turns it red, proving the seam under test is the production dispatch path and not a test-only shim - `pending/tests/gate.rs`: flush driven through a stub relay that logs every `POST /events` with its accept/reject status and enforces the real ±900s ingest gate, for both 30176 and 30178 — within-window publish+dominate; beyond-window stays-pending with **zero POSTs** (the gate never receives a rejectable event); and the delayed/offline-retry case where a tombstone signed strictly past a then-future head has aged more than 900s into the past, so flush must re-date to `now` to publish. The reversal check — restore the byte-frozen replay in `persona_events.rs` and the delayed-retry and zero-POST assertions go red — is what proves the suite discriminates the fix from the rejected implementation - `teams/tests.rs`: 30176 tombstone timestamp domination and no-head fallback - `event_sync_team_catalog_tests.rs`: reconcile scenarios including multi-head continuation - 26 shared JSON parity fixtures (`tests/fixtures/team_catalog_content/`) consumed by both this PR's Rust tests and #3995's TS tests ## Durable ordering, deletion reconcile, and relay-contract alignment The catalog paths above sit on the shared inbound/deletion retention seam. This PR makes that seam's ordering structural rather than conventional, adds a negative-side (deletion) counterpart to the existing positive-side boot backstop, and aligns inbound resolution with the relay's actual soft-delete and NIP-33 winner rules. **Preflight-then-commit (`commit_inbound_with_store`, `commands/personas/inbound.rs` + `retention.rs`).** A named primitive runs the fallible store mutation first and advances the durable retention head only on success; an event that loses preflight returns `Skipped` without touching the store. The persona/team upsert arms and the inbound kind:5 removal path all route through it, so no inbound arm can advance the head ahead of the store write it represents. The managed-agent arm keeps its own preflight (its runtime transition must not run for a skipped event) and still advances the head only after `save_managed_agents`. **Atomic + monotonic tombstone helpers (30175 / 30176 / 30177).** The three ordinary tombstone helpers (`commands/personas/pending.rs`, `commands/agents_pending.rs`, `commands/teams/mod.rs`) each read the prior head, sign, delete, and retain inside one `BEGIN IMMEDIATE` transaction, and sign the kind:5 with a `created_at` that strictly dominates the prior head (`monotonic_created_at(prior_head)`). This mirrors the 30178 `tombstone_team_catalog_coordinate` precedent. The three siblings deliberately duplicate the `BEGIN IMMEDIATE` shape rather than sharing a helper this round — the shared-helper consolidation is deferred to the persona-tombstone follow-up PR where the flush-replay class already lives. **Deletion reconcile (`event_sync.rs`, negative-side counterpart of the positive-side boot backstop).** The positive side already reconstructs missing retained heads at boot from surviving disk records. Deletion is the asymmetric gap: an atomic tombstone helper preserves the head on failure, but boot reconcile enumerated disk records — deletion already removed them — so an orphan retained head was never tombstoned. The deletion reconcile enumerates retained 30175/30176 heads and tombstones only genuine orphans (a retained head with no matching disk record), routing through the now-atomic helpers. It reconciles only against a successfully parsed store: a truncated, malformed, or wrong-shape JSON store fails loudly (never triggers a tombstone); a missing file is treated as empty. Managed agents (30177) are excluded by design: their inbound sync retains a head *without* minting a local disk record (agents carry device-local secrets that can't come from a relay event), so a retained 30177 head with no matching record is the normal cross-device state for every agent created on another device — not a lost deletion. Sweeping it would tombstone and archive another device's live agents at boot. Agent deletion-retry therefore stays a pre-existing gap owned by the direct delete path (tracked in Follow-ups). **Inbound relay-contract alignment (`retention.rs`, `commands/personas/inbound.rs`).** Two inbound rules now match the relay: - *Equal-second tie-break.* `inbound_event_outcome` previously treated every equal timestamp as stale; the relay resolves equal `created_at` by lowest event id. Preflight now matches the relay tuple — strictly newer timestamp wins, equal timestamp resolves by lower event id, an exact echo skips — so two devices authoring different same-second successors converge instead of one silently republishing an event the relay refuses. - *Covered-head resolution on inbound kind:5.* `reconcile_inbound_tombstone` previously consulted only the retained kind:5 row, so a historical tombstone replayed after a newer recreation deleted the recreated record and never purged the covered head. Preflight now resolves both the tombstone row and the covered `(target_kind, owner, d_tag)` head: a target head strictly newer than the tombstone preserves JSON and skips; an actually-covered head is removed from disk first (fallible), then the tombstone row commit and covered-head purge happen atomically. A JSON-save failure advances neither, so the identical event stays retryable. **Atomic agent-archive coupling (`commands/agents_pending.rs`).** The 30177 tombstone and the 9035 archive request now enqueue in one `BEGIN IMMEDIATE` transaction, with the archive's `persona_id` derived from the retained head (`persona_id_from_head`) rather than the deleted record, where it survives the tombstone as owner-signed historical alias data. The standalone `archive_managed_agent_pending` production callers are removed — there is no double-enqueue path. Unlike personas/teams, this coupling is not re-enqueued by the boot deletion reconcile (30177 is excluded, above): a crash after the disk-authoritative record is removed but before this transaction commits leaves agent deletion-retry a pre-existing gap owned by this direct delete path (tracked in Follow-ups). ## Follow-ups `publish_prepared_persona` (`commands/personas/sharing.rs`) has the identical direct-submit-outside-the-lock race for 30175 persona heads that this PR fixes for 30178 team heads. It predates this work and is tracked in the separate persona-tombstone follow-up PR, not folded here. The app-wide `http_client` leaves reqwest's connect/read/total timeouts unset (`app_state.rs` builder configures only pool options). This PR bounds the team-publish call site with a `tokio::time::timeout`, but every other consumer of the shared client remains exposed to a non-responding endpoint. A client-wide default timeout is the broader fix; it is pre-existing on main and affects all consumers, so it is out of scope here. Managed-agent deletion-retry: the atomic 30177 tombstone + 9035 archive is best-effort, and unlike personas/teams it is deliberately excluded from the boot deletion reconcile (a device-local-absent 30177 head is the normal cross-device state, not a deletion). A crash between the delete's store write and its retention enqueue therefore has no boot backstop for agents. The durable fix is a local deletion intent written before the record is removed; deferred rather than folded into this round. Stack: this PR → [#3995](https://github.com/block/buzz/pull/3995) Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>