mirror of
https://github.com/block/buzz.git
synced 2026-10-29 21:41:33 +00:00
am/thread-window-websocket
664 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
am
|
7db92b7d93 |
fix: isolate finite websocket window work from shared handler capacity
Bound in-flight finite delivery by process, community, and principal before the shared handler permit. Preserve live queue headroom under concurrent finite senders and restore the terminal EOSE failure fixture. Co-authored-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: Kalvin Chau <kalvin@block.xyz> |
||
|
am
|
abbe89e525 |
fix: preserve live fan-out capacity during websocket thread windows
Reserve outbound queue headroom for live subscriptions and clear consecutive backpressure on successful finite enqueues. Exercise interleaving and deadline behavior with the production send paths. Co-authored-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: Kalvin Chau <kalvin@block.xyz> |
||
|
am
|
32416d88e6 |
fix: bound websocket thread window enqueue waits
Wait for the writer to drain within the finite request deadline, terminate when a CLOSED frame cannot be queued, and select parser and wire regressions in CI. Co-authored-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: Kalvin Chau <kalvin@block.xyz> |
||
|
am
|
18f97852ee |
fix: terminate truncated websocket thread windows
Make queue-truncation and replacement-topic regression guards falsifiable; cancel finite windows when EVENT or EOSE cannot be queued, and document incomplete-page retry behavior. Co-authored-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: Kalvin Chau <kalvin@block.xyz> |
||
|
am
|
b082ae71f8 |
fix: guard websocket thread windows with batch authorization
Share token-scoped batch access checks with HTTP, terminate malformed cursor requests per subscription, replace live subscriptions before finite windows, and suppress EOSE after failed enqueue. Co-authored-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: Kalvin Chau <kalvin@block.xyz> |
||
|
am
|
02450f65bb |
feat: serve thread windows over websocket req
Co-authored-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz> Signed-off-by: Kalvin Chau <kalvin@block.xyz> |
||
|
Kalvin C
|
5a349db91c |
feat(relay): add opt-in newest-first thread windows (#7823)
## Summary Add an opt-in, newest-first thread window to the authenticated HTTP query path. A filter with `thread_window: true` returns: - reply rows ordered by `(created_at DESC, id ASC)` with an exact composite cursor; - optional bounded reactions, edits, deletions, and delete-of-aux closure; - one relay-signed `kind:39007` bounds event bound to the normalized host, authenticated reader, channel, root, request cursor, kinds, depth, limit, and aux choice. Filters without the flag keep the existing oldest-first thread behavior. Channel windows and `kind:39006` are unchanged. No client opts into this mode in this PR. ### Why extend NIP-CW NIP-CW already defines relay-computed, cursor-paged conversation views over extended NIP-01 filters. Thread windows use the same protocol ideas: ordinary signed rows, composite keyset pagination, optional aux closure, and relay-signed bounds. Adding a thread mode keeps those related wire contracts together instead of creating a second NIP with duplicated terminology and compatibility rules. Extending a draft, optional NIP with an additive mode is normal protocol evolution. Existing fields and meanings are not reinterpreted: - `top_level: true` remains channel mode; - `kind:39006` remains channel-window bounds; - `thread_window: true` selects the new mode; - `kind:39007` has a distinct identity and root/request binding; - extension-unaware clients never send the selector; - legacy thread queries remain unchanged. NIP-CW documents only the wire contract and relay semantics. Database plans, replica internals, and rollout procedures remain implementation concerns. ### Why thread windows use independent modules The implementation deliberately keeps `thread_window.rs` separate in `buzz-core`, `buzz-db`, and `buzz-relay` rather than adding branches throughout the existing channel-window path. The modes share pagination concepts, but their invariants differ: - channel mode selects top-level rows; thread mode requires one root and depths `1..=depth_limit`; - thread mode accepts only conversation row kinds and rejects unknown/conflicting fields; - thread bounds bind the root and authenticated reader because cross-channel aux visibility can differ by reader; - thread aux includes the root and requires complete two-hop closure; - thread mode refreshes writer authorization before selection and before signing; - historical cursor pages may use a proved replica transaction, but fallback must restart both aux hops on the writer; - legacy thread traversal is oldest-first and must not inherit descending cursor semantics. Forcing these rules into channel-window code would create a mode flag across parsing, SQL predicates, bounds construction, authorization, replica fallback, and tests. That abstraction would not remove complexity. It would distribute it. Separate modules keep each contract local and leave the established channel and legacy paths untouched. ### Technical design - Strict request parsing normalizes UUIDs, event IDs, kinds, defaults, and cursors before computing the response binding. - SQL applies community, root, channel, depth, kind, deletion, and cursor predicates before `limit + 1`. - Ordered metadata scanning plus exact event-key lookup avoids the stale-statistics plan that previously exceeded the four-second SQL budget on a 10k-reply thread. - Aux work is bounded by target batches, query count, raw candidate count, response bytes, SQL timeouts, and one shared HTTP deadline. - Fresh authorization-set comparison catches both grants and revocations during closure. The relay returns retryable `503` rather than signing an incomplete page. - Replica failure permanently degrades the request to the writer and restarts closure from its original targets without resetting budgets. - Migration `0049_thread_window_index.sql` adds the mixed-direction index used by newest-first scans. It validates a prebuilt index and bounds startup lock/build time. ### Tradeoffs - The stale-statistics-safe plan is intentionally less clever. Measured warm 100k deep-cursor execution increased from roughly 0.30 ms to 4.80 ms, plus about 8 ms planning, in exchange for avoiding a reproduced multi-second timeout after rapid 10k-thread insertion. - Bounds are authoritative only for the served page. There is no cross-page snapshot guarantee, and replica insertion coverage does not imply latest edit/deletion/aux visibility. - Strict parsing rejects unsupported constraints instead of silently serving a different row set. - Required aux closure fails closed. The relay does not omit aux and sign plausible-but-incomplete bounds. - This PR ships relay capability only. Client rendering, retry UX, live-history overlap, and compatibility fallback remain separate work. ### Related issue No existing issue found. This work follows the thread-loading design and review discussion. ### Testing - `cargo test -p buzz-core`: 258 unit tests and 2 doctests passed. - `cargo check --workspace --all-targets`: passed at the rebased implementation head. - `cargo test -p buzz-db`: 131 non-PostgreSQL tests passed; 275 infrastructure tests were ignored as declared. - Focused PostgreSQL migration/runtime regressions: 3/3 passed against an isolated PostgreSQL container, including both corrected migration-ledger tests. - `cargo fmt --all --check`: passed. - Pre-rebase feature validation passed 398/398 PostgreSQL tests, signed-wire multi-tenant/auth/pagination/aux/revocation probes, clean Compose boot/reboot, and full core tests. - Independent HA validation exercised three relay pods and a physical PostgreSQL standby on the same runtime logic before the final documentation-only cleanup. - Independent final-delta review confirmed exactly two changed files, byte-identical executable migration SQL after comments/diagnostic normalization, matching prebuild/catalog/recovery instructions, and `git diff --check`; no runtime receipts were relabeled to the new SHA. - The pre-push hook ran repository checks but its Rust unit lane exited 100 without exposing an individual failed test in the captured multiplexed output. The independently run affected-package checks above passed; current-head repository CI remains required. Known unrelated baseline behavior remains: the mesh-demo echo test can return 504, and same-second archive → unarchive can deduplicate its membership notification. Neither path is changed here. --------- Signed-off-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz> Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz> Co-authored-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz> Co-authored-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz> |
||
|
Salman Mohammed
|
5621006bcf |
refactor: move agent Git bootstrap into ACP harness (#7819)
discovered in https://github.com/block/buzz/issues/7742#issuecomment-5783512985 ## Summary Goose uses its native shell, so it bypassed the Git identity and signing setup inside `buzz-dev-mcp`. Agent commits could use the host Git identity, while MCP shells had a separate config. This moves Git setup into `buzz-acp`, where both paths receive the same agent identity, Nostr signer, and relay-scoped credential helper. Before: ```text Buzz Desktop └─ buzz-acp ← Desktop supplied relay Git authentication ├─ Goose → native shell ← no agent Git author or signing setup └─ Buzz Agent └─ buzz-dev-mcp └─ shell ← MCP created the keyfile and Git signing config ``` After: ```text Buzz Desktop or Sprig └─ buzz-acp ← owns the keyfile, identity, signing, and scoped credentials ├─ Goose → native shell ← inherits the complete Git environment └─ Buzz Agent └─ buzz-dev-mcp └─ shell ← receives the same Git environment via mcpServers[].env ``` The harness keeps the signing key in a private temporary file and removes it on normal exit, startup failure, and graceful termination. It composes inherited `GIT_CONFIG_*` entries, forwards the complete block to MCP servers, and bundles the helpers through the ACP multicall entrypoint. For standard `buzz-acp` launches, Desktop and Sprig no longer write Git config. A custom Desktop ACP command bypasses the harness, so Desktop preserves its prior relay-scoped credential helper for that supported override. This changes Git settings for every ACP runtime. The startup signal path also changes so a termination during adapter initialization removes the keyfile. The temporary file can remain after SIGKILL or a machine crash. ### Related issue Related: #6177 covers stricter commit identity enforcement and touches the same Git paths. That PR and this one need to be reconciled before either lands. No issue was found for this runtime bootstrap gap. ### Testing - Ran the ignored `git_runtime_tests` with real Goose 1.50.0 and Buzz Agent ACP sessions. `real_goose_native_git_shell` passed again on this PR branch with `BUZZ_TEST_BIN_DIR` pointing to built binaries. Goose made a signed commit and tag through its native shell; Buzz Agent did the same through its MCP shell. Both verified signatures, identity, credential scoping, and keyfile cleanup. - Built the Linux Sprig image and ran `scripts/test-sprig-image.sh buzz-sprig:acp-git-bootstrap`. The container made and verified a signed commit and tag, then confirmed keyfile removal after SIGTERM. - `just ci` passed on the original branch. After rebasing onto `main` and addressing review feedback, the ACP Git bootstrap integration tests, focused desktop smoke test, custom-command regression test, formatting, and ACP/Desktop Clippy passed. The pre-push hook hit two timing-sensitive ACP deadline tests under parallel load; both passed alone and the complete ACP package passed serially (964 library, 2 Git bootstrap, 9 pool lifecycle tests). The push used `--no-verify` after those checks; GitHub CI is rerunning on the new head. - Under injected `BUZZ_ACP_ALLOWED_RESPOND_TO`, `BUZZ_ACP_LAZY_POOL`, and `BUZZ_ACP_IDLE_POOL_SLEEP`, the clean-env serial package command in `crates/buzz-acp/TESTING.md` passed: 964 active library tests, 2 Git bootstrap tests, and 9 pool lifecycle tests. Both named Databricks OAuth tests passed locally; another reviewer reproduced their environment-specific failures on `main`. - No authenticated relay clone or push was run. The local probes used disposable repositories and did not contact a relay. Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> |
||
|
ravarora2
|
4142d2b84e |
Use worker snapshots for relay storage metrics (#7845)
## Summary Make the relay read completed storage-accounting snapshots from PostgreSQL by default. An environment with no snapshot emits no storage metrics. When a worker publishes its first result, the relay picks it up on the next successful leader usage tick, without a restart or configuration change. This changes how storage usage is measured and reported. **S3 remains the object store. PostgreSQL stores the completed counts, byte totals, community breakdown, and calculation metadata—not the objects themselves.** ### Why move relay storage accounting entirely to database reads? Today the relay supports two sources: its own S3 scan (`inline`, the default) and the worker's saved snapshot (`external`). That requires operators to deploy a worker and then coordinate a separate relay-mode change in each environment. The worker already owns the S3 listing and calculation. It runs in a separate process with its own object cap, memory limit, and deadline. Keeping a second scan inside the serving relay retains the resource pressure and failure modes that this isolation is meant to remove. PostgreSQL provides a durable handoff. The worker replaces one completed snapshot atomically after a successful calculation. The relay reads that row and exposes its values through the existing metrics endpoint. A relay restart or leadership change can load the last completed result without scanning the bucket again. There is deliberately no S3 fallback when a snapshot is absent or a read fails. Such a fallback would put the expensive scan back in the relay precisely when a worker is absent or unhealthy. It would also leave two calculation paths with different resource limits and freshness behavior. ### What happens if no worker exists? **A worker that has never published is a normal, successful inactive state.** The snapshot table must exist through the normal database schema, but it may contain zero rows. The reader returns success, emits no storage totals or failed-load gauge, and checks again on the next usage tick. It does not exit the relay, start a worker, or list S3. The relay checks for a completed database row; it does not inspect Kubernetes Jobs or CronJobs. Worker presence and snapshot presence are therefore different: | Situation | Storage-reader behavior | |---|---| | No worker and no snapshot row | Return success, emit no storage metrics, and query again next tick. | | Worker starts later and publishes its first row | Read and emit it on the next successful leader usage tick; no restart or mode change. | | Worker is removed or fails, but its last valid row remains | Continue reporting the last completed totals with their increasing age. | | Query succeeds but a previously present row has been removed | Clear the cache and stop refreshing storage series; existing exporter series expire through the idle timeout. | | Database read fails, times out, or returns an invalid snapshot | Log a warning, set `buzz_storage_snapshot_load_ok=0`, retain any last-good cached totals, and retry next tick. | | Snapshot table is missing | Treat this as a schema error, not normal worker absence; follow the failed-read behavior above. | | `BUZZ_STORAGE_METRICS=off` | Skip the snapshot query and all storage metric emission. | No row means “not measured,” not zero bytes. A valid completed snapshot of an empty bucket can report zero. If a read fails before any good snapshot has been cached, only failed-load health is emitted. These reads happen in the existing background metrics task, not the relay startup path. A storage-read failure does not request process exit or change readiness directly. A wider database outage can still affect the relay's existing database-dependent operations and readiness. ### Read, parse, and emit path ```text Worker: S3 listing -> BucketSnapshot -> JSON -> PostgreSQL Relay: PostgreSQL -> JSON -> BucketSnapshot -> existing metric gauges Collector: relay /metrics endpoint -> Datadog ``` 1. `main.rs::run_storage_sweep_tick` calls the reader from the existing leader-only usage task. The default interval remains 300 seconds. 2. `storage_sweep.rs::run_storage_metrics_tick` coordinates the read, cache update, failure health, and metric emission. 3. `refresh_persisted_snapshot` calls the existing `Db::load_storage_accounting_snapshot()` method. A five-second timeout covers pool acquisition and the query. It uses the relay's existing pool. 4. The existing query in `buzz-db/src/store/storage_accounting.rs` reads `snapshot`, `completed_at`, `duration_ms`, `max_objects`, and `code_sha` from the singleton row. 5. `serde_json::from_value(stored.snapshot)` decodes the JSON into the existing `buzz_media::BucketSnapshot` type. Rust infers that type from `cache_persisted_snapshot`'s argument. The generated `Deserialize` implementation handles all fields, including the UUID-keyed `per_community` map. 6. `emit_cached_storage_metrics` sets the existing physical/logical totals and community byte/object gauges. The existing Prometheus exporter exposes them for Datadog collection; this code does not send snapshot JSON to Datadog. The worker's `BucketSnapshot`/`CommunityStorage` structures, JSON field names, SQL publication method, and database schema are unchanged. The old external-mode JSON conversion moves out of `main.rs` into the reader helper. No new format or hand-written field parser is needed. Duration and object-cap metadata are validated before replacing the cache. The five-second relay read timeout is separate from the worker's 30-second startup acquisition budget introduced in #7770. This PR does not change the worker's pool or startup logic. ### Configuration and code changes | Configuration | Before | After | |---|---|---| | Unset, `inline`, or `on` | Relay-local S3 scan | Read completed database snapshots. `inline` logs one migration warning. | | `external` or `snapshot` | Read completed database snapshots | Continue reading snapshots. | | `off` | Disabled | Disabled. | | Unknown or empty value | Disabled with a configuration error | Same. | Existing `inline` settings intentionally become reader aliases. Merely changing the default would leave older deployments on their explicit `inline` setting and preserve the need for coordinated configuration PRs. Remove relay-local scan scheduling, in-flight scan state, scan configuration, and obsolete scan-attempt tests and health metrics. Preserve the physical/logical total and per-community metric names, leader-only emission, community scope filtering, and cleanup of old community labels. The shared relay-state field remains in place; its comment now describes a cached worker result. The Helm chart defaults to `relayMode: external` while leaving `storageAccounting.enabled: false`. Its rendering test verifies that the reader is configured even when no worker is created. The reader regression tests cover missing rows, later publication, updates, invalid data, timeout, removal, reactivation, and the explicit off switch. `docs/storage-accounting.md` describes this operating model. ### Freshness and rollout impact `buzz_storage_snapshot_age_seconds` uses the worker's original completion timestamp. Re-reading an old row never makes it fresh. `buzz_storage_snapshot_load_ok=1` means the row was read and decoded; it does not prove that the worker's latest attempt succeeded. Monitor snapshot age against the worker's schedule and allowed runtime, and monitor Job failures separately. The old relay scan-attempt health gauges are retired because the relay no longer runs those scans. Deploy this relay version through the usual release process. Once it reaches an environment, workers can be enabled independently. Already-released charts that still set `inline` work with the new binary; future chart releases default explicitly to `external`. An intentional `off` override still requires an operator to enable the reader. **Environments with no completed snapshot lose relay-calculated storage totals after this upgrade.** That is the intended tradeoff for removing scans from the serving process. Environments with a prior snapshot continue showing its last completed totals and age. This PR does not install workers in additional environments or deploy the new relay image. ### Related issue Related to #4601. Follows the worker startup fix in #7770. The separate fleet-usage collector work in #7176 overlaps this code and may need reconciliation when it merges. ### Testing Built and ran the relay locally against isolated PostgreSQL and Redis with the legacy `inline` setting, five-second usage ticks, and a 15-second metric idle timeout. The same process was exercised through these steps: 1. Start with an empty snapshot table: relay becomes ready and exposes no storage series. 2. Publish a snapshot containing 123 bytes: the community gauge becomes 123. 3. Replace it with 456 bytes: the gauge becomes 456. 4. Remove the row: storage series expire without fabricating a zero-byte measurement. 5. Publish 789 bytes: metrics reactivate without restarting the relay; readiness remains healthy. The local S3 endpoint recorded zero requests. The unrelated Git conformance startup probe was disabled for this isolated reader check; the test covers storage accounting, not every other use of S3 in the relay. No new relay image was deployed to staging or production during this test. The broad local integration run also hit an existing `p0_pool_acquisitions_use_typed_operation_pairs_without_other` failure in `buzz-db/tests/observability_source.rs`. The same source-only test fails on unmodified `main` at `0ef7a2222`; this PR changes neither the test nor its failing source. The full local `just ci` run stopped at mobile native-asset setup: the `objective_c` build hook received no SDK path from `xcrun`. Mobile tests did not run. Generated with Codex Signed-off-by: Ravneet Arora <rarora@squareup.com> |
||
|
Tom Brow
|
0ef7a2222e |
fix(mobile-infra): render push grant lifetimes as decimal in chart 0.3.2 (#7820)
A valid 30-day push gateway grant lifetime could render in scientific notation (e.g. `2.592e+06`) in the chart, which the gateway binary does not accept. Always ormat the integer as decimal text, preserving the strict gateway parser. ### Testing Added regression coverage that feeds actual Helm-rendered environment values into the production config parser. Covers defaults, values files, command-line overrides, exact lifetime boundaries, and invalid input rejection. The default case failed before the fix and passes afterward. Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Wes
|
0cc63fe3e6 |
fix(agent): preserve Databricks Opus UC reasoning and tool continuation (#7840)
## Summary Fix `reasoning_effort: Extra inputs are not permitted` for Databricks v2 model `data_workflow_tools.goose.goose-claude-opus-5-5` without dropping explicit High. - Add one provider-qualified exact capability record: Anthropic Messages, adaptive thinking, `output_config.effort`, medium upstream default. Mirror exact-record precedence in Rust/TS; keep uncurated UC services neutral and existing GPT routing unchanged. - Preserve ordered native content, including signed/empty thinking and redacted blocks, across tool results. Do not cache-stamp thinking blocks or leak native state into Chat reasoning details. Discard truncated native state; reject more than 64 native calls before any tool executes instead of altering signed content. - Add production-dispatch/ACP regressions and six shared capability vectors. No relay/ACP protocol, schema, settings, inheritance, retries, or other harness changes. ## Validation At final head `82f43f8d07bfcc3f4af492afc0a58a02f8e549c4`, clean tree: - Rebuilt real `buzz-agent`, drove ACP with a harmless MCP fixture against the actual service. Two POSTs to `/ai-gateway/anthropic/v1/messages`, both HTTP 200, exact FQN, High via `output_config`, adaptive/summarized thinking, no `reasoning_effort`. First response contained signed thinking + tool use; second request preserved signature/content order and paired result; final ACP stop `end_turn`. No credentials, signature contents, or conversation data included here. - Mandatory pre-commit/pre-push checks passed: formatting/lint, file-size gate, Rust test lanes, desktop typechecking and full JS tests, Tauri clippy/tests. - Full `cargo test -p buzz-agent` passed on the pre-commit tree (only hook formatting followed), including the new native truncation/65-call ACP regressions, ordered interleaved signed/redacted two-tool capture, nearby-ID negatives, and existing provider regressions. - Independent API/replay, regression-boundary, and Rust/TS capability reviews cleared. Full `just ci`, packaging and manual desktop UI testing were not run locally; repository CI remains the merge gate. Cross-harness internals and other UC service contracts are outside scope. ## References - [Databricks model services](https://docs.databricks.com/aws/en/ai-gateway/query-model-services) - [Anthropic thinking preservation](https://platform.claude.com/docs/en/build-with-claude/thinking#thinking-block-preservation-by-model) - [Anthropic effort](https://platform.claude.com/docs/en/build-with-claude/effort) - Originating Buzz channel: `fccdea96-0f31-4343-ba6a-324214dc8d73` - Originating thread: buzz://message?channel=fccdea96-0f31-4343-ba6a-324214dc8d73&id=33f242923a317e269c1ff7b960cacd6ec1f3371eff2f97592857804c50fb100c Signed-off-by: Wes <wesbillman@users.noreply.github.com> Signed-off-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> |
||
|
Will Pfleger
|
b6a26556ec |
feat(canvas): add version history with atomic restore (#6780)
🤖 Adds append-only channel canvas history, restore, and relay-authoritative optimistic concurrency so stale edits cannot silently replace a newer canvas. ## What this adds - **Desktop history and restore** — browse revisions with author, timestamp, preview, and diff; restore an older revision as a new signed head after explicit confirmation. Persisted empty canvases remain reachable, mutation outcomes are announced accessibly, and focus moves to a stable destination after save or restore. - **CLI history and restore** — `buzz canvas history --channel …`, `buzz canvas get --channel … --revision <id>`, and `buzz canvas restore --channel … --revision <id>` use keyset pagination over the retained revision stream. - **SDK canvas builders** — optional `expected-revision` tags, bounded ancestry classification, and shared timestamp discipline keep first-party writes ahead of the head they read without propagating poisoned future timestamps. ## Authoritative concurrency contract Tagged canvas writes use compare-and-swap semantics in the relay and database: - `expected-revision=none` succeeds only when no live canvas exists. - An event ID succeeds only when it names the canonical live head. - A byte-identical replay succeeds only when that event is already the canonical live head. - A successful candidate must sort strictly ahead of the current head under `created_at DESC, id ASC`. - Malformed preconditions and stale or non-advancing writes are rejected before event rows, mentions, or fan-out are committed. The database serializes tagged writes, unconditional kind `40100` writes, and canvas soft deletion on the same `(community, kind, channel)` advisory key. Untagged `buzz canvas set` remains an unconditional write, but participates in that serialization boundary so it cannot race a tagged save or deletion. Deletion classification is derived inside the datastore transaction; unrelated event kinds do not take the canvas lock. ## Read and timestamp consistency Precondition and post-write ancestry reads request strong consistency from `/query`, pinning them to the writer when replica routing is enabled. Display-only history remains replica-eligible. First-party canvas writers stamp `created_at = max(now, head + 1)` with a conservative client skew limit. The relay applies a canvas-specific future-timestamp ceiling before persistence, preventing an accepted boundary head from making subsequent first-party writes invalid while preserving the broader ingest limit for other event kinds. ## Durable outcomes and recovery After an accepted publish, a failed verification read is reported as durable success with `verified: false` rather than a failed save. Desktop and CLI preserve the new event ID and direct users to History when verification is unavailable. A successful read walks the bounded, cycle-guarded `expected-revision` ancestry chain so transitive descendants are not misclassified as supersession. Canvas caches invalidate for every settled mutation outcome, including a retained but superseded revision. The 409 reconciliation path in `cmd_restore_canvas` now separately establishes persistence evidence from ancestry classification. `canvas_write_survived` checks reachability only; a secondary writer-pinned IDs lookup distinguishes "A absent" from "A present but superseded by a legacy write." The four-way outcome (survived / persisted-but-superseded / genuinely-absent / read-fail) is now correct for all lost-response shapes including unconditional legacy writes. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz> |
||
|
Tom Brow
|
7e96f91229 |
feat(mobile-infra): support development App Attest (#7744)
Development-signed iOS apps cannot enroll with the current gateway because their Apple App Attest AAGUID is rejected. Add an explicit personal development build feature and environment setting so a development gateway can use a developer's own application identity and sandbox APNs credentials. Ordinary gateway builds remain production-only. Special builds also default to production, enforce exactly the selected signed AAGUID, and retain certificate-chain, pinned-root, nonce, app-ID, key, credential-ID, and counter verification. Document the matching mobile signing configuration and expose the feature through an optional Docker build argument. ### Validation The development image deployed successfully to a personal Railway stack; private readiness, enrollment challenges, authenticated relay access, and APNs sandbox TLS/HTTP2 connectivity passed. Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Jordan Mecom
|
7881447c19 |
fix(audit): frame hash inputs with TLV (#7492)
Hash new audit entries from a TLV buffer built with one-byte field tags, big-endian u64 byte lengths, and raw values. Absent optional fields are omitted; present empty values retain their tag and zero length. This prevents bytes from moving between fields without restricting object IDs or inferring boundaries from field contents. Store an explicit `hash_version`. Migration 0048 labels existing rows as version 1, and new writers use version 2. Historical hashes are unchanged and mixed-version chains verify. The legacy encoding retains its original ambiguity; TLV protects new entries. Apply the migration before deploying the updated writers and verifiers. Tests cover both boundary collisions, absent versus empty fields, stable digests, unknown versions, mixed-version chains, and stored-row tampering. All 16 audit and 128 database unit tests passed. The six PostgreSQL audit tests passed against both migrated and fresh schemas, and a populated-row migration check preserved the historical hash. Full `just ci` passed. Related audit fixes: #4216 and #4249; issue #4173. Signed-off-by: Jordan Mecom <jm@squareup.com> |
||
|
ravarora2
|
a929532a04 |
fix(admin): allow cold storage worker DB startup (#7770)
## Summary The standalone S3 accounting worker can exit before its scan begins because database startup inherits the relay’s three-second connection budget. A cold DNS probe took about five seconds. Give `buzz-admin storage-snapshot` its own pool: one connection, no minimum idle connections, and a 30-second acquisition budget. Retry startup timeouts and transport failures at most twice, after two and five seconds. Total startup wait is bounded to 97 seconds. Configuration, authentication, TLS, and protocol errors fail immediately. Log each startup attempt’s elapsed time and error category without connection credentials. Relay defaults stay unchanged. Retries stop before lock acquisition; the same lock-owning database session still publishes the completed snapshot. ### Related issue Follow-up to #7543. No duplicate startup fix found. ### Testing Archived-command check: built a nextest archive, temporarily moved `target/debug/buzz-admin` out of the checkout, and ran `scripts/postgres-test-run.sh --archive-file <archive>`. Both tests failed before the path fix and passed afterward. The tests use nextest’s relocated executable path, with the original Cargo path retained for local runs. Manually restored the old startup call and ran `scripts/postgres-test-run.sh -p buzz-admin --test postgres_storage_snapshot -E 'test(storage_snapshot_retries_then)'`. It failed before reaching S3; restoring the fix passed. This confirms the test exercises the actual worker command. Local `just ci` passed all stages before mobile tests, which stopped at SDK discovery in the `objective_c` native hook. The default Xcode tools report an unaccepted license. Two existing PostgreSQL tests hit connection timeouts in the parallel run and passed individually. After merge, build a worker image and repeat the cold-pod staging smoke test through snapshot publication before changing relay metrics mode. Generated with Codex --------- Signed-off-by: Ravneet Arora <rarora@squareup.com> |
||
|
Will Pfleger
|
77729abfb6 |
feat(relay): add admin HTTP routes for member restriction management (#7302)
## What
Three admin HTTP routes under `GET|DELETE /api/admin/v1/members/...`:
- `GET
/members/restrictions?communityId=<uuid>[&limit={1-200}][&cursor=<token>]`
— lists active bans and timeouts with stable keyset pagination
- `DELETE /members/{pubkey}/ban?communityId=<uuid>` — lifts an active
ban; 409 if no active restriction
- `DELETE /members/{pubkey}/timeout?communityId=<uuid>` — lifts an
active timeout; 409 if no active restriction
## Auth
All three routes pass through the existing NIP-98 authorization gate.
Both DELETEs call `require_mutation_principal`; GET follows the
established read-authorization pattern.
## Atomicity
Both DELETE routes use transactional DB helpers
(`unban_member_with_audit` / `untimeout_member_with_audit`) that perform
the conditional lift UPDATE and the `moderation_actions` INSERT in one
SQL transaction. If the audit INSERT fails the lift rolls back; the
restriction stays active.
## Expired-ban predicate
`unban_member` WHERE clause requires `banned AND (ban_expires_at IS NULL
OR ban_expires_at > now())` — matches all read paths. An already-expired
ban returns `false` → 409.
## Pagination (GET /members/restrictions)
`GET /members/restrictions` returns `{ "items": [...], "nextCursor":
"<token>"|null }`.
- `limit` — page size 1–200 (default 200), enforced as a SQL `LIMIT`.
Returns 400 for values outside range.
- `cursor` — opaque continuation token from a prior page's `nextCursor`.
Omit for the first page.
- Order: `ORDER BY updated_at DESC, pubkey DESC`. The `pubkey`
tie-breaker ensures a deterministic cursor even when multiple rows share
the same `updated_at`.
- Cursor encoding: base64url of `{updated_at_micros}_{pubkey_hex}`.
Treat as opaque.
- `nextCursor` is `null` only when the returned page is smaller than
`limit`. An exactly-full final page emits a non-null cursor; the
subsequent request returns an empty page with `null`.
- The existing `list_community_restrictions` DB method (unbounded) is
kept intact for the embedded dashboard bridge consumer (`GET
/moderation/restricted`), whose bare-array response contract must not
change.
## Tests
All Postgres-gated tests are in the `postgres_tests` module
(`api/admin/mod.rs`):
- `list_restrictions_returns_active_bans_and_timeouts` — seeds a ban +
timeout, GET returns both; `nextCursor` is null when all records fit in
one page
- `list_restrictions_pagination_exhaustive` — seeds 201 rows: 199 with
distinct timestamps (1s apart, newest first) plus a tied pair sharing an
identical older timestamp (`now()-1000s`), pinned to sort positions
200/201 so the default-200 page boundary splits the tie. Asserts: (1)
`limit=201` → 400; (2) a no-`limit` request returns exactly 200 items +
non-null `nextCursor` (falsifiable binding of default=200 and SQL cap —
reverting the route default to the shared helper's 50 fails this
assertion); (3) continuation walk to exhaustion with exactly-once
coverage over all 201 pubkeys, exercising the tie-breaker at the page
boundary.
- `unban_member_returns_204_clears_ban_and_inserts_audit` — 204, ban
cleared, timeout survives (restriction independence), audit row
verified, tenant isolation confirmed
- `untimeout_member_returns_204_clears_timeout_and_inserts_audit` — 204,
timeout cleared, ban survives (restriction independence),
second-community timeout survives (community_id predicate), audit row
verified
- `unban_member_returns_409_for_expired_ban` — expired ban → 409, no
audit row
- `unban_with_audit_rolls_back_lift_when_audit_insert_fails` (buzz-db) —
forced CHECK constraint failure proves Err + ban still active (rollback
proof)
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
|
||
|
Will Pfleger
|
ce41be9c4b |
fix(relay): fire kick live side effects at convergence; persist target; fence re-add race with held lock (#7298)
## What
Ensures admin kick live side effects (subscription eviction + workflow
disable) fire correctly on every path — fresh HTTP, crash recovery, and
upgrade recovery from pre-migration rows — without a race window where a
concurrent legitimate re-add can be revoked.
### Finding 1 — Lock held through effects
Replace `verify_member_still_removed` (which acquired + released the
advisory lock before returning) with `membership_removal_fence`, a RAII
guard that keeps the `pg_advisory_xact_lock` alive until the caller
drops it.
Update `apply_kick_live_side_effects` to use the fence on every path
(fresh and recovery), holding it through eviction and workflow-disable.
Remove the `is_recovery` parameter entirely.
**The bug:** the old code released the lock before returning a bool; a
concurrent `add_member` could commit between the check and the effects,
silently evicting a re-added member.
**New guarantee:** any `add_member` either serializes before the fence
(fence observes `removed_at IS NULL`, skips effects) or waits until
after effects complete.
### Finding 2 — Persisted-context path scoped to kick only
Gate the persisted-context branch in `admin_action_worker.rs` on `action
== "kick"`. Non-kick actions (ban, timeout, delete) always re-derive
from the report.
**The bug:** any action with `enforcement_target_pubkey` set took the
persisted-context branch, which hard-coded `target_event_id = None`. A
stranded delete either failed pre-marker recovery ("delete requires
target_event_id") or finalized post-marker without the tombstone outbox
row.
### Finding 3 — Legacy kick row upgrade recovery (Carl's
CHANGES_REQUESTED)
Migration 0047 (renumbered from 0045 to avoid collision with two main
merges: 0045_retain_push_revocation_tombstones and
0046_storage_accounting_snapshots) adds
`enforcement_target_pubkey`/`enforcement_channel_id` without
backfilling.
**The bug:** the convergence gate read only the persisted columns and
rejected NULL/NULL rows as corrupt, causing every recovery retry to fail
forever. A cancellation cannot release a post-mutation action, so these
rows remain permanently stuck.
**Fix:** when persisted columns are absent, fall back to the function
parameters (which the recovery worker re-derives from the report row).
Error only when both sources are absent (genuinely unresolvable). This
preserves new-writer safety while allowing pre-migration rows to
converge.
An old writer in a rolling deployment can also create NULL rows after
the DDL applies; the same fallback handles these.
### Finding 4 — Test moved to PG lane
`kick_live_side_effects_clears_membership_cache_and_evicts_subscription`
was in the infra-free unit lane but called `membership_removal_fence`,
which requires a real Postgres connection. Moved to the PG fixture lane
(`#[ignore = "requires Postgres"]`) with a real member row so the fence
query sees the expected `removed_at` state.
## Tests
- **`crash_recovery_redrive_fires_kick_live_side_effects`** (existing):
re-drive via real recovery worker path, asserts all three side effects.
-
**`crash_recovery_after_readd_preserves_membership_subscriptions_and_workflows`**
(updated): uses real `add_member` (not direct SQL) — re-add completes
before fence is acquired.
- **`membership_removal_fence_blocks_concurrent_add_member`**
(existing): holds the fence explicitly, spawns `add_member`, asserts it
blocks until the fence is dropped.
- **`stranded_delete_pre_marker_recovers_via_worker`** (existing):
stranded delete (crash before mutation) converges via `recover_one` to
succeeded.
- **`legacy_kick_row_pre_marker_recovers_via_worker`** (new): old-writer
shaped row (NULL columns, no step_marker) converges through the full
state machine via recovery worker. Falsifiable: reverting the
convergence-gate fallback leaves the action stuck in enforcing.
- **`legacy_kick_row_post_marker_recovers_via_worker`** (new): same but
with step_marker set (post-marker crash). Falsifiable: same revert.
-
**`kick_live_side_effects_clears_membership_cache_and_evicts_subscription`**
(moved to PG lane): verifies cache eviction and subscription removal
against a real member row.
## CI
`Rust / Rust Lint` and `Windows Rust` failures in the prior run were
main-inherited (same lanes fail on main at the prior base). The branch
is now at main's current head.
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
|
||
|
Wes
|
6a86531ac9 |
feat(relay): add atomic complete read-state snapshots (#7572)
Opened by Brain on behalf of Wes (GitHub: wesbillman). ## Summary Add an opt-in, host-scoped completeness primitive for NIP-RS read-state clients. Ordinary query arrays and WebSocket behavior remain unchanged. - Advertise a version-1 `read_state_snapshot` capability in NIP-11 only when the request resolves a community. - Accept exactly one self-author kind-30078 snapshot filter on authenticated `POST /query`, after the existing admission, replay and membership gates. - Read all retained, nondeleted self-author kind-30078 coordinates from the writer at one SQL-statement MVCC cut—including unrelated application coordinates and events without a read-state tag. - Return a complete versioned envelope bound to community, signer and a content-identity hash. Reject malformed contracts, reconstruction/signature/storage failures and whole-request overflow; never return a truncated complete response. - Bound snapshots at 4,096 events and 8 MiB of compact encoded event-array bytes, advertised as `max_event_array_bytes`; envelope metadata is additional. Retain the separate stored-payload preflight budget. ## Safety and scope This is point-in-time enumeration, **not** a live freshness guarantee, cursor, CAS token, cross-relay transaction, or unread total. NIP-RS now explicitly requires fresh complete enumeration from every write relay before each override action, canonical publication, carry-forward or deletion. It does not enable synchronized manual-unread overrides in any client. No schema migration or deployment is included. ## Validation Executed against the source committed as `1fd5a5e0c0d893c1259386885517c0c44ae7630f`: - Four real-router/PostgreSQL snapshot cases: beyond-page-cap enumeration and overflow; corruption/storage failure; production payload-bound NIP-98, Redis replay guard and membership enforcement; host discovery/scope/strict request shape. - Two database cases: replacement during preflight preserves a single statement cut; encoded-byte overflow and writer-not-replica routing. The cluster-wide probe uses the required `cluster_global_` serialization prefix. - Production-call-site mutations killed for dispatch, strict filter validation, discovery tenant, snapshot tenant, encoded-byte guard, membership enforcement and replay enforcement; isolated mutation tree restored. - Both affected crates pass all-target Clippy; formatting, differential file-size and repository PostgreSQL test-discovery checks pass. - Normal pre-push gates pass without bypass: branch skew, push-head scope, file size, Rust unit lanes and desktop Tauri checks. - Independent read-only review by Pinky found no remaining relay blocker at this commit. The execution/mutation results above were run by Brain, not duplicated by the reviewer. Hosted CI and maintainer review remain separate gates. No production traffic or deployment was used for these fixture tests. Originating Buzz channel: `a394ecdb-9c67-4c4e-a3c6-4a95f69e8cc8` (unread-but-good), thread `2413d0ba7fa5b2ac5c777df6b3754e2099b042ec8fb96ca64302a0919a0550b3`. ## Review repair (2026-09-21) Carl updated this existing branch to `2f9c9b11266c24947412cf28cefae4e96e7167b7`, merging main `5079c770fe30bb3d8204822ce6c2431eacac6d4b` without rewriting published history. - Resolved the DB export conflict, preserving both snapshot and storage-accounting exports. - Addressed the blocking review using its accepted alternative: rename `max_bytes` to `max_event_array_bytes` and explicitly document that success-envelope metadata is outside the array budget. No auth, query, runtime-limit, or client-behavior redesign. - Added a PostgreSQL/router regression proving discovery, exact 8,388,608-byte array success with a larger envelope, and one-byte-over rejection (413, no partial complete response). Exact-head validation: all seven snapshot PostgreSQL/router tests passed in an isolated local database; both affected crates passed all-target Clippy; mandatory pre-commit/pre-push gates passed without bypass (Rust unit lanes, Tauri default/mesh Clippy and tests, differential file size, branch-skew/ref checks). Reverting only the descriptor rename made the new regression fail before the final commit. Independent bounded source-delta review by Mordecai found no blockers; execution was Carl's. A broader `cargo test -p buzz-db -p buzz-relay --all-targets` run on the merged candidate passed 131 DB tests and 1,057 relay tests, then encountered the same unrelated `mesh_demo::demo_join_forwarded_arm_round_trips_echo` 504 recorded in the prior review. Full repository `just ci` / `just test` were not rerun locally; hosted CI remains a separate gate. No deployment or production traffic was exercised. --------- Signed-off-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz> Signed-off-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> |
||
|
Tom Brow
|
a61239f0d8 |
fix(mobile): avoid opening empty threads on message tap (#7756)
Tapping a mobile channel message with no replies currently opens an empty thread. Only enable message-row thread navigation when the existing thread summary reports replies, including replies known to the relay but not loaded locally. Long-press → Reply still starts a new thread. Added widget tests for zero replies, loaded replies, relay-only replies, and starting a thread through Reply. The zero-reply test failed before the fix and passes afterward. Validation: all 227 channel-page tests, the full mobile pre-push checks, and all PR CI checks passed. The full repository gate failed in unchanged Rust ACP timing tests (idle reset, keepalive, and steering deadlines); this PR changes only mobile Dart files. No duplicate open PR or issue found. --------- Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Wes
|
5079c770fe |
fix(workflows): make deletion persistent and retryable (#7735)
## Summary Fix workflow deletion leaving its definition queryable, and make public deletion acceptance agree atomically with executable/definition state. - Delete the executable workflow and soft-delete its definition under the existing coordinate lock, in the same transaction as kind-5 acceptance and mention indexing. Failed deletion leaves no public accepted event; identical retries can succeed. Legacy already-stored failed requests still repair. - Choose one committed dispatch/audit owner for concurrent identical requests. Completed duplicates remain silent. Preserve owner/community isolation, newer revisions, orphan cleanup, and intentional backdated recreation. - Align Desktop confirmation and architecture documentation with recreation semantics. Add deterministic PostgreSQL, live WebSocket, and Desktop regressions. No schema migration or new retry/delivery subsystem. ## Current validation Head: `40d86ddab0853ffb292316653b22c3fd4f983f0c`. - **Exact clean head:** live relay workflow-deletion E2E **4/4 passed**. All applicable normal pre-push hooks passed: Rust unit suites, full Desktop unit suite, Desktop lint/typecheck, Tauri clippy/tests, and file-size/branch checks. No hook bypass. - **Pre-commit implementation tree, then committed at this head:** full discovered DB/relay PostgreSQL lane **351/351 passed**; built Desktop deletion Playwright smoke **3/3 passed** (success, failure preserving editor/draft, corrected confirmation copy). - **Deterministic mutation evidence:** unchanged previous relay `659b79282` fails both new regressions: concurrent identical requests produce two normal/dispatch-owner responses, and rejected deletion leaves one public event. Both controls pass. Fixed relay passes all four. - **Independent source review:** transaction/authorization/lock-order and regression/cleanup lanes clear. Reviewers did not run tests independently. ## Honest limits / merge gates The broad pre-commit DB/relay/test-client selection passed **1,215/1,216**; the untouched mesh-demo echo test returned HTTP 504 instead of 200. It remains outside this repair and is not counted green. Initial hook attempts also exposed an untouched Desktop inventory ordering assertion and unidentified ACP/agent test failures; standalone package reruns and the final complete hooks passed without changing those packages. Causes are not established. The previous Desktop CI shard failed before tests on a Hermit `just` download (HTTP 500). New-head CI/security review and reviewer disposition remain required. No deployment, fresh interactive Desktop session, full local `just ci`, or multi-replica runtime validation claimed. Postgres/Redis validation used task-owned local containers, fresh identities, and disposable fixtures. ## Compatibility No automatic cleanup of existing workflows: repeating deletion repairs definitions left by the old path. Canonical UUID coordinates and legacy name fallback (`LIMIT 1` for duplicate names) are unchanged. Backdated recreation remains intentional per [Tyler’s decision](https://github.com/block/buzz/pull/7735#issuecomment-5729881295), not an irreversible tombstone. Mention-index failure now rolls back the complete deletion rather than best-effort accepting partial work. Detailed reviewer disposition and evidence: https://github.com/block/buzz/pull/7735#issuecomment-5737367825 --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: Mordecai <f314b6033b6e13df94c8e9e1187d3666b6b9ce54e66ee986e31cdaa909e8dcbc@buzz.block.builderlab.xyz> |
||
|
Wes
|
5511b56fcf |
fix(relay): exclude ephemeral activity from message quota (#7736)
## Summary Stop charging ephemeral WebSocket activity against the per-minute `Messages` quota. Presence, typing, and observer traffic should not consume the allowance intended for persistable event attempts. - Use the existing `buzz_core::kind::is_ephemeral` predicate (kinds `20000..=29999`) at the second admission gate. All authenticated EVENT/REQ/COUNT traffic still consumes the shared `WsEvents` budget first. Non-ephemeral EVENT attempts remain charged, including attempts later rejected by authorization or validation. - Add bounded `bucket` labels (`messages`, `ws_operations`, `api_calls`) to `buzz_admission_rejections_total`. HTTP changes are metric labeling only. Wire rejection text remains byte-for-byte unchanged, preserving existing client parsers. - Add real-Redis admission regressions and select them in the existing relay integration CI job. No quota increases, identity/keying changes, client changes, new limiter, or deployment. Downstream authorization and fail-closed Redis behavior are unchanged. ### Related issue Duplicate search found overlapping work: open #4912 and closed #4902. They propose the same ephemeral-accounting correction alongside broader wire/client/config changes. The relay fix is a deliberately narrower four-file implementation on the current admission module, with metrics-only diagnostics and real-Redis regression coverage. A separate test-only CI repair is documented below. It does not close or claim to supersede those PRs. Related symptom: #7411. This PR does **not** resolve subscription bursts exhausting the shared WS budget; that budget is intentionally preserved. It also does not establish or fix the cause of HTTP quota exhaustion. ### Testing Validation ran against base `779af8886caae1317b4de962082429867ab61503` plus the exact patch now committed as `70ea0528594e25e6dcb40cd74620bd5ef0177592` (binary diff SHA-256 `5064e1cbb87ecb9eff10336bae14b912cac442e1edc247aa4f093de30e4f9717`). Normal commit/push hooks subsequently passed on the committed head, without bypass. Passed: - Independent review and rerun: 11 rejection tests (including real Redis) plus 4 admission tests. Covers ephemeral range boundaries, human/agent budgets, persistable events, ephemeral admission after message exhaustion, mixed EVENT/REQ/COUNT flood exhaustion, Redis outage, exact wire payloads, and bucket metrics. Three mutations were caught: reintroducing ephemeral billing, exempting stored events, and bypassing WS rejection. - `just test-unit`: all 17 runner groups; `just test-integration`: all 3 runner groups. Auth integration reported no tests; ignored database integration tests are not claimed as executed. - Relay all-target Clippy, formatting, workflow actionlint, pre-commit/signoff hooks, and all applicable pre-push lanes (`push-head-scope`, `branch-skew`, `file-size-check`, `rust-tests`, `desktop-tauri-checks`). - Release relay/CLI/admin build and isolated release-binary workflow: channel creation, signed root/reply sends, exact-ID readback, and thread retrieval. Debug and release WS smoke admitted three presence events with a Messages limit of two, admitted two persistable events, rejected the third, still admitted presence afterward, and rejected author mismatch. Debug Prometheus confirmed `bucket="messages"`. **Outstanding validation gap:** the additional `cargo test -p buzz-relay --no-fail-fast` run was not green: library 1,055 passed / 2 failed / 100 ignored; binary 13 passed / 1 ignored; boot integration 9 passed. Failures were `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` (504 instead of 200) and `telemetry::tests::trace_context_lookup_does_not_enable_callsites` (tracing assertion). Neither failure was diagnosed or repaired in this scope, and this PR does not claim a fully green package suite. `just ci` was not run locally; hosted CI remains to validate the PR. All runtime tests used isolated local services. Owned services have been stopped; nothing was deployed and no production data was touched. ### CI cancellation-test repair At the original relay head, hosted Rust unit tests failed on the pre-existing cancellation harness race tracked in #4945. With owner authorization, this PR now adapts the one-file repair from #5003 (original commit `cc8dc6124957acfd67fe2336396b58e0ff43ec60`), preserving author credit. #5003's old hosted workflows failed without starting jobs; their status is not used as validation evidence. Commit `55fe5f16618a51d03ad48d6f2f97b2f8b463607a` holds the fake provider response until a successful cancellation acknowledgement, then replays all pre-acknowledgement usage/prompt frames in order. It retains the cancellation and usage-before-response assertions and explicitly requires prompt completion. No production code changes in this follow-up. Validation on parent `70ea052859` plus this test patch, before the pre-commit-only assertion wrapping: full `cargo test -p buzz-agent --no-fail-fast` passed (715 tests, one ignored); 100 targeted repetitions passed with four concurrent workers; all-target buzz-agent Clippy passed. The unmodified parent also passed 50 local repetitions, so this is not claimed as a local reproduction of the intermittent CI failure. The synchronization defect was confirmed from the source, original CI log, and #4945. Hosted CI must validate the updated head. Independent source review cleared exact commit `55fe5f16618a51d03ad48d6f2f97b2f8b463607a`. Normal pre-commit/signoff and all applicable pre-push hooks passed without bypass. The original relay-package mesh-demo and telemetry failures remain disclosed above and were not modified by this test-only repair. --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: kiranmagic7 <262980978+kiranmagic7@users.noreply.github.com> |
||
|
ravarora2
|
4c2086c084 |
feat(db): expose connection setup metrics (#7286)
## Why
Current pool metrics show checkout outcomes and pool state after the
fact. They do not show when a checkout began or which writer-connection
setup step failed. During startup and pool growth, operators need to
distinguish pool saturation from a slow or unsafe connection setup.
## What
- Count every instrumented database checkout when it starts, with a
fixed operation label.
- Measure writer-pool creation and every physical writer connection
across `physical_connect`, `created_at_floor`, `session_timeouts`,
`isolation`, and `ready`.
- Publish fixed-cardinality start, terminal-outcome, duration, and
waiter metrics without database URLs, SQL, raw errors, connection
ordinals, or per-connection lifecycle logs.
- Document how to identify the active setup bottleneck during a rollout
or incident.
## How
The existing typed checkout wrapper records starts and current waiters.
The huddle-history path uses that wrapper instead of a raw pool
checkout.
The production SQLx `after_connect` hook records start, terminal
outcome, and duration for each sequential safety step. A drop guard
records cancellation once if setup exits before a terminal result. Fixed
enums keep every label bounded.
For a phase, `started_total - sum(attempts_total)` is the number of
in-progress attempts on that pod. Because setup is sequential, a later
phase starting also proves the earlier phases succeeded.
The database path emits metrics only. This PR intentionally does not add
per-connection lifecycle receipts or direct `stderr` writes.
## Risk
Medium. This changes the production writer-pool `after_connect` hook and
adds one counter update to instrumented checkout paths. It does not
change the database safety statements or their failure behavior. The
telemetry has fixed labels and no per-connection identifiers.
## Testing
At exact head `ffb5fcb114a9c986ffd0cb2cfc9d413a32d5b1aa`, a release
relay was started against isolated local PostgreSQL, Redis, and MinIO
services. The main health endpoint returned `ok`, readiness returned
`{"status":"ready"}`, the connection-step metrics were exported, and no
database lifecycle receipts were emitted. A release CLI then created a
channel, sent a message, and read the same event back successfully.
The earlier staging deployment used pre-rebase head
`f58e9480a4f068db0c591f604fd6800fdd4bfc45`. The deployed
multi-architecture image came from [GitHub Actions run
33780888255](https://github.com/block/buzz/actions/runs/33780888255),
manifest
`sha256:b9351fa644e08376cbe1999f9bee311d33d1799a68eadef7929c4f862a832fec`.
The [staging
deployment](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/317)
brought both pods in ReplicaSet `buzz-6c8758bd7d` to Ready with zero
restarts, and the connection metrics produced data in the [rollout
dashboard](https://app.datadoghq.com/dashboard/tm7-qxr-wt2/buzz-startup--rollout-safety).
That staging image predates the cleanup that removed per-connection
lifecycle logs; the metric schema and database safety statements are
unchanged.
## Verification
- `cargo test -p buzz-db -- --test-threads=1`: 128 tests passed across
the package and integration target; 255 opt-in tests remained ignored.
- Four opt-in production-path PostgreSQL regressions passed: initial
minimum connections, post-startup pool growth, isolation failure, and
session-timeout setup failure.
- All 35 relay media tests passed against the isolated PostgreSQL
database.
- Rust formatting, workspace clippy, desktop/Tauri clippy, web checks,
mobile analysis, security checks, and file-size policy checks passed.
- The repository-wide unit stage also exposed three unrelated existing
`buzz-acp` failures: one timing-sensitive keepalive test and two
environment-default tests.
## Bigger picture
This is the database-metrics part of the startup and rollout
observability work. The early-startup lifecycle foundation merged in
#7258, so the rebase removed that duplicate commit from this PR.
Originating discussion:
buzz://message?channel=6ac85131-70cd-4bda-a031-38d34114934e&id=fa2bed181c092697210a60bb6eedc55a665d5c1c6cabc1413a04686647011f71
Generated with Codex
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
|
||
|
ravarora2
|
6b1e45d2f6 |
Isolate S3 storage metrics from the relay (#7543)
## Why **This PR moves the S3 storage scan out of the serving relay and into a separate batch job.** The job saves a complete result in PostgreSQL. The relay reads that result, exports the existing storage totals, and reports the snapshot's freshness and database-read health. ```text Today: relay -> scan S3 -> calculate totals -> Datadog metrics After rollout: CronJob -> scan S3 -> PostgreSQL snapshot -> relay -> Datadog metrics ``` The code in this PR is the new pipeline. It adds the worker, database snapshot, relay read mode, and Helm CronJob template. It does **not** enable the pipeline in production. The chart keeps the CronJob disabled, and the relay keeps its current inline mode by default. [squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244) will set the production limits, enable the job, and switch the relay to the saved snapshot after this PR merges. The relay currently lists the media bucket once an hour and calculates storage totals inside the serving process. The production bucket now exceeds the relay's one million object cap, so each sweep fails and the last good per-community values remain stale. The Datadog dashboard can no longer show current storage use by community. Raising that cap inside the serving relay would move a larger, memory-heavy scan into the process that owns live WebSocket connections. The scan retains state for unique blob hashes and community bindings, so its memory use grows with the bucket. A large scan could raise relay memory use, trigger restarts, and drop user connections. The goal is to restore current metrics without putting a production-scale bucket scan in the relay's failure domain. No media objects move or change. Only the process that calculates the totals and the handoff to the relay change. ## What - Add `buzz-admin storage-snapshot`, a run-once worker that scans S3 and saves one complete accounting snapshot in PostgreSQL. - Add `external` relay mode to read the saved snapshot. Keep the current `inline` mode as the default, and add `off` as a fail-closed option. - Add an opt-in Helm CronJob for the worker. The 10 million object cap exists only on this Job. - Keep the current fleet and per-community gauges, and add snapshot age, generation duration, configured cap, cap utilization, and database-load health gauges. - Add an additive migration for the latest complete snapshot and its run metadata. ## Overall plan 1. Merge this PR to add the worker, database handoff, relay modes, and disabled-by-default chart support. This step does not change production behavior. 2. Publish the relay image and Helm chart `0.1.9` from `main`. 3. Update [squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244) to pin those final artifacts and rerender its manifests. 4. Use the infrastructure PR to deploy the CronJob with a 10 million object cap, a one-hour deadline, and a 20 GiB memory limit. 5. Start one manual Job during a staffed window and verify its snapshot, duration, memory use, relay stability, and Datadog output. 6. Keep the daily schedule and `external` relay mode only after that gate passes, then add alerts for stale snapshots, snapshot-load failure, failed jobs, cap pressure, duration, and worker memory. The code and infrastructure remain separate so this PR can ship the mechanism without silently enabling a production-scale scan. The infrastructure PR owns the production settings and rollout gate. ## How ### Complete snapshot handoff The worker takes a deployment-wide PostgreSQL advisory lock before it lists S3. This prevents scheduled and manual workers from overlapping, even if Kubernetes starts both. The final database replacement runs through that same lock-owning session. If the session dies during the scan, publication fails instead of letting the stale worker overwrite its successor. A worker that no longer owns the lock cannot publish through another pooled connection. The worker reads S3 in pages of 1,000 objects and folds each page into physical totals, logical per-community usage, and anomaly counts. It stops before folding a page that would exceed `--max-objects` and emits structured progress every 100,000 objects. The worker replaces the singleton database row only after the full scan succeeds. A listing error, malformed page, cap breach, serialization error, lost lock session, or database error cannot publish a partial snapshot. The last completed snapshot stays available. Each stored result includes its completion time, duration, cap, and image or source revision. PostgreSQL is the handoff between the batch worker and the relay. The worker writes one complete deployment-wide snapshot, and the relay reads that snapshot on its normal leader metrics tick. No new network service or worker endpoint is required. ### Relay behavior `BUZZ_STORAGE_METRICS=inline` keeps the existing in-process sweep and its one million object default. `external` loads the newest completed worker snapshot on each leader metrics tick. `off` emits no storage-family metrics. An unknown value fails closed to `off`. External mode republishes the existing fleet, per-community, and anomaly metric names, so those current queries remain valid. Snapshot handoff health uses a separate family: `buzz_storage_snapshot_age_seconds`, `buzz_storage_snapshot_duration_seconds`, `buzz_storage_snapshot_max_objects`, `buzz_storage_snapshot_cap_utilization`, and `buzz_storage_snapshot_load_ok`. The `buzz_storage_sweep_*` gauges remain inline-only because they describe a scan attempted by the relay process. External mode does not synthesize `sweep_ok=1` from the last successful database row: an old row proves only that an earlier worker completed. Kubernetes CronJob/Job status is authoritative for the latest worker result, including non-zero exit, deadline expiry, and OOM termination. The relay keeps the last complete snapshot available when a worker run fails. Snapshot age shows that no replacement arrived on schedule, while `buzz_storage_snapshot_load_ok` distinguishes staleness from a missing row, database read failure, or invalid payload. If a community disappears, changes host, or leaves the configured emission scope, the relay zeroes the old labeled series instead of leaving stale values behind. Per-community bytes are logical referenced media usage. If two communities reference the same blob, each community receives the blob's bytes. Fleet physical totals still count each S3 object once. This PR preserves those existing metric semantics. ### Deployment boundary The CronJob is disabled by default. When enabled, it uses `concurrencyPolicy: Forbid`, no retry, a configurable active deadline, bounded history, explicit CPU and memory resources, a read-only root filesystem, and no Istio sidecar by default. The worker receives only `DATABASE_URL`, S3 settings, optional S3 credential references, and the code revision. It does not receive the relay private key, Git hook HMAC secret, Redis URL, or the full relay secret as environment variables. Operators may reuse the Buzz service account or select a dedicated one. ### Non-goals - This PR does not enable the worker or `external` mode in production. - This PR does not change media uploads, object ownership, retention, or deletion. - This PR does not change the names or meaning of the existing fleet, per-community, or anomaly storage gauges. - This PR does not add a lease service or allow multiple workers to publish concurrently. ## Risk The application and chart defaults do not enable the new worker or raise the serving relay's cap. The database change is an additive deployment-global singleton table. The relay's default remains `inline`, so merging this PR alone does not change the active metrics path. The main rollout risk is worker memory use because the fold retains state for distinct blob hashes and community bindings. The first production-sized run must confirm peak memory and duration before the schedule is trusted. The 10 million cap and one-hour deadline bound the first deployment, but they do not replace measurement. External mode also needs stale-snapshot, snapshot-load, and job-failure monitors because the relay intentionally keeps publishing the last complete result after a failed job or read error. This behavior avoids gaps and partial data, while snapshot age and load health tell operators when the result or PostgreSQL handoff needs attention. Job status remains necessary for immediate failures and termination reasons that a killed process cannot record itself. ## Testing No production S3 scan was run from this PR. Automated regressions verify that external mode keeps stale last-good totals visible without emitting inline attempt-health gauges, reports database-load failure while retaining the cache, and leaves inline telemetry unchanged. Mutation checks also prove those tests fail if external mode is routed through inline emission or a failed load is reported as healthy. After deployment, create one manual Job from the CronJob during a staffed window. Require a completed Job, exactly one new snapshot, fewer objects than the configured cap, acceptable duration and peak memory, no relay restart regression, restored Datadog points, and no OpenMetrics sample truncation. ## Next steps 1. Complete review and merge this PR. 2. Publish chart `0.1.9` and the final main-branch relay image. 3. Replace the provisional pins in [squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244), rerender, and require clean CI. 4. Run the staffed manual Job gate and record object count, peak memory, duration, snapshot revision, and relay stability. 5. Confirm the relay reads the new snapshot in `external` mode and the existing `block.buzz_relay.*` storage series update again. 6. Enable the daily 03:17 UTC schedule and add dashboard panels and monitors for snapshot age, snapshot-load health, Job failures, cap pressure, duration, and worker memory. If production must roll back to an older relay image, set `BUZZ_STORAGE_METRICS=off` first. Older images do not understand `external` and would resume the failing inline scan. ## Bigger picture This is the application half of the storage-metrics repair. The paired infrastructure PR owns the production resource limits, schedule, artifact pins, secret boundary, and monitored rollout. Keeping those changes separate lets this PR remain safe by default while still providing a complete path to current per-community S3 totals. Related issue: N/A. No matching issue or pull request exists in `block/buzz`. Generated with Codex --------- Signed-off-by: Ravneet Arora <rarora@squareup.com> Co-authored-by: Codex <noreply@openai.com> |
||
|
Tom Brow
|
b36600fc44 |
feat(mobile): show build number with version in settings (#7697)
Mobile Settings now identifies the installed build as e.g. `v0.16.0 (432)` rather than just `v0.16.0`. If the platform supplies no build number, the footer keeps the existing version-only display. | Before | After | | --- | --- | | Marketing version only | Version and native build number | |  |  | --------- Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
ravarora2
|
891208914e |
Add authenticated WebSocket recovery telemetry (#7546)
## Summary
Argo rollout recovery needs to distinguish an open WebSocket from a
connection that completed NIP-42 authentication and is usable. The relay
currently exposes total WebSocket connections, but it cannot answer
whether authenticated clients recovered after a pod drain or why
authentication failed.
This change adds a bounded recovery contract:
- `buzz_auth_attempts_total{method="nip42"}`
- `buzz_auth_outcomes_total{method="nip42",outcome}`
- `buzz_auth_duration_seconds{method="nip42",outcome}`
- `buzz_ws_authenticated_connections_active`
- `buzz_auth_post_terminal_frames_total{state}`
`buzz_auth_attempts_total` now has one consistent recovery-oriented
unit: a challenge lifecycle successfully queued to the connection
writer. This intentionally replaces the historical unit of AUTH frames
that reached the pending handler. AUTH frames received after a lifecycle
is already authenticated or failed are protocol noise; they use the
separate bounded post-terminal counter and cannot inflate rollout-gating
attempts or outcomes.
Each issued challenge reaches exactly one terminal outcome.
Authentication transitions and active-gauge changes share one short
synchronous state lock, cancellation can claim a pending database-backed
AUTH before a late handler result, and a drop guard reconciles aborted
or panicked connection futures. Accounting is terminalized before writer
joins, while terminal socket delivery has a one-second best-effort bound
so a stalled sink cannot retain the authenticated gauge or connection
permit indefinitely.
Dependency failures are distinct from policy verdicts: allowlist and
relay-membership lookup errors now emit `allowlist_check_error` and
`relay_membership_check_error`, never `allowlist_denied` or
`not_relay_member`. The fail-closed allow/deny policy is unchanged. All
rollout-facing labels come from fixed enums; identities, challenges,
URLs, and raw error strings are never exported.
All fixed counter and histogram label sets, including zero histogram
buckets, counts, and sums, are present on the first real exporter
scrape. This is the first implementation slice of the
rollout-reliability plan. A later PR will consume these metrics from
opt-in Argo Rollout analysis; merging this PR alone does not change
deployment behavior.
### Related issue
None found.
### Testing
Validated at final head `f9992cac835284c25ee0f99e17ddba0b932d7c69`,
which contains current `origin/main`:
- `cargo fmt --all -- --check`
- `cargo clippy -p buzz-relay --all-targets -- -D warnings`
- compiled every `buzz-relay` test target
- release builds for `buzz-relay`, `buzz`, `buzz-admin`, and
`buzz-test-cli`
- 16/16 connection lifecycle/writer/production-dispatch tests
- 7/7 AUTH handler tests
- bounded metric-contract exporter test
- 13/13 relay main-target tests (1 PostgreSQL-only test ignored)
- 9/9 real boot-lifecycle tests, including a production install and
first Prometheus scrape
Mutation checks proved that the production regressions fail when any of
these are removed or reintroduced:
- challenge attempt start in `handle_active_connection`
- authentication series initialization in the real metric installer
- separation of post-terminal AUTH floods from authoritative attempts
The exact-head release binaries were also exercised against the isolated
review database:
1. Started the release relay and verified health/readiness.
2. Created a channel with the regular CLI.
3. Held one authenticated WebSocket subscription open and authenticated
a second WebSocket publisher.
4. Published a message and read it back through the regular CLI.
5. Observed `attempt_delta=2`, `success_delta=2`,
`active_while_subscribed=1`, and `active_after_disconnect=0`.
6. Sent SIGTERM and waited for graceful relay shutdown.
### Local caveat
The full relay library target consistently reached 1,049 passes with 89
intentionally ignored tests, but the untouched
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`
intermittently returned HTTP 504 instead of 200. One isolated rerun
passed; later isolated and full reruns reproduced the 504. No mesh files
are changed by this PR, and all changed-path, main-target,
boot-lifecycle, lint, build, mutation, and live-local checks above
passed at the final head.
Generated with Codex
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
|
||
|
Luke Tornquist
|
213092074c |
Instrument database pool roles (#7356)
## What
Name the relay's four Postgres pool roles—writer, reader, audit, and
search—and export one fixed-cardinality utilization metric contract for
all of them. Existing writer and reader gauges remain unchanged.
## Why
Audit and search already use separate pools, but they are missing from
pool pressure telemetry. Operators cannot see all process-level
connection demand before setting a deployment-wide connection budget.
## How
A closed `DbPoolRole` vocabulary now labels
`buzz_db_pool_connections{pool_role,state}` and
`buzz_db_pool_configured{pool_role}`. The relay retains cheap pool
clones for statistics only; ownership, capacities, timeouts, query
routing, and failure behavior do not change. The docs also replace the
stale async search-index description with the generated-column behavior
used today.
This simplifies four ad hoc observability states into one bounded model
without adding a pool manager or changing service boundaries.
## Risk
Low to moderate. This changes relay metrics and pool construction
plumbing, but not SQL execution or routing. Legacy metric names remain
available for current dashboards.
## Testing
- Push gate at `3f5468a0463bdfc5879bcdd3878fa06b42ca84c2`: all 14 Rust
test lanes passed; desktop Tauri checks passed.
- Blox: `cargo test --workspace --all-targets` with repository-declared
Postgres services.
- Blox: `cargo clippy --workspace --all-targets -- -D warnings`.
- Blox: `cargo fmt --all -- --check`.
## Bigger picture
Aggregate deployment budget enforcement remains deferred. This PR
exposes the per-role facts that a later deployment-policy change can
consume.
Generated with Claude Code
---------
Signed-off-by: tornquist <tornquist@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Jarrod Sibbison
|
c507a4d488 |
fix(db): allow naturally expired push lease replacement (#7653)
## Summary Allow a naturally expired push lease to be replaced by a newer lease for the same author and installation. ## Failure mode Expiry cleanup marked the old lease inactive but left active-only fields populated. The push_leases_check constraint requires those fields to be NULL when a lease is inactive. When the client later registered a replacement lease, the cleanup ran before the replacement and violated the constraint with SQLSTATE 23514. The transaction rolled back, so the replacement failed instead of taking over the expired lease. ## Fix Clear all active-only fields as part of natural-expiry cleanup, preserving the existing constraint and replacement flow. No schema change is required. ## Testing - Passed the focused Postgres regression test: cargo test -p buzz-db replacing_naturally_expired_lease_replaces_old_state -- --ignored - Passed scripts/test-postgres-test-discovery.sh - Ran just ci: formatting, lint, security, file-size, desktop, web, mobile, and 12 unit-test lanes passed. The overall run was red only because existing timing-sensitive buzz-acp tests failed. - The pre-push test-unit hook reproduced the same unrelated buzz-acp failures, so the already-run verification was documented and the branch was published with that hook bypassed. ## Related issue No matching open issue or pull request was found. ## Scope One file changed: crates/buzz-db/src/store/push.rs. The test uses the production accept_lease_event path and identifies the replacement by its cryptographic source event ID and generation. Signed-off-by: Jarrod Sibbison <jsibbison@squareup.com> |
||
|
Tom Brow
|
76f7e5ef0c |
feat(push): make gateway origin configurable (#7261)
Make the push gateway origin configurable across the relay, gateway, mobile client, and deployment chart. This is a fresh deployment. The gateway has only run on personal development infrastructure, and push has no existing enabled users. Legacy migration, configured-to-unconfigured mobile transitions, and rolling upgrades from pre-launch gateway binaries are not supported. Populated pre-launch databases are rejected with an informative error before migrations run; their data is untouched. Mobile builds may omit the gateway URL, leaving push unavailable. Once push is enabled, subsequent builds must retain that configuration. ### Validation - All applicable CI checks pass at `5f546cfaf403f0386f146d213adfb4392f4ec29c`, including Rust, mobile/Swift, desktop/integration, security, image builds, and Helm checks. An unrelated agent cancellation test passed on retry. - PostgreSQL CI passed 384 tests, including the production-entry-point regression for refusing populated pre-launch databases before migrations, fresh initialization, and subsequent initialized deployments. - Focused local validation passed all 10 gateway PostgreSQL tests under the CI discovery filter and all 3 unconfigured-mobile tests. Repository-wide PostgreSQL discovery validation and the complete pre-push hooks passed. - Physical-device APNs/App Attest behavior and deployment to a live cluster were not verified in this work. --------- Signed-off-by: Tom Brow <tomb@block.xyz> Co-authored-by: Codex <noreply@openai.com> |
||
|
Salman Mohammed
|
4d08194ea9 |
Deduplicate thread context by ACP session delivery (#7620)
## Summary Buzz can repeat a thread reply in later prompts after the provider already received that event as context. The old fix treated every message signed by the agent key as retained by the current provider session. That could hide an unseen heartbeat reply because independent sessions share the same signing key. This PR now removes only exact event IDs already delivered to the live ACP session. It uses bounded overfetch so removed events do not shrink the new-context window, and it preserves unseen same-key replies from other sessions. The context header reports `truncated=true` whenever session history was omitted. The harness calls `/count` only when the relay query finds a sentinel beyond its fetch window. This deliberately does not parse event IDs from shell output. A reply that the agent just published may appear once in the next context delta; after that event reaches the provider as context, later prompts omit it. --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> |
||
|
Salman Mohammed
|
deda09c18c |
Document incoming Buzz turn structure (#7624)
## Summary Document how agents should read Buzz’s semantic turn envelope. The base prompt now identifies current event content as the request, conversation context as support for follow-ups, and context/event fields as routing and identity metadata. The existing compiled-prompt regression test now protects that contract. The lockfile also updates rustls to 0.23.45 to clear RUSTSEC-2026-0285 from the security gate. |
||
|
Wes
|
4cd82f5132 |
feat(buzz-agent): add opt-in workspace-scoped Databricks reuse (#7606)
Posted by Brain, Wes Billman’s AI assistant, on behalf of `wesbillman`. ## Summary Add an opt-in `databricks::DatabricksConnection` over the existing PKCE/coordinator and v2 catalog machinery. It accepts an explicit HTTPS workspace origin, absolute caller-owned cache root, and browser opener; it does not start an agent or infer credentials/host from the environment. - Validate both OAuth endpoints on the actual discovery response used for grants; native HTTP redirects are disabled and requests have a 30-second timeout. - Keep strict cache/single-flight state in a separate namespace. Connect is explicit/user-initiated; catalog lookup and its one 401 refresh stay headless. Tokens are not exposed by the connection API. - Preserve existing legacy constructors, endpoint/redirect policy, cache layout, intents, and catalog filtering/pagination/partial/default semantics. No internal host default, credential migration, app integration or release configuration changes. ### Intentional shared changes OAuth success/discovery JSON is capped at 1 MiB; error JSON at 16 KiB. Oversized/malformed responses are infrastructure failures, while bounded client-error `invalid_grant` retains its existing classification. OAuth diagnostics omit raw bodies/URLs/opener/callback details. Overflowing token expiry returns an infrastructure failure. These changes affect legacy callers too and have synthetic regression coverage. See [the reuse contract](crates/buzz-agent/DATABRICKS_REUSE.md) for API, compatibility and caller obligations. Total operation deadlines, stale-result fencing, private root ownership, nonlogging opener behavior and credential-removal UX remain the caller’s responsibility. ### Related issue Related discovery work: #6918. Searched existing Databricks PRs and OAuth issues; no duplicate of this strict opt-in API was found among those results. This is a new reuse boundary, not a claim of a live credential leak. ### Testing Published head: `cf33eb2f4f267d5808e068e35127d5504f0bc674`, based on main `78618804ec86a014524ad7d1fb55928e8f5c3edf`. Two implementation/doc commits were transplanted from a release-derived local base to exclude unrelated release files. `git range-diff` reports both patches unchanged; independent source review of the transplant found no blocker. **At the published head:** normal pre-push hooks passed: branch skew, push-head scope, file-size checks, Rust test lane and desktop Tauri Clippy/tests. The fallback Rust lane actually ran `buzz-agent --lib`: 542 passed / 1 ignored, NOT the full package integration suite. Hosted full package/repository CI remains pending. The first push attempt was interrupted by the command runner’s 300-second limit; the second completed normally with all hooks enabled. **At reviewed implementation checkpoint `8cb3b72d4`:** 16 strict, 27 auth, 32 catalog and 63 coordinator/OAuth/CLI tests passed, independently rerun; 18 author-selected mutations and four independent reviewer mutations failed tests with passing restored controls. The subsequent `539319411` delta changed only two doc-comment blocks. These earlier test/mutation results are not silently attributed to the rebased head. Synthetic coverage includes HTTPS grants/catalog, off-origin endpoint and redirect rejection, cancellation/socket/listener cleanup, host/root/legacy namespace isolation, bounded declared/chunked bodies and exact-limit controls, redaction with positive log capture, malformed expiry, legacy endpoint/redirect controls, and real CLI cached-auth aliases. Desktop native checks used fail-closed sidecar stubs: compile/test evidence, **not** a runnable or packaged app. No real provider credentials, browser sign-in, app launch, or agent cutover was performed. Cross-platform execution and real-provider acceptance remain separate gates. No merge or downstream adoption is requested by this PR. --------- Signed-off-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz> Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz> |
||
|
Salman Mohammed
|
6c35e82bd5 |
fix(acp): improve hints in pi agent setup guide (#7594)
## Summary Handle missing Pi adapter setup and update its install guidance. ### Related issue None found. ### Testing No manual testing. Generated with Amp --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> Co-authored-by: Amp <amp@ampcode.com> |
||
|
Salman Mohammed
|
e17cdd9d5c |
refactor(buzz-acp): point agents at buzz --help instead of a command table (#7586)
## Summary The ACP base prompt carried a `Group | Key commands` table for the `buzz` CLI — a second copy of a surface that already documents itself, with nothing tying it to the parser. It had already drifted: 16 of the CLI's 23 groups (missing `emoji`, `gifs`, `notes`, `patches`, `media`, `moderation`, `pack`) and 50 of the 91 subcommands under the groups it did list. The adjacent exit-code line stopped at 4; the CLI has a 5th (5 = write conflict, NIP-33 LWW). #7584 was the prerequisite: `buzz --help` now renders the full command tree — every group, its subcommands, and their descriptions — in one invocation (199 lines, 13 KB), generated from clap's own command definition, so it cannot drift. `buzz -h` still prints the group-level summary. Pointing the prompt at `--help` was not viable before that, because discovering `buzz messages send` cost a second call and discovering `buzz canvas set` exists cost a third. So this replaces the table with a pointer to `buzz --help`, fixes the exit codes, and adds the one CLI ergonomic that `--help` does not make obvious (`--format compact` is a global flag that goes before the subcommand). Everything after the table is kept verbatim — it is the non-discoverable semantics `--help` cannot express: multiline content through stdin, the `BUZZ_AUTH_TAG` requirement for agent drafts, `--channel` on `pr open`, the `link` field, mention and assignment rules. Signed-off-by: Salman Mohammed <smohammed@squareup.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Salman Mohammed
|
44c1cc7df8 |
feat(buzz-cli): render an agent-friendly command tree in --help (#7584)
## Summary
`crates/buzz-acp/src/base_prompt.md` carries a hand-maintained `Group |
Key commands` table of the `buzz` CLI. It is a second copy of a surface
that already documents itself, and it has drifted: it lists 16 of the
CLI's 23 groups (missing `emoji`, `gifs`, `notes`, `patches`, `media`,
`moderation`, `pack`) and omits roughly two-thirds of the subcommands
under the groups it does list. its hard to keep it in sync.
The reason the base prompt duplicates that inventory is that `buzz
--help` was not good enough to lean on. It listed 23 group names with
nothing under them, so finding `buzz messages send` cost a second
`--help` call and learning that `buzz canvas set` exists at all cost a
third. An agent paying per group reasonably prefers a stale table it
already has.
This PR fixes the help output so the prompt can point at it instead.
`buzz --help` now prints the whole tree — every group, its subcommands,
and their descriptions — in one invocation:
**Before**
```
Commands:
messages Send, read, search, and manage messages
channels Create, configure, and manage channels
canvas Get and set channel canvas documents
```
**After**
```
Commands:
messages Send, read, search, and manage messages
send Send a message to a channel
send-diff Send a code diff / patch to a channel
edit Edit a previously sent message
...
repos Announce and discover git repositories (NIP-34)
protect Manage branch and tag protection rules on one of your repositories
list List the repository's protection rules
```
This change is formatting only: it adds no new data and no new commands.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
|
||
|
Wes
|
6146c4fd16 |
feat(git): add default-branch management to relay and CLI (#7562)
Authored by Brain and opened on behalf of Wes (`wesbillman`).
## Summary
Add `buzz repos default-branch get/set` and the relay operation it
needs. Git push changes branch tips but cannot select the server's
symbolic HEAD. This selects an existing branch without deleting
branches, moving refs, or changing packs.
- Reuse immutable Git manifests and captured-ETag CAS; concurrent
pushes/default changes conflict, including stale no-ops.
- Require request-specific NIP-98, body binding, fail-closed replay
protection, host tenancy, current channel membership and
repository-management authority. Push/project roles alone are
insufficient.
- Send the CLI mutation once, without redirects; ambiguous delivery
retains the observed digest and is non-retryable.
- Document the narrow HTTP exception to Nostr-first, delegation scope
and admission-time ACL revocation. Kind:30618 remains derived; the
manifest CAS is authoritative.
- Register the Postgres/Redis/MinIO route and fresh-clone tests in the
existing Backend Integration lane.
## Review fixes — `7457bda50afa6b52e36315572fa96928b2e21db6`
- Reject signed valueless `payload` tags when verifying a body; preserve
the shared verifier's intentional absent-payload compatibility. The real
settings route now proves 401, unchanged manifest pointer, and unchanged
kind:30618 event IDs for malformed and other invalid credentials.
- Require a nonempty returned branch and `head == refs/heads/{branch}`;
SET additionally requires the returned branch to equal the attempted
branch. Malformed/mismatched mutation responses remain non-retryable
`DeliveryUnknown`, retaining the attempted branch and original digest.
Invalid reads fail before any POST.
- Strengthen the later-push/fresh-clone regression to select
`release/v1` while `main` exists.
## Verification
At `7457bda50afa6b52e36315572fa96928b2e21db6`:
- Full pre-push passed (all 14 Rust test groups, serialized Tauri
tests/clippy, file-size and branch-skew gates; 522 seconds). No hooks
disabled.
- All four isolated Postgres/Redis/MinIO settings tests pass, including
real Smart HTTP `ls-remote --symref` and fresh clone on `release/v1`.
- All 15 focused NIP-98 verifier tests and three default-branch CLI
tests pass; malformed credentials and missing-branch responses were
reproduced failing before the fixes.
- Five targeted mutants are killed: restoring the valueless-payload
bypass, weakening canonical HEAD validation, accepting an empty branch,
dropping requested-branch confirmation, and omitting hydrated HEAD for
the selected non-main branch. Mutations ran in a separate worktree and
target directory; original patch bytes were verified restored.
- `cargo clippy -p buzz-auth -p buzz-cli -p buzz-relay --all-targets --
-D warnings`, formatting, diff and file-size checks pass on the fixed
tree.
Earlier validation at `bb195b140`: seven Git authorization database
regressions and 28 existing transport tests passed (three infrastructure
cases in that module not selected); four mutations killed
management/replay/CAS/redirect regressions. Full pre-push passed with
`RUST_TEST_THREADS=1` (all 14 Rust groups, Tauri clippy and 3,173 tests,
size/branch-skew gates). Serial execution avoids an unrelated
process-global Desktop discovery counter flake; no hooks disabled or
Desktop changes.
Carl and Jude's review findings are addressed in the new head; **fresh
reviewer and exact-head security confirmation remain required**. The
earlier source review is not approval of this revision. No claim of
green new-head CI.
## Rollout
Merge and **deploy the relay support first**, then build/use the updated
CLI and explicitly set/verify the desired default. No schema migration.
This PR does not deploy anything or change a live repository.
Originating conversation (channel
`8e2fda80-9142-4df8-8e14-da288bea6471`):
buzz://message?channel=8e2fda80-9142-4df8-8e14-da288bea6471&id=8a7a9cf12737abc1e16605f241300be940a1691ef0992b35197afe26c563108d
---------
Signed-off-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
|
||
|
Salman Mohammed
|
813bbd1412 |
fix(acp): integrate the Buzz Pi adapter fork (#7552)
## Summary PR #7335 worked around missing Pi adapter support by generating a private Pi launcher and injecting Buzz's standing prompt and skills at process launch. The Buzz Pi fork now carries the required adapter extensions, so this removes that launcher and returns prompt construction to the normal ACP session path while retaining the base-prompt composition introduced by #7335. The Pi preset now installs `salman1993/pi-acp` and launches its renamed `buzz-pi-acp` binary. Buzz adds `-- --skill <harness-cwd>/.agents/skills` only when launching that binary, sends the complete composed prompt as the `_meta.systemPrompt` replacement string on `session/new` only when `initialize.agentInfo.name` is `buzz-pi-acp`, and sends the scoped title alongside it as `_meta.sessionTitle`. The fork identity is treated as system-prompt capable regardless of its reported ACP protocol version, which prevents duplicate legacy user-message framing. Upstream `pi-acp` does not receive either fork-specific behavior. Observer transcript projection accepts the string, `{ replace }`, and `{ append }` metadata forms. The fork now stores restore metadata in one atomic file per session under `~/.pi/buzz-pi-acp/sessions/`. This prevents concurrent Buzz workers from overwriting another session's prompt or title. The fix landed in [salman1993/pi-acp#9](https://github.com/salman1993/pi-acp/pull/9). This supersedes the closed #7508. No agent-configuration rules changed; this changes the Buzz Pi adapter contract and launch arguments. ### Related issue #7329 ### Testing Installed fork commit `09cf07e436b8f18e52401558f988f31a15702313` through the documented Git URL. The installed bundle matched the committed `dist/index.js` byte for byte and contained the `~/.pi/buzz-pi-acp` metadata path. The fork's 106 non-skipped tests, typecheck, and lint pass. Ran the ignored real-Pi integration test through Buzz's production session composer. The test exercised the renamed package, `agentInfo.name`, and the new per-session metadata store. Base, persona, team, core-memory, huddle, canvas, and skill markers each appeared once after switching sessions and again after restarting the adapter, while the other session and Pi's native default prompt were absent. Added regression coverage proving `buzz-pi-acp` receives fork-specific system-prompt metadata and managed skills while upstream `pi-acp` does not. `just ci` passes. Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> |
||
|
Logan Johnson
|
00209076c7 |
fix(relay): reject presence updates when Redis storage fails (#7532)
## Summary
- Reject kind:20001 presence events with `OK false` / `error: presence
storage unavailable` when Redis SET or DEL fails, before publishing,
local fan-out, or local-event marking.
- Preserve the producer contract needed by snapshot-confirming
consumers: delivered live presence must follow successful mutation of
the Redis state read by snapshots.
- Classify those backend rejections with the existing `IngestError`
taxonomy so a presence storage outage counts as
`buzz_events_rejected_total{transport="ws",reason="error"}`, not client
`reason="invalid"`; genuine client-input refusals (verification failure,
membership gates) stay `invalid`, and every wire message is an unchanged
fixed sanitized string (review follow-up, no protocol wording change).
- Add actual `handle_event` integration coverage for rejected
online/offline transitions, healthy online→offline
accepted/stored/fanned-out behavior, and the rejection-counter routing
on storage failure with an invalid-signature control.
This is standalone on main; it does not depend on the mobile
implementation. Deploy this relay prerequisite before relying on #7526's
snapshot-confirmation policy. Existing
pubsub-failure-after-successful-storage behavior and disconnect TTL
cleanup are deliberately unchanged. A storage error may be an ambiguous
write outcome, not a rollback guarantee; the rejected event is not
published by this handler. Clients may retry the generic `error:`
rejection. Desktop's 60s heartbeat retries non-offline presence, not
every explicit offline transition.
### Related issue
Addresses the relay prerequisite identified in [#7526 review
5157607827](https://github.com/block/buzz/pull/7526#pullrequestreview-5157607827).
Searched open presence/storage PRs; no duplicate relay storage-error
rejection fix found. #7382/#7383/#7526 heads and bases are unchanged.
### Testing
Exact head: `389174df29cc02d0f885c03209eff661d8bb2ec0` (+380/-13; 393
total), one commit `389174df2` on top of the reviewed `c031d6eb1`
(DCO-signed; base `bfc384855889432df4a333a0edf3080f332ee169` unchanged).
- PASS: `cargo fmt --all -- --check`, `cargo clippy -p buzz-relay
--all-targets -- -D warnings`, `git diff --check`, `just
file-size-check`, PostgreSQL discovery validation — all run at the exact
final head with a clean tree before and after.
- PASS: documented native `scripts/postgres-test-run.sh -p buzz-relay
--lib --tests`: **89/89** actual integration tests, including the four
presence cases (online/offline storage rejection, healthy
online→offline, and the new rejection-classification case). Owned
PostgreSQL 17/Redis on isolated loopback ports, schema plus
reconciliation applied; no shared development database.
- PASS: explicit `cargo test -p buzz-relay presence_storage -- --ignored
--nocapture`: **4/4**, not skipped.
- Full isolated relay crate suite at the final head (`cargo nextest run
-p buzz-relay --lib --tests`): **1062 run: 1062 passed, 94 skipped**.
The previously failing
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` passed
in this run (1.5s); it is a known timing-sensitive main baseline failure
tracked open in #7140 and untouched by this PR, so this single passing
run is reported as-is and does not claim environmental clearance or
close #7140. No full-suite-green claim is made beyond this run.
- Mobile is untouched; #7526's existing 2090-test/format/analyze
evidence remains scoped to its unchanged head. Its separate Desktop
Smoke E2E (2) failure remains red; no CI retries requested.
[Production-seam regression
coverage](
|
||
|
Diem Nguyen
|
051c3a270b |
fix(buzz-acp): report missing models without retrying (#7538)
## Summary When an agent reports model-not-found, Buzz retries the unavailable model and delays the failure reply until retries are exhausted. Stop retrying this error and immediately post a threaded recovery notice. The notice tells users to select a different model in agent settings, save, restart the agent to apply the configuration, and re-send their request. This adds one error-handling branch and regression coverage in `buzz-acp`. It matches `-32002` errors containing `model not found`. Other resource-not-found errors, such as stale sessions, retain the existing retry behavior. Detailed error events remain available for diagnosis. The existing restart policy is unchanged. ### Related issue None found in existing issue/PR searches for model-not-found recovery. ### Testing Playwright captured and visually checked the thread UI with seeded conversation data and the exact recovery text. The check opens the request's thread, confirms no reply before the failure, injects the notice, and verifies the full text is visible. [Before/after screenshots](https://github.com/block/buzz/pull/7538#issuecomment-5608196506) show the corrected save-and-restart instructions. These are local test captures, not a deployed provider recovery flow. Generated with Codex --------- Signed-off-by: Diem Nguyen <diem@squareup.com> |
||
|
Logan Johnson
|
c045321a7f |
fix(acp): pace targeted overflow recovery on consumer capacity (#7325)
🤖 ## Summary When a Buzz agent falls behind on incoming messages, its connection can make the backlog worse while trying to recover. The connection buffers messages from the relay server until the agent is ready to process them; if that buffer overflows, recovery previously requested history for **every subscribed channel** and paused socket reads while sending those requests. That adds traffic to an already overloaded connection. This change requests history only for affected subscriptions, once the code consuming those messages has room, with at least five seconds between attempts. The recovery path now: - Combines repeated losses into one pending recovery per affected subscription, keeping the oldest dropped timestamp so replay starts early enough. - Waits until at least half the consumer queue is free and the relay's existing rate-limit delay has expired. The queue wakes recovery when space becomes available; recovery does not periodically sample capacity or hold queue space away from live messages. - Attempts one subscription at a time, choosing the least recently attempted so a busy channel cannot crowd out other channels or membership notifications. The five-second delay starts when an attempt finishes, including a failed write; failed writes leave recovery pending. Recovery is paced by available capacity, not by how often messages are lost. This is not a larger buffer or a cutoff that abandons recovery. Subscription identifiers, message filters, replay timestamp overlap and duplicate filtering are unchanged; no downstream agent changes are required. This targets a reproducible overload **amplifier**, not every cause of overload or every catch-up limitation. The initial live overload's cause has not been established. Recovery remains best effort: a successful request write is not proof of delivery, and existing history/retention limits, bounded duplicate tracking and replay limitations still apply. There is no exactly-once or complete catch-up guarantee. A stalled write can still pause socket reads for the existing ten-second timeout; the pacing bound does not cover initial subscriptions, reconnects or other retry paths. ### Related issue Closest related: #5014 (channel re-subscription); also #6661 (membership reconciliation) and #6090 (relay backpressure gap signaling). This addresses local overflow recovery scheduling, not those separate mechanisms. ### Testing Recorded offline comparisons against the previous behavior, with the final implementation at `8000636f3073167c5a5107bb179c7d91160f1729`: | Same fixture: 18 subscriptions, three overload rounds | Before | After | | --- | --- | --- | | Recovery history requests | 108 | 3 | | Ping-response delay | About 4.6 seconds | Below the measurement's 1 ms resolution | A separate bounded-history fixture delivered all 320 events plus subsequent live traffic in **both** versions. Regression coverage exercises the real socket-handling task, including intermittent consumer capacity, fairness, failed writes and cancellation of capacity waits before live delivery. These are synthetic results, not production throughput measurements or evidence of a deployed cure. The full local `RUST_TEST_THREADS=4 just ci` run passed on September 4, 2026. Earlier unsuccessful local runs remain part of the validation history. The [recorded validation evidence and separate desktop follow-up](https://github.com/block/buzz/pull/7325#issuecomment-5540592398) preserve the original desktop mock-history scroll failure, its passing rerun and the remaining investigation. That desktop path does not run the agent connection code; neither this repair nor the passing rerun fixes the observed scroll problem. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> |
||
|
Mic Neale
|
44316ff72f |
feat(mesh): upgrade to mesh-llm 0.76.0-rc8 and recommend Qwen3.8 27B (#6189)
Upgrades Buzz's mesh-llm dependencies through the released `v0.76.0-rc8` tag (`2040765d`), including the Qwen3.8 curated recommendation and rc8's scheduler/runtime improvements. **Scope note:** the earlier open-relay/unenforced-admission mode has been removed from this PR at Mic's direction — it is not a product mode we want. Mesh admission remains roster/allowlist driven, exactly as on `main`: on a relay with no NIP-43 membership snapshot the mesh runs self-only. No NIP-11 mode probing, no mode-transition restarts. A future perimeter/admission strategy for open relays will be designed separately. This PR also: - seeds `BUZZ_AGENT_LLM_TIMEOUT_SECS=660` for mesh agents, above MeshLLM's 600-second backend timeout; - makes `desktop-tauri-clippy` lint both default and `mesh-llm` cfg graphs; - runs the feature-enabled desktop test suite in CI; - recommends Qwen3.8 27B Q4_K_M for 64 GB-and-larger machines, then ladders down through Gemma 4 E4B and Qwen 9B for smaller machines; - keeps stored shared-compute `auto` translated to MeshLLM's supported wire model `mesh`. RC8 verification: - `just ci` passed locally at `92ecc7ec933bdd4df804cc9f28a2b51efa5313c5`. - Pre-push differential gates passed, including both desktop Tauri cfg graphs and package tests. - A prior isolated runtime smoke used the RC8 binary's OpenAI endpoint for a Buzz-shaped system/user/tool/tool-result/final-response loop; all assertions passed and the isolated process was shut down. Perf previously measured on M5 Metal, Qwen3.8-27B-Q4_K_M: TTFT 0.22–0.32 s, ~25 tok/s streaming; agent-shaped turns ~1 s to first token after the first (prefix cache). --------- Signed-off-by: Michael Neale <michael.neale@gmail.com> Signed-off-by: Jimmy <1fe240cd1a8cf775f6f3060f115e5a303181f3abf28ad4cb0c2515f4a02b36a8@meshllm.communities.buzz.xyz> Signed-off-by: Alessandro Joabar <sandro@squareup.com> Co-authored-by: Michael Neale <michael.neale@gmail.com> Co-authored-by: Jimmy <1fe240cd1a8cf775f6f3060f115e5a303181f3abf28ad4cb0c2515f4a02b36a8@meshllm.communities.buzz.xyz> Co-authored-by: Mic Neale <305999590+micspiral@users.noreply.github.com> Co-authored-by: Alessandro Joabar <sandro@squareup.com> |
||
|
Bradley Axen
|
cd54e26823 |
fix(agent): route GPT-5+ model-service FQNs to Responses (#7358)
## Summary Route Databricks Unity Catalog model services to OpenAI Responses when their service name matches GPT-5 or newer. These models can reject tools plus reasoning on Chat Completions. Match only the service component, using the existing family-token boundaries and a numeric major version. Catalog and schema names cannot select the protocol. Keep neutral effort capabilities and the full model ID unchanged; other services still use MLflow Chat Completions. Keep the Rust and desktop resolvers in sync, add shared boundary cases and a captured-HTTP regression for completion and summarization, and update the documented FQN rule. ### Related issue No duplicate found in searches for “FQN responses” PRs or “astra” issues. Related: #6918 introduced Unity Catalog discovery. Originating conversation: buzz://message?channel=0b881928-a3a6-4c01-b981-8e64268f01ce&id=770949343bc96a9ed88acd90a1b37d358a0efc52c79237d0fdb491ce02b8d4ed ### Testing No live Databricks inference test. The gateway must accept the full model-service ID on its OpenAI Responses route; this remains the integration risk. The local `just ci` attempt exceeded its five-minute deadline during `mobile-check`, so the full repository gate was not completed. All push-hook checks passed. Generated with Codex Signed-off-by: Fizz <400e8babadcee6a7f420103f10a2849d84c4a9c71d5bd04f3948c814216648a3@buzz.block.builderlab.xyz> Co-authored-by: Fizz <400e8babadcee6a7f420103f10a2849d84c4a9c71d5bd04f3948c814216648a3@buzz.block.builderlab.xyz> |
||
|
Salman Mohammed
|
86c189e857 |
fix(buzz-acp): wake held ACP threads and fence forked sessions (#7340)
## Summary Adds an independent deadline wakeup so held thread work dispatches after its 10-second bound even when the relay loop is otherwise quiet. Fences session ownership by generation so a worker returning after a fork cannot make an older provider session claimable again. This follows up on the two post-merge findings from [#7337](https://github.com/block/buzz/pull/7337#pullrequestreview-5116329341). ### Related issue Follow-up to #7337. ### Testing - `cargo test -p buzz-acp` - `cargo clippy -p buzz-acp --all-targets -- -D warnings` - Pre-push file-size, differential Rust test, and desktop Tauri gates No UI changes. --- **Update Sep 4, 15:35:** Addressed both Codex review findings. - Queue-cap eviction now prunes orphaned hold deadlines. - An expired hold stays expired until a worker is successfully claimed. - Hold timers remain disabled while every worker is busy; worker return wakes dispatch directly. - Added regressions for queue eviction and pool exhaustion. Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> |
||
|
Jordan Mecom
|
4d447b9c20 |
Add generic information-flow control core (#7293)
Adds a dependency-free `ifc-core` crate implementing reader-set confidentiality labels and monotonic per-computation flow state. - Defines flow ordering, join, and meet over caller-supplied principal universes. - Fails closed for unknown or cross-universe input and checks reader widening at egress. - Includes property tests for lattice laws and monotonic taint, plus the design paper that motivates the broker integration. The crate deliberately contains no Buzz, Nostr, channel, membership, or grant policy; those remain in the Buzz adapter. --------- Signed-off-by: Jordan Mecom <jm@squareup.com> |
||
|
Salman Mohammed
|
4beffef697 |
feat(buzz-acp): update base prompt; add buzz context and skills to Pi agents (#7335)
## Summary - launch Pi through a private Buzz wrapper so managed Pi agents receive Buzz's base prompt plus the configured thread/channel session model via `--system-prompt` - load the Buzz workspace skill directory with Pi's repeatable `--skill` flag, making `~/.buzz/.agents/skills/buzz-cli` available alongside Pi's normal global skill discovery - report Pi and `pi-acp` installation states separately so setup guidance points to the missing component `pi-acp` does not currently consume the ACP `session/new` system prompt, but its normal new-session and restore paths do honor `PI_ACP_PI_COMMAND` as the executable used to launch Pi. Buzz reserves that variable and sets it to a private generated launcher. The launcher invokes `pi` from Buzz's effective `PATH`, adds `--system-prompt <file>` and `--skill <workspace>/.agents/skills`, and forwards `pi-acp`'s RPC and session arguments unchanged. This keeps the integration entirely in `block/buzz`, without changes to Pi or `pi-acp`. ### How PI_ACP_PI_COMMAND works in `pi-acp` `PI_ACP_PI_COMMAND` selects one executable; it does not accept arguments. Therefore: ```bash # Does not work PI_ACP_PI_COMMAND="pi --skill ~/.buzz/.agents/skills" ``` Buzz does not expose this variable as user configuration. It creates a private launcher and sets `PI_ACP_PI_COMMAND` only on the `pi-acp` child. Conceptually, that launcher executes: ```sh #!/bin/sh exec pi \ --system-prompt "$PRIVATE_SYSTEM_PROMPT" \ --skill "$HOME/.buzz/.agents/skills" \ "$@" ``` ### Related issue Follow-up to #7208. No duplicate issue or open PR found. ### Testing - `just ci` - `cargo test -p buzz-acp pi_launcher` - `cargo test --manifest-path desktop/src-tauri/Cargo.toml managed_agents::env_vars::tests` - `cargo test --manifest-path desktop/src-tauri/Cargo.toml managed_agents::discovery::presets::tests` - pre-push branch checks No screenshots: the UI change is state-dependent setup guidance only. --- **Update Sep 4, 13:35:** Made `PI_ACP_PI_COMMAND` entirely Buzz-owned. - Managed agent configuration now rejects the variable as a user override. - Buzz always launches `pi` from its effective `PATH` and rejects inherited values before creating its private launcher. - Removed the custom Pi executable discovery path and its internal environment alias. --- **Update Sep 4, 14:03:** Reject inherited `PI_ACP_PI_COMMAND` values instead of replacing them. - Pi startup now fails with an actionable message telling the user to unset the variable. - Removed the Pi-specific exception from the generic ACP environment injection path. Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> |
||
|
Jordan Mecom
|
e09f715c9d |
Verify ACP relay events before prompt routing (#7010)
## What changed Verify every Nostr EVENT received by buzz-acp immediately after deserialization. Events with an invalid NIP-01 ID or Schnorr signature are dropped before subscription routing, deduplication, replay-watermark updates, membership handling, or either harness queue. ## Safety Signature verification runs on the blocking pool so cryptographic work does not block the relay task. A verification failure drops only that event and keeps the connection available for subsequent valid traffic. The existing observer-control verification remains as defense in depth. Regression tests cover valid delivery; changed content, ID, signature, pubkey, tags, and timestamp; a forged owner shutdown command with a recomputed ID; forged membership notifications; and forged observer-control events. ## Testing cargo test -p buzz-acp cargo clippy -p buzz-acp --all-targets -- -D warnings just ci --------- Signed-off-by: Jordan Mecom <jm@squareup.com> |
||
|
Will Pfleger
|
b17c0776b7 |
fix(buzz-acp): bound busy-owner hold to prevent cross-channel starvation (#7337)
## Problem #6732 added a busy-owner hold to the ACP harness: when a scope's recorded session owner (`session_owners`) is checked out on **any** turn, `dispatch_pending` holds the scope's batch instead of dispatching it. The hold was added to keep one provider session per thread — but it is unconditional: it applies to `Conversation` scopes too, and it has no time bound. Under the default `session_policy=channel`, every channel collapses to a single `Conversation` scope, so once two channels' sessions land on the same worker (pass 2 of `try_claim` picks the first idle worker by index, so this happens quickly after any restart), channel A's mention starves behind channel B's in-flight turn — for up to the full `max_turn_duration` (7200s by default) — while other workers sit idle. The only signal is a DEBUG-level log, and the 👀 seen-reaction is added at queue admission *before* the hold decision, so the user sees the agent acknowledge the mention and then nothing. Observed in production on the first day of the v0.5.22 rollout: three separate incidents where a mention got 👀 but no turn started until an unrelated channel's turn ended on the shared worker (in the worst case the blocking turn sat in a single tool call for 6+ minutes). ## Fix One new seam, `AgentPool::hold_decision`, replaces the raw `should_hold_for_busy_owner` check in `dispatch_pending` (the predicate itself is unchanged and remains the inner check): - **`Conversation` scopes never hold.** Channel-policy channels and all DMs dispatch immediately; a busy owner means forking onto an idle worker, exactly the pre-#6732 behavior. This removes the cross-channel head-of-line blocking entirely for the default policy. - **`Thread` scopes hold for a bounded window.** `HOLD_BUSY_OWNER_TIMEOUT` (10s) is measured from the first time the batch is held (`held_since` stamp); once elapsed, the batch stops holding and forks a fresh session on an idle worker, rebuilding thread context from the relay. This preserves #6732's session-continuity intent for the momentary-busy case while capping the worst-case wait. No new timer is needed: held batches are requeued with preserved timestamps and re-evaluated on every dispatch trigger (turn end, relay event, 30s maintenance tick), so the effective worst-case re-check gap on a fully silent system is one maintenance tick. - **Holds are observable.** Holding logs at INFO and a hold expiry logs at WARN (previously DEBUG-only), and both emit observer-feed events (`busy_owner_hold`, `busy_owner_hold_forked`) with the scope, owner index, and held duration. `held_since` is derived state and is cleared on every removal path: dispatch/fork (inside `hold_decision`), `invalidate_channel_sessions`, `invalidate_scope_session`, and `switch_idle_agent_model`. ## Accepted trade-offs - A fork after an expired hold leaves the old owner's now-orphaned thread session in its session map until natural rotation/invalidation — benign, and identical to pre-#6732 fork semantics (`loadSession: false`; sessions are worker-pinned, so migration is not an option). - Under sustained pool exhaustion the hold stamp is cleared on the fork attempt and re-stamped next cycle, so the bound is effectively "timeout after a worker frees up," not absolute wall clock. ## Tests - New table test `hold_decision_covers_variant_session_busy_and_timeout` over the full input space (scope variant × idle-session presence × owner busyness × elapsed vs. window). The `Conversation` + busy-owner row is the cross-channel regression guard; the past-window row guards the bound. Both were mutation-checked: removing the variant gate or the timeout branch fails the suite. - `busy_session_owner_holds_batch_instead_of_forking_session` extended with the Hold → ForkAfterHold transition, the `Conversation` dispatch guard, and `held_since` pruning on channel invalidation. - Scope-invalidation and idle-model-switch tests extended to cover `held_since` cleanup alongside the existing `session_owners` assertions. Signed-off-by: Will Pfleger <pfleger.will@gmail.com> |
||
|
Salman Mohammed
|
ce9decb235 |
fix(acp): rename system tag to agent-instructions (#7332)
## Summary Rename the paired persona prompt boundary from `<system>` to `<agent-instructions>` in both modern and legacy ACP delivery paths. Update Desktop diagnostics to label the new boundary as Agent Instructions while keeping archived `<system>` captures readable. `system` is confusing (esp with system role) and agent instructions is what shows up in the UI today <img width="767" height="298" alt="Screenshot 2026-09-04 at 11 37 54 AM" src="https://github.com/user-attachments/assets/4e34ac06-db50-461e-a555-ce17885b0031" /> ### Related issue Follow-up to #6701. No duplicate open issue or PR found. ### Testing - `cargo test -p buzz-acp` - `cargo clippy -p buzz-acp --all-targets -- -D warnings` - `node --import ./desktop/test-loader.mjs --experimental-strip-types --test desktop/src/features/agents/ui/agentSessionTranscriptHelpers.test.mjs desktop/src/features/agents/ui/agentSessionTranscript.test.mjs` - `pnpm --dir desktop typecheck` - Pre-push hook: 6,237 Desktop tests, 3,164 Tauri tests, and all 13 Rust unit-test lanes passed Generated with Codex Signed-off-by: Salman Mohammed <smohammed@squareup.com> |
||
|
Luke Tornquist
|
cd02b693aa |
refactor(relay): extract NIP-29 membership authorization (#7285)
## Why NIP-29 membership authorization for kinds 9000, 9001, and 9022 was embedded in a large async handler, requiring Postgres and Redis to test and repeating the last-owner rule across five call sites. This addresses [TheSentinel454/buzz#24](https://github.com/TheSentinel454/buzz/issues/24) without adding the broader state/repository abstraction suggested there because HTTP and WebSocket writes already share ingestion. ## What - Extract pure, typed membership authorization decisions while leaving database reads and mutations in `validate_admin_event`. - Preserve existing client-visible errors and independent database last-owner safeguards. - Collapse five last-owner policy restatements into one predicate and share the identical self-departure policy. - Include the relay decision modules in `just test-unit`, so these tests execute in CI. ## Risk Assessment Medium-low. This touches production relay authorization, but intentionally preserves wire behavior and database defense in depth; exhaustive decision tables and relay-backed tests cover the affected paths. ## Simplification This removes repeated policy from the orchestration path and makes the rule set directly testable without introducing a repository trait or second transport path. ## Verification Verified at `15255a090797f85874921120003c645962fefaed`: - `just test-unit` — all 10 package summaries passed. An initial run hit two unrelated timing-sensitive `buzz-acp` failures; the complete retry passed 905/905 in that package. - `cargo fmt --all -- --check` - `cargo clippy --workspace --all-targets -- -D warnings` - `just file-size-check` — 10/10 policy tests passed. - `just security-review-check` — 13/13 tests passed. - Push preflight — `push-head-scope`, `branch-skew`, `file-size-check`, `rust-tests`, and `desktop-tauri-checks` passed. - Blox Postgres relay lane — 84/84 at the byte-identical source patch; full live `e2e_relay` behavior matched the clean base (45 pass and the same pre-existing kind:9002 failure on each). Generated with Claude Code --------- Signed-off-by: tornquist <tornquist@squareup.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com> |
||
|
ravarora2
|
88687876f7 |
feat(relay): add early startup lifecycle logs (#7258)
## Why Early relay failures can currently appear as a container restart without a trustworthy in-process account of whether crypto, structured logging, configuration, relay identity, or the metrics listener failed. Most of those steps happen before the Prometheus exporter exists, so their chronology belongs in logs rather than metrics. Implements the logs-only early-startup slice of #7238. Post-bind Prometheus exporter supervision is tracked separately in #7284. ## What changed - create a process lifecycle recorder before the Tokio runtime and emit a fixed, versioned JSON schema directly to stderr; - record started and exactly one terminal event for `crypto_init`, `tracing_init`, `config_load`, `key_load`, `metrics_bind`, and the aggregate `process_telemetry` phase; - keep every status and reason bounded and suppress raw errors that could contain credentials, keys, URLs, or other secrets; - return typed metrics-install errors so `metrics_bind` can be classified without logging raw values, while preserving the existing public `metrics::install` API; - document the logs-only evidence contract and add real child-process regressions for success and failure paths. This PR adds **no startup metric families** and no dashboard contract. Existing application metrics remain unchanged. ## Verification Exact head: `8faf7526822a119efa035e58b2b3c59aa67fc81d` - `cargo fmt --all -- --check` - `cargo clippy -p buzz-relay --all-targets -- -D warnings` - relay binary target: 13 passed, 1 PostgreSQL-only test ignored - real relay child-process lifecycle target: 9 passed - full relay package library target: 1,023 passed, 89 ignored; the same six media tests failed at `crates/buzz-relay/src/api/media.rs:1145` with `Sqlx(PoolTimedOut)` because local PostgreSQL is unavailable - three independent exact-head reviews found no correctness, security, compatibility, lifecycle-accounting, logs-only-scope, or test-adequacy finding All exact-head GitHub CI gates are green, including lint, unit tests, PostgreSQL, relay/backend/desktop integration, both Linux server cross-compiles, Windows/macOS builds, and security checks. ## Staging verification - exact multi-architecture image: `dev-sha-8faf7526822a119efa035e58b2b3c59aa67fc81d-run-33708188952-1` - immutable manifest: `sha256:26cad28266a6bb0b0e7081eb6091d374e5489f8bb78c475a4a65737dee86cc67` - image workflow: https://github.com/block/buzz/actions/runs/33708188952 - focused staging deployment: https://github.com/squareup/builderbot-platform-core-infrastructure/pull/314 - replacement ReplicaSet `buzz-d68764bc7` has two Ready pods with zero restarts - Datadog received one complete, contiguous sequence 1-12 from each pod; both end with `process_telemetry/terminal/succeeded` at 3 ms - queries scoped to the replacement ReplicaSet return no data for the removed `buzz_startup_phase_terminal` or `buzz_startup_phase_duration_seconds` families The experimental Row 7 was removed from the Buzz Startup & Rollout Safety dashboard. This logs-only PR deliberately adds no replacement dashboard row. --- **Update Sep 3, 12:26 ET:** Clarified the review boundary: this PR does not close the broader #7238. Later exporter-task termination is pre-existing runtime behavior and is now explicitly tracked in #7284; no production code or staged image changed in this update. Generated with Codex Signed-off-by: Ravneet Arora <rarora@squareup.com> |
||
|
klopez4212
|
c6ca9d94d9 |
Show status and huddle indicators beside names (#7112)
## Summary - show status and huddle emoji beside names in DMs and message rows - provide accessible tooltips, fallback status emoji, and profile-menu icon replacement - add the desktop status editor with preset durations, a ShadCN calendar, and a capped half-hour time menu ## Validation - desktop checks, typecheck, and file-size guard - 5,802 desktop tests - focused Playwright coverage (3 passed) - E2E build and native Builderlab staging verification Updated visual snapshots are attached in the PR comments. --------- Signed-off-by: kenny lopez <klopez4212@gmail.com> Signed-off-by: Princess Donut <b238ea756dee4d98afa5883fc7f1de61eeabe65bf700e3a5a5a80db5e42e2c2b@buzz.block.builderlab.xyz> Co-authored-by: Princess Donut <b238ea756dee4d98afa5883fc7f1de61eeabe65bf700e3a5a5a80db5e42e2c2b@buzz.block.builderlab.xyz> |
