## Why
The partition manager checks partition existence by name only and treats
Postgres 42P17 "would overlap partition" errors as success. On databases
where a catch-all partition (`events_p_future` /
`delivery_log_p_future`) covers future months, the monthly creation path
silently does nothing while real coverage erodes toward a write outage.
There is currently no visibility into partition catalog health and no
signal before writes start failing.
## What
- **Read-only catalog audit** in `buzz-db` (`partition.rs`): classifies
the live partition catalog for `events` and `delivery_log` via
`pg_inherits` and bound parsing — coverage bounds, catch-all detection,
anomalous children, and trigger parity checked recursively down to
routable leaf partitions.
- **Creation gated on the audit verdict**: the 42P17-to-success swallow
is removed; monthly CREATE runs only for months the audit proves
uncovered, a canonical-name mismatch is an error, and per-month outcome
capture replaces first-error abort. New
`BUZZ_PARTITION_MANAGER_CREATE_ENABLED` kill switch (default `true`).
- **Prometheus metrics** per table: `buzz_partition_serving_safe`,
`buzz_partition_uncovered_months` (audit window is now through +3
months), `buzz_partition_catch_all_covered_months`,
`buzz_partition_anomalous_children`,
`buzz_partition_trigger_parity_missing`/`_extra`,
`buzz_partition_audit_duration_seconds`,
`buzz_partition_audit_runs_total{outcome}`,
`buzz_partition_audit_failures_total`,
`buzz_partition_create_attempts_total`, and
`buzz_partition_audit_last_success_timestamp_seconds` for staleness
alerting.
- **Cached diagnostics**: `/_status.partition_catalog` reports partition
serving-safety, audit age, and freshness. `/_readiness` stays
process-local, consistent with #7341; missing, unsafe, or stale audits
do not remove pods from service.
- **Periodic audit task** in the relay, plus config surface for the
audit interval and the monthly-create kill switch. Catch-alls are
detected from catalog bounds rather than configured names.
- Doc correction: the mesh heartbeat comment no longer claims readiness
gating (it never was gated; behavior unchanged).
## Behavior changes & risk
- No DETACH/DROP/ATTACH and no schema migrations. The only DDL is the
pre-existing monthly CREATE path, now strictly narrower
(audit-proven-uncovered months only) and behind a kill switch.
- Partition audit health does not gate readiness. Until the first
completed audit, read-only retries back off from 5 seconds to a
60-second cap. Any completed audit, including unsafe coverage, restores
the normal interval.
- Partition-overlap errors now surface as errors instead of being
swallowed as success.
- Rollback: revert the commit, or set
`BUZZ_PARTITION_MANAGER_CREATE_ENABLED=false` to disable the creation
path while keeping the audit and metrics.
## Validation
- New Postgres-backed regression tests for catalog classification and
trigger parity (disabled child trigger, `ENABLE ALWAYS` parent, disabled
nested leaf clone in a multi-level layout, healthy-catalog zero-parity
checks). Each new regression was verified to fail against the previous
logic before the fix.
- Local affected-target validation passed against a disposable
current-schema database, with the known mesh forwarded-echo and tracing
callsite failures excluded. This is not a claim that the full suite
passes.
Follow-up work (separate PRs): bounded catch-all advance (empty-only
golden path; non-empty reattach only behind an explicit operator flag,
default off), fresh-install convergence, and proactive coverage-runway
alerting.
---
## Update — 2026-08-21
- Fixed nested-leaf trigger parity auditing so child-only row triggers
are checked on every routable descendant leaf and reported as
`{leaf}:{trigger}` under the owning root child.
- Added a Postgres regression test for a child-only trigger on a nested
leaf; mutation-reverting to immediate-child-only extra-trigger scanning
makes it fail.
Generated with Codex
Signed-off-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz>
Signed-off-by: Ian Oberst <ioberst@squareup.com>
Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
We are introducing a companion service to the relay that needs pubkey
event mentions for pubkeys it manages. This PR introduces an outbox for
pubkey mentions that will be sent to these "operator listener" services.
## Summary
Deliver mentions of registered pubkeys to deployment-global operator
listeners.
- Configure listeners with `BUZZ_OPERATOR_LISTENERS`; listeners manage
target pubkeys through NIP-98-authenticated `POST`/`DELETE
/operator/listener/pubkeys` requests.
- During supported event ingestion, match `p` tags to registrations and
insert one outbox row per listener in the event transaction.
- A background worker claims up to 10 rows, sends NIP-98-signed
notifications, and retries transient failures with exponential backoff
for up to 9 attempts (idle polling backs off from 250 ms to 2 s).
- Outbox rows are ineligible for delivery after 15 minutes and are
reaped every 5 minutes. Registered pubkeys live for 30 days;
re-registering renews them, and a daily reaper removes expired
registrations.
- Keep persistence app-managed: no database triggers, functions, foreign
keys, or separate match queue.
## Changed behavior
Listener outbox rows are enqueued inside the shared event-insert and
thread-metadata transactions; enqueue failure rolls back the event.
`event_mentions` behavior is unchanged: ordinary event/reaction mention
indexing remains post-commit best effort with warning-only failures.
## Testing
- `cargo fmt --all`
- `cargo test -p buzz-db --lib`
- `cargo test -p buzz-relay --lib
config::tests::operator_listener_routes`
- `cargo check -p buzz-db -p buzz-relay`
The relay can use a configurable HTTP(S) gateway delivery URL and signs
the same URL it sends. The gateway verifies NIP-98 against the received
request URL, including authority, port, path and query. App Attest,
scoped grants, replay checks and delivery quotas are unchanged.
Forwarding proxies must preserve Host and supply the original request
scheme through X-Forwarded-Proto. The NIP relay-delivery wording now
defines the configured HTTP(S) URL as the signed URL and labels the
public URL as an example.
<!-- buzz-review-completed -->
---------
Signed-off-by: Tom Brow <tomb@block.xyz>
Make the push gateway origin configurable across the relay, gateway,
mobile client, and deployment chart.
This is a fresh deployment. The gateway has only run on personal
development infrastructure, and push has no existing enabled users.
Legacy migration, configured-to-unconfigured mobile transitions, and
rolling upgrades from pre-launch gateway binaries are not supported.
Populated pre-launch databases are rejected with an informative error
before migrations run; their data is untouched. Mobile builds may omit
the gateway URL, leaving push unavailable. Once push is enabled,
subsequent builds must retain that configuration.
### Validation
- All applicable CI checks pass at
`5f546cfaf403f0386f146d213adfb4392f4ec29c`, including Rust,
mobile/Swift, desktop/integration, security, image builds, and Helm
checks. An unrelated agent cancellation test passed on retry.
- PostgreSQL CI passed 384 tests, including the production-entry-point
regression for refusing populated pre-launch databases before
migrations, fresh initialization, and subsequent initialized
deployments.
- Focused local validation passed all 10 gateway PostgreSQL tests under
the CI discovery filter and all 3 unconfigured-mobile tests.
Repository-wide PostgreSQL discovery validation and the complete
pre-push hooks passed.
- Physical-device APNs/App Attest behavior and deployment to a live
cluster were not verified in this work.
---------
Signed-off-by: Tom Brow <tomb@block.xyz>
Co-authored-by: Codex <noreply@openai.com>
## What this does
In a channel, people often run several unrelated conversations at once
(separate threads). Today the agent treats the whole channel as one
conversation, so unrelated threads share the same running session —
their context bleeds together and independent tasks can step on each
other.
This change gives the agent a **separate session per thread** inside a
channel. Direct messages stay as one conversation (unchanged). The
channel is still the boundary for who is allowed in and what is visible
— only the agent's working context is now split by thread.
## How it is turned on
Off by default. Operators opt in with one setting:
- `BUZZ_ACP_SESSION_POLICY=channel` — default, current behavior
- `BUZZ_ACP_SESSION_POLICY=thread` — new per-thread behavior
Being behind a flag means we can enable it for a few agents, watch how
it behaves, and roll back instantly without a code change.
## Key design decisions
- **Decide the thread once, up front.** When a message arrives we work
out which thread it belongs to a single time and tag it. Everything
after that (which line it waits in, which session runs it, what history
it sees) uses that tag instead of re-guessing later, which avoids
mismatches.
- **Default stays identical to today.** Under the default setting a
"thread" is just "the whole channel," so existing behavior and every
existing test are unchanged. The new, riskier behavior is strictly
opt-in.
- **Give the agent only its thread's history.** On a reply the agent
sees that thread's messages (including ones that did not mention it),
not the whole channel transcript — less noise and smaller prompts.
- **Don't let one channel use more memory than before.** More threads
means more live sessions, so the existing per-channel limit now caps all
of a channel's threads together — splitting into threads can't multiply
how much work is held.
## Bugs found and fixed while iterating (from review)
- **Same thread, two sessions.** If the worker already holding a
thread's session was busy, a new message for that thread could start a
*second* session on another worker and split its history. Now it waits
for the right worker instead of forking.
- **Interrupting the wrong thread.** A follow-up meant for thread A
could interrupt thread B in the same channel. Interrupts now target the
exact thread.
- **Stuck thread after a crash.** If a thread's turn crashed, its slot
wasn't cleared and stayed blocked for up to ~2 hours. It now clears
right away and retries.
- **Lost the original request.** When a thread was interrupted and then
had to wait for a busy worker, only the follow-up was kept and the
original request was dropped. The full request is now preserved on
retry.
- **Same thread seen as two.** Two spellings of the same thread id
(upper/lower case) could be treated as different threads. Normalized so
they count as one.
## Not in this PR
- The desktop Settings toggle and rollout wiring for managed agents —
https://github.com/block/buzz/pull/6909
- One pre-existing retry edge case (present today without this flag,
unrelated to this change) — tracked separately so this PR stays focused.
## Testing
The full `buzz-acp` test suite passes (830+ unit and integration tests),
plus new focused tests for thread routing, session reuse, interrupt
targeting, crash recovery, and request preservation. Behavior with the
flag off is unchanged.
---------
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
Signed-off-by: Leo <5faf251baee50ee6bcde338aef6acdd70bb3e60115664c2cd490d94a55dfc488@buzz.block.builderlab.xyz>
Co-authored-by: Leo <5faf251baee50ee6bcde338aef6acdd70bb3e60115664c2cd490d94a55dfc488@buzz.block.builderlab.xyz>
## Summary
Introduces a protected-build boundary for the default-off Bestie
experiment without adding any Bestie product surface.
- Official OSS builds select an empty protected-feature module and emit
no Bestie/Chief metadata or implementation content.
- Protected internal builds select a separate module graph containing
the Bestie experiment definition.
- Within an internal build, Bestie remains disabled until the user opts
in under Settings → Experiments.
- The production build runs an artifact matrix and fails if OSS output
contains protected content or internal output lacks the Bestie manifest.
## Build contract
| Build variant | User opt-in | Result |
| --- | --- | --- |
| Official OSS | Any/forged | Bestie absent from the compiled artifact |
| Protected internal | Off | Bestie available but disabled |
| Protected internal | On | Bestie enabled |
The companion protected-release change is squareup/buzz-releases#91. It
sets `VITE_BUZZ_BESTIE=1`, requires that exact value, forwards it into
the signed macOS build, and asserts the contract in release validation.
## Why this is separate
This gives later Bestie PRs one build-selected import seam. Protected
implementations must be reachable only from the internal module so they
never enter the official OSS module graph.
## Non-goals
- No Bestie persona or provisioning
- No sidebar, app-chrome, or message-toolbar UI
- No entitlement or secrecy claim: the source is public; this boundary
controls official Block artifacts
## Verification
- Exact commit `523cf49ced03cba9be43836a54d6aa5d6923cc82`
- Full `just ci`: 5,673 Desktop tests, 2,773 Tauri tests, 1,860 mobile
tests, Rust/Tauri/web/mobile static checks and builds
- OSS production artifact: scanner confirms no `Bestie`, `Chief of
Staff`, or `builtin:bestie` content
- Internal production artifact: scanner confirms the protected Bestie
manifest is emitted
- Both build orders verified; `dist` retains the requested variant for
Vite/Tauri packaging
---------
Signed-off-by: Arjun Mahanti <arjun@squareup.com>
Signed-off-by: Fizz <fizz@buzz.local>
Signed-off-by: Fizz <dae5f6af70b8695a8b83c8deae555f63be41630ec2b8cd493e41a439c9527dd8@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Fizz <fizz@buzz.local>
Co-authored-by: Fizz <dae5f6af70b8695a8b83c8deae555f63be41630ec2b8cd493e41a439c9527dd8@buzz.block.builderlab.xyz>
## Why
A wedged relay boot pod holding a relation lock can park every other
writer in the fleet behind it: DB load pins at pool capacity in
`Lock:relation` waits while CPU stays flat, and nothing server-side
releases the lock until the holder dies. We hit exactly this in
production — ~1,400 sessions queued behind one crash-looping pod's boot
transaction for ~20 minutes until kubelet killed the container.
## What
Applies session-level Postgres timeouts to every **writer** connection
inside the existing single `after_connect` hook in `buzz-db`, all
env-tunable through the same `Config::from_env → DbConfig` path as the
existing pool-size knobs:
| Env var | GUC | Default | Effect |
|---|---|---|---|
| `BUZZ_DB_LOCK_TIMEOUT_MS` | `lock_timeout` | 5000 | statements waiting
on any lock fail fast instead of parking behind a wedged holder |
| `BUZZ_DB_IDLE_TXN_TIMEOUT_MS` | `idle_in_transaction_session_timeout`
| 60000 | reaps wedged clients idling inside an open transaction while
holding locks |
| `BUZZ_DB_STATEMENT_TIMEOUT_MS` | `statement_timeout` | 0 (off) |
opt-in runaway-statement cap; off by default because startup
migrations/backfills legitimately run long statements |
`0` disables a timeout (Postgres semantics) and deliberately passes
through the env parsing — unlike the pool-size knobs where `0` falls
back to the default. The reader pool is untouched: replica sessions
never take contended locks and already fail acquire in 150 ms.
Deployers tune these via plain env vars (`.env`, or `relay.extraEnv` in
the Helm chart) — no code changes needed.
## Behavior change to note
With the 5 s default `lock_timeout`, a boot-time migration or backfill
that waits >5 s on a lock now errors (surfacing in logs / crash-looping
the pod) instead of stalling silently. That is the intended
visible-failure-over-fleet-stall tradeoff; deployers with slow contended
migrations can set `BUZZ_DB_LOCK_TIMEOUT_MS=0`.
## Testing
- `cargo test -p buzz-db -p buzz-relay` — buzz-db green; buzz-relay has
9 failures that also fail on clean `main` in this environment
(api::admin/api::media/mesh_demo — unrelated, pre-existing).
- New config test covers override / `0`-passthrough / invalid-fallback
for all three env vars.
- Extended the existing `writer_pool_safety_hook_is_single_and_composed`
source-shape test so the timeouts can't drift out of the single
`after_connect` hook (SQLx replaces hooks — a second hook would silently
disarm the floor guard).
- `cargo fmt --check` and `cargo clippy --all-targets` clean for the
touched crates.
Closest existing PR/issue: none found.
---
**Update Aug 28, 17:06 EDT:** Rebased onto `main` at `a3730784fc` and
addressed the latest correctness review.
- Ported the timeout policy onto the refactored `buzz-db::runtime` pool
constructor and kept the shared env overlay for relay, admin, deletion,
and audit writers.
- Migration/schema-destruction connections now disable `lock_timeout`
and `statement_timeout` for their intentional long wait/DDL path. This
supersedes the earlier “Behavior change to note”: contended boot
migrations wait for the current migration owner rather than
crash-looping after five seconds.
- The audit worker now preserves and retries the same entry on
PostgreSQL `55P03` lock timeouts, using exponential backoff capped at
one second. Other database errors retain the existing terminal error
behavior, and retries emit `buzz_audit_log_lock_retries_total`.
- Added CI-backed PostgreSQL regressions for writer GUC
installation/migration exemption, audit-pool lock timeouts, and worker
recovery. The worker regression holds the real audit advisory lock past
`lock_timeout`, observes a retry, releases the lock, and proves the
original entry is appended exactly once.
Current verification supersedes the earlier testing notes: workspace
Rust clippy passed with warnings denied; all nine infrastructure-free
backend unit-test lanes passed; all three focused PostgreSQL regressions
passed against PostgreSQL 17; formatting, diff checks, file-size guards,
and desktop frontend checks passed. The Linux Blox workstation could not
run the unrelated Tauri native lane because `glib-2.0` is absent, so
that platform check is left to PR CI.
---
**Update Aug 31, 11:09 EDT:** Rebased onto current `main` at
`c3132c3ee9` and reran the requested audit-lock contention scenario on
Blox at head `896c3fe9ed`.
- `git range-diff` reports both PR commits unchanged by the rebase; the
branch remains two commits and the worktree is clean.
- `cargo fmt --all -- --check` and clippy with warnings denied passed
for `buzz-db`, `buzz-relay`, `buzz-admin`, and `buzz-deletion`.
- All three focused PostgreSQL 17 regressions passed: writer session
timeout/migration exemption, audit writer timeout bounds, and audit
worker recovery of the original entry exactly once.
- Live protocol verification used a head-built relay and CLI, native
PostgreSQL 17/Redis, `BUZZ_DB_LOCK_TIMEOUT_MS=300`, and an eight-second
hold on the community audit advisory lock. The real message was accepted
and persisted once while the lock was held; its audit-row count remained
zero during contention while retries accumulated. After release, exactly
one `event_created` audit row appeared and remained exactly one after an
additional two-second duplicate check. The run recorded nine
lock-timeout retries, zero audit failures, and event ID
`7f8c4ffae28e78555fcf2d56396d6e6c01b3712e5411288dc79e9a54af9d9444`.
Generated with Codex
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
This PR implements MVP, iOS-only,
[NIP-PL](8d2d0ff5ad/docs/nips/NIP-PL.md)-compliant
push notifications.
A relay with `BUZZ_PUSH_ENABLED` will send a push notification for any
message that appears in the in-app Notifications tab.
## Enrollment flow
The first time the client first connects to a relay with
`BUZZ_PUSH_ENABLED`:
```mermaid
sequenceDiagram
autonumber
participant App as Buzz iOS app
participant iOS
participant Relay as Buzz relay
participant Attest as Apple App Attest
participant Gateway as Push gateway
App->>Relay: Fetch NIP-11 push capability
Relay-->>App: Push profile, current relay public key, and limits
par
App->>iOS: Request notification permission
iOS-->>App: Permission result
and
App->>iOS: Register for remote notifications
iOS-->>App: Device token
end
App->>Gateway: Request installation challenge
Gateway-->>App: Single-use challenge
App->>Attest: Attest installation transcript
Attest-->>App: Attestation proof
App->>Gateway: Enroll device token and proof
Gateway-->>App: Installation handle
App->>Gateway: Request delegation challenge
Gateway-->>App: Single-use challenge
App->>Attest: Assert relay-key delegation
Attest-->>App: Assertion
App->>Gateway: Create delegation
Gateway-->>App: Opaque endpoint grant
App->>Relay: Publish encrypted push lease and filters
Relay-->>App: Lease acknowledged
```
## Push-time flow
When a notification-eligible event is received by the relay:
```mermaid
%%{init: {
"sequence": {
"actorMargin": 20,
"width": 110,
"messageMargin": 18,
"diagramMarginX": 8,
"wrap": true
}
}}%%
sequenceDiagram
autonumber
participant Relay as Buzz relay
participant Gateway as Push gateway
participant APNs as Apple Push<br/>Notification service
participant iOS
participant NSE as Notification service<br/>extension
Relay->>Gateway: POST /v1/deliveries/apns<br/>opaque endpoint grant, request ID, expiry, NIP-98 authorization
Gateway->>APNs: POST /3/device/{device-token}<br/>topic, request ID, expiry, constant mutable-content payload
APNs-->>Gateway: 200 OK: request accepted
Gateway-->>Relay: 200 OK: accepted status
APNs-->>iOS: Notification: constant reconnect alert<br/>mutable-content = 1
iOS->>NSE: Invoke extension<br/>original notification content
NSE->>Relay: POST /query: subscription filters, limit 10<br/>NIP-98 authorization
Relay-->>NSE: 200 OK: signed Nostr events<br/>kinds 9, 40002, 45001, or 45003
NSE->>iOS: Complete notification: title, body, subtitle<br/>thread ID, exact-message target
```
relay → push gateway → APNs -> NSE -> Notification Center
## Known limitations
The APNs wake payload is intentionally constant and opaque: it contains
no originating community or message identifier, in keeping with the
implemented NIP-PL privacy design.
The Notification Service Extension must therefore reconnect to the relay
and resolve eligible messages after each wake. Around overlapping wakes,
timing boundaries, or resolution windows, notification presentation may
occasionally omit an expected message or display a message more than
once.
This best-effort behavior is deliberately accepted for the current
implementation and will be measured during the internal rollout to
determine whether the user experience is acceptable before any broader
deployment; the implementation does not claim exactly-once presentation.
## Validation
Live end-to-end hardware validation used an internal remotely hosted
development relay and push gateway, the APNs sandbox, and a physical
iPhone 12 mini:
- A second real Buzz client published a uniquely marked message through
the hosted relay.
- The relay matched the message and sent the constant opaque wake
through the hosted gateway. The gateway made an actual APNs request; no
`simctl push` or simulated notification was used.
- The iPhone received the notification on its lock screen. The
Notification Service Extension reconnected to the relay, fetched the
event, verified its ID and signature, and replaced the placeholder
content with the real notification title and body.
- After the app populated its shared presentation cache, a final marked
notification visibly showed the sender display name, sender avatar, and
hashtag-prefixed channel name.
- Tapping a lock-screen notification opened Buzz and exercised the
notification-response path and navigated to the corresponding message.
Final validation with a dogfood-signed artifact and production App
Attest/APNs configuration remains a release step.
## Independent pre-reviews
- **First pass:**
[Carl](buzz://message?channel=18882f4c-289f-41db-942f-81f6f8066da1&id=74ab9a93bb227f3e762568f1cf9fee66d7495b0edc3918735ff787238b9cc585)
found missing transient retries, executor-key rotation suppression,
duplicate installation renewal, and an unauthenticated challenge write
amplifier. These were resolved by [retry-safe
bootstrap](https://github.com/block/buzz/pull/6269/changes/12c66ea62)
and [authenticated renewal plus a cross-replica
quota](https://github.com/block/buzz/pull/6269/changes/8e5ece0bd).
[sol-max](buzz://message?channel=ad83385f-8e9e-4461-9a35-c1bf2e208532&id=d26d53daa4684669e2ed354638241f13f36c3a97027fe8b4dd738aff09038962)
found delegation generation burning and an edited applied migration,
resolved by [exact-generation
revocation](https://github.com/block/buzz/pull/6269/changes/c26d2159d)
and a [forward-only
migration](https://github.com/block/buzz/pull/6269/changes/956c1d099).
[k3-max](buzz://message?channel=5e46055d-a766-4065-ae25-05d1e4aaa6b2&id=d43139138a0b15f806cbdbeeedd8f69d992cadf2e805876db6fdde6a34c7eda1)
found no blockers.
- **Exact-head re-review:**
[Carl](buzz://message?channel=18882f4c-289f-41db-942f-81f6f8066da1&id=a897721673459301b0cf26e8b85a1478d7ebbb56a4621f93d774c98d395b8f68),
[sol-max](buzz://message?channel=ad83385f-8e9e-4461-9a35-c1bf2e208532&id=fb2159f709ec68f74f7b21459acd76da0e8a7c5c0f3d469f99826b0cc2380849),
and
[k3-max](buzz://message?channel=5e46055d-a766-4065-ae25-05d1e4aaa6b2&id=2ce2842910435f562e9d9cc718595848f281b122c94605e523a4b964254b8bfb)
independently returned **NO BLOCKERS** at `7eb3a650b`; k3-max also
revalidated every remediation and the endpoint-specific App Attest
enrollment bound.
---------
Signed-off-by: Tom Brow <tomb@squareup.com>
Signed-off-by: Tom Brow <tomb@block.xyz>
Co-authored-by: Tom Brow <tomb@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Jordan Mecom <jm@squareup.com>
Adds authenticated, role-based moderation to the relay admin API. On
`main` the admin API is read-only and gated only by `Host`/`Origin`
matching; this branch adds NIP-98 authentication, a two-tier
Operator/Moderator principal model, mutation and staffing endpoints, and
NIP-11 auto-discovery so clients never type the admin URL by hand.
## Authentication (`BUZZ_ADMIN_AUTH`)
`BUZZ_ADMIN_AUTH` accepts `nip98` or `disabled`. Leaving it unset
defaults to `nip98` (fail-secure). Configuration fails closed: any other
value aborts startup, while a lingering `BUZZ_ADMIN_TOKEN` is ignored
with a startup warning — token (bearer) authentication is not supported.
`Host`/`Origin` matching is retained in every mode as defense-in-depth.
- **`nip98`** (default) — per-request signed NIP-98 (kind 27235) events,
resolved to an Operator or Moderator principal with per-person
attribution and individual revocability. Read-write per resolved
principal.
- **`disabled`** — no credential; relies entirely on network-layer
controls (reverse proxy, VPN, firewall) and logs a `WARN` on every boot.
Always read-only: `authorize()` resolves no principal, so mutation and
staffing routes always `403`.
## Roles
Buzz has two independent authority axes after this change.
**Relay-level** roles (new here) are deployment-global: they act across
every community on the relay, through the admin API. **Community-level**
roles (pre-existing, unchanged by this PR) are tenant-scoped: they act
inside one community, through signed Nostr moderation commands.
### Relay level (new)
| Role | Description |
|---|---|
| **Operator** | Full control of the deployment's moderation surface:
read all reports, feedback, and attachments across every community;
resolve reports with enforcement (`delete`/`kick`/`ban`/`timeout`) or
decisions (`dismiss`/`escalate`); reopen and cancel; update feedback
status; and manage the Operator/Moderator roster via the staffing
endpoints. |
| **Moderator** | Day-to-day triage: everything an Operator can do
except staffing — cannot view or change the roster. |
How a pubkey acquires a relay role (resolution order; config always
outranks DB):
1. Listed in `RELAY_OPERATOR_PUBKEYS` → **Operator** (source `config`)
2. Equals `RELAY_OWNER_PUBKEY` while `RELAY_OPERATOR_PUBKEYS` is empty →
**Operator** (source `owner_fallback`, a break-glass grant for
self-hosters that deactivates once any operator is configured)
3. Row in the `relay_operators` table → **Operator** or **Moderator**
(source `db`, managed via the staffing endpoints)
4. No match → `403`
### Community level (pre-existing, unchanged)
| Role | Description |
|---|---|
| **Owner** (community) | Full authority within their community: every
moderation action (delete, kick, ban/unban, timeout/untimeout, resolve
reports, view queue) plus member, role, and invite management. No guard
rails. |
| **Admin** (community) | Same community-wide moderation capabilities as
owner, except an admin cannot ban or time out the owner or a fellow
admin — only the owner may action an admin. Manages members and invites;
only the owner grants the admin role. |
| **Member** (community) | Standard participant; no moderation
capability. |
| **Owner / Admin** (channel) | Channel-local authority only: delete
messages and kick users within their own channel. |
| **Member / Guest / Bot** (channel) | No moderation authority. |
There is no community-level Moderator tier in v1; relay-level Moderator
is the only role by that name.
## Escalation scoping
The operator report queue is an escalation backstop, not the community's
day-to-day triage surface (per `VISION_MODERATION`, the severe class is
the platform's to review rather than the community's). Two rules enforce
that:
- **Escalated-by-default listing.** `GET /reports` with no `status`
parameter returns only `escalated` reports. An explicit
`status=<open|resolved|dismissed|escalated>` filter is always honored as
given, and full visibility across every status stays available for
platform-safety and legal review via `scope=all` (which lists reports
regardless of status). `scope` accepts only `all` and is ignored when an
explicit `status` is present.
- **Auto-escalated `illegal` reports.** Member reports whose category is
`illegal` are ingested with `status=escalated` rather than `open`, so
the severe class reaches the operator backstop without waiting for a
community admin to forward it. Every other category still lands `open`.
Auto-escalation only sets the queue status — it records no moderator
decision and stamps no resolver, so an auto-escalated report is
indistinguishable downstream from an admin-escalated one: the reopen
route returns it to `open` on the same terms, keyed only on status,
never on how the report became escalated.
## Principal resolution and NIP-98 admission
`resolve_admin_principal()` returns `AdminPrincipal { pubkey, role,
source }` per the resolution order above; `None` never falls through as
a role. Admission is ordered so the replay guard is a privilege, not a
public surface: signature/URL/method/payload-hash verification first,
roster check second, and only then is the deployment-scoped replay id
atomically consumed — a validly-signing but unrostered key never
allocates a replay slot. Redis failure fails closed.
## Report resolution, recovery, and enforcement provenance
`POST /reports/{id}/resolve` is a crash-safe enforcement state machine:
decision-only outcomes (`dismiss`/`escalate`) are a single
CAS-plus-audit transaction; enforcement
(`delete`/`kick`/`ban`/`timeout`) claims the report
(`open`→`processing`), runs the durable mutation, then finalizes — a
re-drive resumes at the step marker and converges to exactly-one
enforcement, fenced by a lease and an outbox claim token.
Person-directed enforcement on an `event`-kind report derives its target
from the stored event's author (server-owned truth, never the reporter's
`p` tag) via a single `derive_enforcement_target` shared by the HTTP
driver and the recovery worker. If the reported event was purged before
its author could be read, person-directed actions are rejected pre-claim
and the report stays `open`; `delete` needs only the event id and is
exempt.
`GET /reports/{id}` and the resolve response carry an `activeAction`
field surfacing the enforcement that actually executed — a report
dismissed after a reopen still reports the ban that ran. `POST
/reports/{id}/reopen` returns a terminal report to `open` (idempotent on
`requestId`). `POST /reports/{id}/cancel` is the sole recovery path for
a pre-mutation `failed` action, attributed via
`relay_admin_actions.cancelled_by`.
## Feedback
`GET /feedback` and `/feedback/{id}` survive a tenant purge: provenance
columns are severed to `NULL` rather than cascade-deleted, and the
attachment path fails closed to `404` on a severed row. `PATCH
/feedback/{id}` updates lifecycle `status`
(`new`/`reviewed`/`archived`).
## Staffing and probe
`GET/PUT/DELETE /operators/{pubkey}` are Operator-only; mutating a
config-backed pubkey returns `409 Conflict`. `GET /operators` returns
the union of config and DB principals with per-entry `source`. `GET
/probe` reports auth mode, role, source, `canAct`, and `canStaff` for
the desktop console.
## NIP-11 auto-discovery
The NIP-11 relay-information document gains an optional `admin_api`
field carrying the canonical admin origin (`scheme://host[:port]`, no
path), present iff `BUZZ_ADMIN_HOST` is set and omitted otherwise. The
scheme follows the same loopback rule as NIP-98 `u`-tag verification via
a shared `scheme_for_host` helper, so the advertised origin and the
origin the relay verifies against can never diverge.
## Operator API origin decoupling
`RELAY_OPERATOR_API_ORIGIN` is no longer required at boot when
`RELAY_OPERATOR_PUBKEYS` is set — it is used only by the
community-provisioning endpoints, which fail closed at request time
(with a boot-time `WARN`) until it is set. The admin console needs no
origin.
## Admin-web adaptation
The standalone `admin-web` dashboard signs each request as a NIP-98
event via a NIP-07 browser extension, discovers the auth mode with a
single unauthenticated probe (`200` → `disabled`, anything else →
`nip98`, fail-secure), and carries no token entry surface. Playwright
coverage exercises the NIP-98 and CSP paths.
## Security hardening
Three findings from security review are folded in:
- **Append-only roster audit.** `PUT`/`DELETE /operators/{pubkey}`
mutate the deployment-wide root of trust, but the upsert overwrites
`role`/`added_by` in place and the delete removes the only row — so a
grant→revoke sequence left no trace of who was ever granted or by whom.
Each mutation now writes an `relay_operator_audit` row (actor, target,
`grant`/`revoke`, pre-image `prev_role`, `new_role`, timestamp) inside
the same transaction as the mutation. A per-target transaction-scoped
advisory lock serializes concurrent mutations of the same pubkey before
the pre-image read, so the recorded `prev_role` is always the true
predecessor even under a concurrent-grant race. Chronology is keyed on a
`BIGINT GENERATED ALWAYS AS IDENTITY` `seq` column, not the wall clock:
the serializing lock guarantees insertion order and `seq` captures it,
so ordered reads (`ORDER BY seq`) follow the true privilege chain even
across a backward NTP step that a `clock_timestamp()` ordering would
invert. `created_at` (`clock_timestamp()`) is retained as informational
occurrence time only. Append-only by construction — no `UPDATE`/`DELETE`
path and no API surface. A no-op delete writes nothing.
- **`expirationSecs` overflow.** The timeout path built `Utc::now() +
Duration::seconds(secs as i64)` from an attacker-controlled `u64`:
`i64::MAX` panicked the handler, and a wrapped-negative magnitude minted
a *past* expiry that still passed validation. `compute_timeout_until`
now rejects zero, rejects magnitudes above a documented
`MAX_TIMEOUT_SECS` (365 days), and uses checked
`try_seconds`/`checked_add_signed` so no input can panic or produce a
past expiry — over-cap, zero, `i64::MAX`, and wrapping-negative inputs
all return a clean `4xx`.
- **Uppercase-hex config-backed bypass.** Config pubkeys are lowercased
at parse, but the `409` immutability check raw-string-compared the path
param while `decode_hex_pubkey` accepted uppercase — so `PUT
/operators/{UPPERCASE}` skipped the guard and wrote a shadow row for the
same 32 bytes. The validated param is now canonicalized (lowercased)
before the `409` check, DB write, `DELETE`, and response body.
## Migrations
- `0035_relay_operators.sql` — `relay_operators` roster table
(deployment-global), `actor_authority` on `moderation_actions`,
`processing` status plus `active_action_id` on `moderation_reports`,
`status` on `product_feedback`.
- `0036_relay_admin_actions.sql` — enforcement-action table with a
`request_id` idempotency key, a `step_marker` for crash recovery, and a
`cancelled_by` attribution column.
- `0037_relay_admin_action_lease.sql` — lease fencing for the action
worker.
- `0038_relay_admin_outbox_claim_token.sql` — fenced claim token on the
outbox worker.
- `0039_relay_operator_audit.sql` — append-only `relay_operator_audit`
trail for roster mutations (see Security hardening).
`docs/admin/README.md` documents the full principal model, NIP-98 event
requirements, capabilities by role, the startup error matrix, and the
discovery field.
## Production blast radius
A relay without `BUZZ_ADMIN_HOST` is completely unaffected — the admin
surface stays disabled and `BUZZ_ADMIN_AUTH` is ignored; a lingering
`BUZZ_ADMIN_TOKEN` logs a startup warning and must be removed. Where
`BUZZ_ADMIN_HOST` **is** set, unset `BUZZ_ADMIN_AUTH` defaults to
`nip98` (per-person signed auth); `BUZZ_ADMIN_AUTH=disabled` reproduces
`main`'s prior `Host`/`Origin`-only gating but is read-only (mutation
routes `403`). The five migrations add tables and columns without
touching existing data.
---
Related: [block/buzz#4768](https://github.com/block/buzz/pull/4768)
(desktop admin console consuming the `admin_api` field),
[squareup/bb-public#339](https://github.com/squareup/bb-public/pull/339)
(Phase 4 rollout config)
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
## Summary
Add Unity Catalog model-service discovery to `buzz-agent` Databricks v2,
preserve full Unity Catalog FQNs through ACP model discovery, route
model services through MLflow chat, and add the optional
`DATABRICKS_MODEL_FILTER` visibility filter.
The resilience follow-up bounds each catalog attempt across headers,
bounded body reads, and JSON parsing; retries transient 499/5xx,
transport/body failures, malformed JSON, and attempt timeouts; and emits
only safe catalog/status diagnostics.
## Validation
The live validation and test gates below were run on the pre-rewrite tip
`1cf6c66e7c5bf9a00c357bda24df48e3beeb310b`. The rewritten tip
`0364cf44f3eb5cfbd5771b0a0dd7868ce0ad2c19` has identical trees for both
commits, so the attribution rewrite changes no tested code.
- 31 catalog tests passed
- 19 Databricks OAuth/routing integration tests passed
- `cargo check -p buzz-agent` passed
- `cargo clippy -p buzz-agent --all-targets -- -D warnings` passed
- `cargo fmt --check` passed
- `git diff --check` passed
- Desktop Tauri validation passed through the repository-supported setup
path, including 2,777 library tests and auxiliary terminal suites
The full `buzz-agent` package has one separately reproduced pre-existing
failure: `regressions::cancel_kills_inflight_tool_via_mcp_notification`,
failing on both baseline `eb8e97cfe` and the follow-up worktree. No
Databricks path reaches that test. The live validation proves discovery
and ACP exposure, not inference against every listed model.
---------
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Co-authored-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
## Summary
- Supersedes #1913 with a KLIPY-hosted URL implementation.
- Adds KLIPY GIF search and trending results to desktop message and
forum composers.
- Keeps selected GIFs hosted by KLIPY; Buzz stores only the external URL
and media metadata (no imeta tag, since relays only accept hash-backed
local `/media/` entries).
- Aligns the Emoji/GIF picker with Buzz's standard segmented control,
theme surfaces, and motion behavior.
## Relay-to-provider boundary
- The relay proxies KLIPY search/share so the `BUZZ_KLIPY_API_KEY` never
reaches the desktop; the key stays server-side behind a redacted `Debug`
impl.
- The dedicated GIF `reqwest` client sets `redirect::Policy::none()`.
Because the API key rides in the request path, following a provider
`3xx` could replay a key-bearing URL to an attacker-chosen host (an
SSRF/key-disclosure primitive). With redirects disabled, a `3xx` returns
as a non-success status that the handlers map to a generic `502`; the
`Location` target is never read or forwarded.
- Admission reuses the established NIP-98, membership, replay, and
per-pubkey rate-limit gates, with an upstream response-size cap and
allowlisting so KLIPY error bodies never cross the relay boundary.
## Accessibility
- Under `prefers-reduced-motion: reduce`, the picker grid renders a
static provider poster (a normalized `jpg` asset) instead of the
animated preview, or a named static placeholder when no poster is
available. It reacts to preference changes while mounted.
`no-preference` keeps the animated preview.
- Selected GIFs carry their title through `ImetaMedia`'s `displayLabel`,
so the composer thumbnail, preview dialog, editor, lightbox, and remove
control all derive one non-empty accessible name instead of an empty
`Attachment ` label. Ordinary hashed uploads keep their existing
hash-derived names.
---------
Signed-off-by: kenny lopez <klopez4212@gmail.com>
Signed-off-by: Kenny Lopez <klopez4212@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
Co-authored-by: Princess Donut <b238ea756dee4d98afa5883fc7f1de61eeabe65bf700e3a5a5a80db5e42e2c2b@buzz.block.builderlab.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
Require `BUZZ_RELAY_PRIVATE_KEY` on every relay startup and remove the
shared public fallback key completely.
For local development, `just bootstrap` now generates a random key once
in the gitignored `.env` file. Re-running bootstrap preserves that key,
so relay-authored events retain the same identity across restarts. The
local relay recipes reload `.env`, while isolated CI and E2E launchers
pass explicit per-run keys.
Startup fails immediately when the key is missing or invalid, before
connecting to Postgres or Redis. Deployed relays continue to receive
their stable key from the existing chart-managed secret.
Validated with:
- `./scripts/test-ensure-local-relay-key.sh`
- `cargo test -p buzz-relay --bin buzz-relay`
- `cargo clippy -p buzz-relay --bin buzz-relay -- -D warnings`
- `cargo fmt --all -- --check`
- `shellcheck scripts/ensure-local-relay-key.sh
scripts/test-ensure-local-relay-key.sh`
Signed-off-by: Jordan Mecom <jm@squareup.com>
This change requires a valid signed Blossom authorization request and
current relay membership for every media GET and HEAD request. It
removes the unauthenticated compatibility path and updates desktop reads
to send the required authorization.
This blocks anonymous retrieval and access after relay-membership
revocation. It does not yet bind a blob to its originating channel, so
someone removed from a private channel can still read a known blob while
remaining a relay member. That channel-ACL follow-up remains required
before closing the full finding.
## Testing
- `git diff --check origin/main...codex/security-media-read-auth`
- Rebased onto `origin/main` at `5c98932`
- Full CI pending
Originating Buzz thread:
`buzz://message?channel=3928fe05-df61-4b5d-b9c7-d623b9b10ea1&id=3c6c02312f763fbe0d2bfc33a6c1a362f91d0354f3d18b039cf7a0558c1439d1`
---------
Signed-off-by: Jordan Mecom <jm@squareup.com>
Signed-off-by: Alex Rosenzweig <arosenzweig@squareup.com>
Signed-off-by: Eli Foster <efoster@squareup.com>
Co-authored-by: Eli Foster <efoster@squareup.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
## Summary
- add one strict `BUZZ_S3_ADDRESSING_STYLE=path|virtual` setting shared
by media and Git/CAS storage
- preserve path-style defaults for bundled Compose/Helm MinIO while
supporting Railway's virtual-hosted bucket contract
- fail startup on invalid or non-Unicode values before dependency
connection, and validate the Helm value with the same two choices
- document operator mappings and why endpoint and bucket remain separate
for routing and SigV4 signing
## Best-practice rationale
AWS documents both URL forms and favors virtual-hosted addressing for
S3, while compatibility endpoints such as the bundled MinIO deployment
can require path style. `rust-s3` defaults to virtual/subdomain
addressing and provides `with_path_style()` for the explicit
compatibility case.
Some providers buckets only support as virtual-hosted bucket styles.
This PR therefore uses one explicit, provider-neutral switch rather than
endpoint heuristics or fallback behavior, while retaining `path` as
Buzz's backward-compatible default.
Sources:
-
https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html
- https://docs.rs/rust-s3/0.37.0/s3/bucket/struct.Bucket.html
- https://docs.railway.com/storage-buckets#url-style
-
https://github.com/minio/minio/blob/master/docs/config/README.md#domain
## Validation
- `cargo fmt --all`
- `cargo check --workspace --all-targets`
- targeted `buzz-media` and `buzz-relay` parsing/client-construction
tests for defaults, strict errors, and both URL styles
- Helm unittest: 45/45 passed
- Compose config/render validation passed
- local MinIO path-mode relay startup passed the Git A3 conformance
probe and became ready
- unreachable object storage failed startup and readiness never opened
- push hooks completed the broader Rust and desktop suites successfully
---------
Signed-off-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Co-authored-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
## Summary
- Raise the relay's Postgres pool cap from the `buzz-db` default of 20
to 50 per pool, and expose `BUZZ_DB_POOL_SIZE` for per-deploy tuning
- Applies to the writer pool and, when `READ_DATABASE_URL` is set, the
reader pool; zero/unparsable values fall back to the default
- The `buzz-db` library default is unchanged — only the relay opts into
the larger cap
## Why
During the 2026-07-27 18:40–19:05Z traffic burst on bb-public, per-pod
PG pools pinned at 20 fleet-wide and ~380 requests failed on the 3s
acquire timeout — membership checks, channel access lookups, and
historical queries returning errors to users. The database was nowhere
near a limit: Aurora (db.r8g.8xlarge, ~5,000 max connections) sat at 19%
CPU, 201 connections (~4% of capacity), commit latency flat at 0.01ms.
The 20-connection default was sized for "four relay pods against PG
max_connections=100" (the comment in `buzz-db` says exactly that).
Production now runs 12–15 pods against Aurora — the per-pod cap is the
binding constraint, not the DB.
Budget at the new default: 15 pods × (50 writer + 50 reader + 5 audit) ≈
1,575 potential connections, ~30% of Aurora's ceiling — and actual usage
stays demand-driven (`min_connections` stays 2, connections only open
under load).
Same shape as #2521 (`BUZZ_REDIS_POOL_SIZE`), which fixed the identical
class of ceiling on the Redis side.
## Testing
- `cargo test -p buzz-relay`: 762 passed, 1 failed — the lone red is
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`, the
known pre-existing flake; it fails identically on clean `main` at the
same SHA (verified via `git stash` / rerun)
- New test `db_pool_size_env_override_and_invalid_fallback` covers
override, zero, and unparsable fallback
- `defaults_are_valid` extended to pin the new default
- `cargo clippy -p buzz-relay --all-targets -- -D warnings` and `cargo
fmt --check` clean
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Configure a dedicated datastore tracing target on the OTLP layer while
preserving explicit logging filters and avoiding span overhead when OTLP
is disabled.
This is in preparation for adding trace spans for datastores used in
Buzz
## Update — 2026-07-27
- Export HTTP requests as `INFO` server spans under `buzz_relay`,
preserving request parentage for datastore spans.
- Configure OTEL span filtering independently with `BUZZ_OTEL_FILTER`,
so `RUST_LOG` changes cannot break trace topology.
- Verify exported HTTP and datastore spans share a trace ID and have the
expected parent/child relationship.
Co-authored-by: Amp <amp@ampcode.com>
