mirror of
https://github.com/block/buzz.git
synced 2026-11-07 01:41:06 +00:00
0ca5a044473cd3d6faf30f0ec57ccbbcf2f51cab
302 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Brad Seiler
|
a5e5661010 |
test(relay): import root fixture shared ownership type
Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
8382528486 |
test(relay): retain root lifecycle registration during revalidation
Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
1b1f5a23f4 |
test(relay): bind durable revocation to socket admission
Authenticate through root and audio handlers before a durable-only ban. Removing either production enrollment call must fail its regression. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
seiler
|
4aaf747ed2 |
refactor(relay): address revalidation review
Restore CommunityDisconnectReason near the top of state.rs and document scan deadline and reconnect-load limitations. Authorization behavior is unchanged. Signed-off-by: seiler <seiler@squareup.com> Co-authored-by: Codex <noreply@openai.com> (cherry picked from commit a4d1ab288d761d44eba25cc62d25b7d2858788dc) Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
seiler
|
1b81b01b01 |
Add durable live member authorization checks
Signed-off-by: seiler <seiler@squareup.com> Co-authored-by: Codex <noreply@openai.com> (cherry picked from commit 92f13229ed3038d63f2a2d9c889bfd4c212e38e2) Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
a1f79e3aba |
feat(relay): persist NIP-FI upgrades (BUZZ-510) (#8985)
## Summary Record that a community requires NIP-FI and provide an idempotent operator API to enable it and recover the original receipt. Acceptance checks the current owner, host, lifecycle and existing issuer trust in one transaction. It never creates trust. Receipts survive community deletion so an operation UUID cannot be reused. New acceptances default to disabled through `BUZZ_NIP_FI_UPGRADES_ENABLED`. Disabled calls return HTTP 503 `upgrades_disabled`; identical retries and authenticated readback still recover existing receipts. Policy is permanent: stopping new calls or rolling back code does not clear it. Last-issuer removal is refused for required communities, and bulk backfill leaves missing bindings for explicit operator repair. Apply additive migration 0082 before code. Existing communities remain optional. This layer records acceptance and reports only `pending` or `unavailable`; deploy and verify [per-request enforcement](https://github.com/block/buzz/pull/8988) and [session activation](https://github.com/block/buzz/pull/8991) before enabling acceptance. The global enforce floor remains. See [the operator contract](https://github.com/block/buzz/blob/seiler/BUZZ-510/relay-auth-upgrade/docs/community-auth-upgrades.md) for signing, recovery and containment. ### Related issue [BUZZ-510](https://linear.app/squareup/issue/BUZZ-510/relay-persist-community-nip-fi-requirements-and-expose-idempotent). Base of native stack `8989`; retains trunk's enforcement telemetry from [#8974](https://github.com/block/buzz/pull/8974). Includes a separate commit that repairs an observer timing flake in a desktop test required by the repository gate. ### Testing Controlled local regressions detect the old disabled error code and removal of the last issuer with `allow_last=true`; intended source is restored. Full PostgreSQL validation and the updated live relay/admin exercise remain pending. Human acceptance remains open. No deployment or production activation. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz> |
||
|
Luke Tornquist
|
4813b8cd15 |
relay: time every WebSocket read and every stored EVENT outcome (#9052)
## Summary
Two gaps in relay telemetry: WebSocket reads (`REQ`, NIP-AR sync,
`COUNT`) have no latency or outcome metric, and
`buzz_event_processing_seconds` only measures stored EVENTs that
succeed.
**What this does**
- **`buzz_ws_read_seconds{type, outcome, reason}`** (new histogram).
It's recorded from `access::ReadLog::drop`. Every WebSocket read already
finishes through that one seam, whether it ends `served`, `denied`,
`error`, or `aborted` (future dropped mid-read), so this adds no new
exit paths and changes no handler. Duration runs from the moment the
handler takes the frame until `EOSE` is queued (`REQ`), the initial set
ends (sync), or the total is sent (`COUNT`). `reason` is the existing
fixed denial/error code (`query_timed_out`, `database_error`,
`not_channel_member`, …), or `none`. The histogram's `_count` doubles as
the outcome counter, so one metric gives both read latency and read
failure rate.
- It's recorded *before* the refusal budget check, because the budget
only exists to suppress log lines. When `refused_read` is over budget,
it records the sample without building the log line, so its cost under a
refusal flood stays the same.
- **`buzz_event_processing_seconds{outcome}`**. Before this change it
was recorded only when `ingest_event` returned `Ok`. A store that was
slow *and failing* (statement timeouts, pool exhaustion) produced no
samples, so latency looked better while the store was degrading. It's
now recorded on every persistent-ingest outcome: `accepted`,
`not_accepted` (for example a duplicate), `invalid`, `auth`, `error`.
Queries that sum across all outcomes still work.
**Cardinality.** Every label value is a `&'static str` from a
code-defined set: 3 types × 4 outcomes, with about 25 reasons that only
appear alongside `denied`/`error`. On purpose, there's no `kind` label:
`bounded_kind_label` still admits around 200 values, which is too many
to cross with histogram buckets.
**What got simpler:** `ReadOutcome::parts()` replaces the inline
outcome→label match in `Drop`, so the log line and the metric share one
mapping. There's no new instrumentation seam: the read metric reuses the
access log's lifecycle.
### Related issue
None found. This came out of a monitoring-gap review: without these
signals, alerts can't catch a slow or failing WebSocket read path, or a
failing EVENT store.
### Testing
-
`access::tests::every_read_is_measured_even_when_its_line_is_suppressed`
(unit). It drives the real `ReadLog` constructors and `Drop`: 20
admission refusals against a 16-line budget, a handler denial past the
budget, a `COUNT` error, a served sync and an aborted `REQ`. It asserts
the exact `(type, outcome, reason, samples)` rows. Two mutants each fail
it: removing the over-budget branch in `refused_read`, and moving the
measurement after the budget check in `Drop`.
-
`handlers::event::tests::postgres_tests::event_processing_is_timed_for_accepted_and_refused_ingest`
(Postgres lane). It sends one accepted text note, then a refused one
from a cached-ban sender, through `handle_event`, and asserts one sample
each under `accepted` and `auth` (a ban maps to the `auth` class).
Removing the `Err`-arm measurement fails it.
- Blox gate at `b314df43f2` (tree `4ca9ebc1`): `cargo fmt --check`
clean. The new Postgres test passes, and removing the `Err`-arm
measurement fails it. buzz-relay unit lane (`cargo nextest run -p
buzz-relay --lib --bins`): 1708/1708 passed.
- At `539796bd51` (the same code; only the second commit's test
assertion differs): workspace clippy (`--all-targets -D warnings`)
passed. The new unit test passed, and each of its two mutants failed it.
buzz-relay Postgres lane: 604/605 passed. The one failure was this
test's old `invalid` expectation, which `b314df43f2` corrects.
- CI at `b314df43f2`: both new tests appear in the CI logs as passing
(Rust / Unit Tests, PostgreSQL Domain / PostgreSQL Tests).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
John Matthew Tennant
|
f9ef721e68 |
fix(media): keep transparency in image thumbnails with WebP (#9003)
## Summary
This fixes transparent images showing a black background. Image
thumbnails were always JPEG, which can't store transparency, so a
transparent PNG or GIF upload, such as an agent avatar, rendered on
black wherever a client showed its thumbnail.
The relay now writes a lossy WebP thumbnail at `{sha256}.thumb.webp`,
which keeps transparency, alongside the existing `{sha256}.thumb.jpg`:
- **Encoding:** the WebP comes from libwebp through the `webp` crate,
built from vendored C source, so the relay needs no system library. A
GIF uses its first frame.
- **Upload responses:** the upload descriptor's `thumb`, and its
per-tenant rewrite, point to the WebP.
- **JPEG kept:** installed clients derive `.thumb.jpg` themselves, so
the relay keeps writing it until clients move to WebP with a JPEG
fallback. Existing images still have only their JPEG thumbnail.
- **Serving:** `/media/` serves both variants, each with its own content
type. Other `.thumb.*` extensions still return 404.
- **imeta:** a `thumb` may name either variant, and the relay checks
that the named object exists.
- **Storage sweep:** the bucket taxonomy counts both variants as the
image's thumbnail bytes instead of classifying `.thumb.webp` as unknown.
| 320px thumbnail | JPEG | PNG | lossy WebP (q80) |
|---|---|---|---|
| Transparent avatar (starter-team `fizz.png`) | 11.3 KB, black
background | 65.1 KB | 13.8 KB |
| Opaque avatar (`sprout-agent.png`) | 23.6 KB | 156.5 KB | 12.3 KB |
| Screenshot (`channel-thread.png`) | 13.4 KB | 62.5 KB | 6.4 KB |
Encoding a WebP thumbnail takes 3 to 20 ms.
### Related issue
Supersedes #8772, which served a PNG at the `.thumb.jpg` path.
### Testing
Red/green against a local relay on `:3000`, built from the branch under
test, with Postgres, Redis and MinIO from `docker compose`:
```bash
cargo test -p buzz-test-client --test e2e_media test_transparent_png_thumbnail_is_webp_with_alpha -- --ignored
cargo test -p buzz-test-client --test e2e_media_extended test_ws_imeta_webp_thumb -- --ignored
```
- **On `main`:** both tests fail. The descriptor's `thumb` is
`.thumb.jpg`, and the WebSocket send is rejected with `invalid: imeta
thumb must be a local .thumb.jpg path`.
- **On this branch:** both tests pass:
- Uploading a half-transparent PNG returns a `thumb` ending
`.thumb.webp`.
- HEAD and GET on that URL return `Content-Type: image/webp`.
- The decoded thumbnail keeps alpha 0 on the transparent half and alpha
255 on the opaque half.
- `.thumb.jpg` still returns `image/jpeg`.
- A kind 9 message whose imeta `thumb` names the `.thumb.webp` is
accepted.
## Screenshots
The starter-team `fizz.png` avatar uploaded to a relay built from this
branch. Both thumbnails are shown as round avatars beside the original
PNG. `.thumb.jpg` fills the transparent area with a solid disc, and
`.thumb.webp` keeps the transparency.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: John Matthew Tennant <jtennant@squareup.com>
Co-authored-by: Simone <0828be588d5c9c4d092c5b1cf01761d5aef63d6b104807fd8e8284d517dd1a99@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Luke Tornquist
|
d21f0c0fdb |
perf(relay): keep 1 s read caches warm when the pool is saturated (#9012)
## What The relay's 1-second read caches (restriction state, tenant host, open channels) now date each entry from when the read checked out its pool connection, not from when the cache miss began. Concurrent misses on the restriction and tenant-host caches now share one database read. The open-channels cache already did this. ## Why Each of these caches serves an entry for 1 s after "the read that produced it began". Today that start time is taken before waiting for a pool connection. Once pool waits reach about a second (for example, many clients reconnecting at once), every entry is already expired by the time it lands. The cache hit rate falls toward zero exactly when the pool is saturated, so every request does its own read. Those reads queue for the same connections, which stretches the waits further. Requests that time out in the pool then fail closed (`internal error checking restriction state`, `community active check failed; refusing the socket`), and their clients reconnect and add more load. In a local synthetic reconnect load test with a 50-connection pool: - the restriction cache served fewer than one request in five from cache; - the tenant-host cache, which has a single key for the whole deployment, missed more often than it hit; - the open-channels cache missed hundreds of thousands of times on expired entries. ## How - **Stamp at checkout.** `buzz_db::ReadAt<T>` carries a value plus the instant its connection was checked out. The statement's snapshot is no older than that instant, so serving the value for 1 s after it keeps the documented staleness bound, however long the read queued. Three `*_read_at` methods return it: `moderation_restriction_state_read_at`, `lookup_community_by_host_read_at` and `get_listed_open_channel_ids_read_at`. The existing methods now call these and drop the value's timestamp. Each operation keeps one datastore span under its existing name, carried by the `*_read_at` method. - **Share concurrent misses.** `SharedReads<K>` is the per-key gate the open-channels cache already used, moved into its own module. The open-channels cache, the restriction cache and the tenant-host cache now all use it. The first miss reads; the others wait for it, then recheck the cache. A waiter waits for at most the writer pool's acquire timeout plus `SHARED_READ_STATEMENT_WAIT` (1 s). That covers the leader's own wait for a connection, which is what grows in a storm, so waiters keep sharing its read instead of queueing duplicate reads behind it. A failed or cancelled leader releases its waiters at once. A leader whose statement hangs stalls them for at most 1 s past the acquire timeout, never the statement timeout. - **Reject hosts no community can have.** `bind_community` rejects a host longer than `communities.host VARCHAR(255)` can hold (`buzz_core::tenant::MAX_HOST_LEN`, counted in characters) as unmapped, next to the empty-host check. An unauthenticated caller's arbitrarily long host therefore never reaches a resolver, cache, gate or the database. No staleness bound changes: every entry is still served for at most 1 s after its read's snapshot. Invalidation paths are unchanged. A read in flight at an invalidation can still refill the old row for at most that bound, as documented today. ## What got simpler The open-channels gate's hand-rolled `try_lock`/`timeout` block is replaced by `SharedReads::turn`. One gate implementation now serves three caches instead of one implementation that only one cache had. The gate's wait is set once per gate rather than passed at each call site. Provisioning and host binding share one `MAX_HOST_LEN` instead of a private constant. ## Tests - `shared_reads::tests`: one caller reads while the rest wait; a waiter gives up on a hung read after its bound. - `concurrent_restriction_and_host_misses_share_the_in_flight_read`: waiters on both caches use the leader's entry; with an unreachable database, any waiter that read on its own would fail. - `postgres_tests::cache_misses_behind_a_held_pool_share_one_fresh_read`: with a one-connection pool held for longer than every max age, eight concurrent misses on each of the three caches cost one database read, and each entry still lands fresh. - `tenant::tests::an_overlong_host_fails_closed_before_the_resolver`: a host over the limit fails as unmapped with no gate entry and no lookup; a host at the limit, including one in multibyte characters, is still looked up. - Mutation-checked: each of the following turns these tests red: stamping any of the three entries before the pool wait, dropping either new waiter path, letting waiters give up after the cache's max age, dropping the host length check, or counting the host length in bytes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Kalvin C
|
3d26ed7dfd |
feat(relay): match verified names in profile search (#9019)
## Summary `POST /profiles/search` only matches kind:0 `name` and `display_name`. In communities that publish NIP-FI verified profiles, a person's surname or directory username often appears only in their relay-signed kind:30382, so typeahead cannot find them by it. - Where the community advertises verified profiles, the search also matches the `display_name`, `username` and `full_name` tags of each current kind:30382 head signed by the relay key. - Verified names need a trailing prefix of at least three characters, as in NIP-50, so a short prefix doesn't make the relay rank every verified head. kind:0 keeps one-character typeahead. Lookups from heads to `events` also match `created_at`, so Postgres prunes to one partition. - Results are paged by person. Each person returns their current kind:0, when they have one, then their current kind:30382, when they have one, whichever matched. A person is ranked by their best match across both sources, using the existing tiers. - NIP-11 `federated_identity.public_profile` gains `searchable: true`, so clients can rely on the response instead of a separate kind:30382 read. - Migration `0081` adds a GIN expression index over the head's name tags on `nip_fi_profile_heads` (built-in `jsonb_path_query_first` and `to_tsvector`, no new functions). The table holds one row per verified person. Like `0078`, the migration builds the index only while the table is empty; populated databases prebuild it with `CREATE INDEX CONCURRENTLY` (`docs/profile-search.md`), and the migration then checks it is valid and has the expected definition. Compatibility: the request schema is unchanged. Existing clients that keep only kind:0 events work as before and also find people who match only by verified name. Trust stays pinned to the relay `self` key; the event shape follows NIP-85 user assertions, but discovery is relay-advertised rather than kind:10040. ## Testing - `buzz-search` Postgres integration: new `profile_names_match_verified_names` covers forged-key heads, replaced heads, other communities, people without kind:0, per-person grouping, paging, and words split across sources; the existing profile-name tests pass unchanged. The test fails when the author filter is inverted. - `buzz-relay` end to end in enforce mode: `profile_search_matches_verified_names_where_advertised` publishes through the real writer and finds the person by surname only where verified profiles are advertised. - Desired-state and migration index definitions match. - `profile_names_use_verified_name_index` asserts the search scans the new index. Migration `0081` checked against an empty table, a rerun, a populated table without the index (raises), the documented concurrent prebuild, and an index with a different definition (raises). - Latency, 50k people each with a kind:0 and a verified head, release build, local Postgres, best of 5, 30 people per page: 11–22 ms with verified names versus 1–8 ms kind:0 only. `EXPLAIN` shows a bitmap scan on the new index. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kalvin Chau <kalvin@block.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Will Pfleger
|
0a06c0b3cb |
feat(relay): log reads, logins and connections on access target (#8376)
🤖 Adds an access log for relay reads, logins, connections and HTTP requests on the dedicated `buzz_relay::access` target. It writes to stdout, not `audit_log`, so reads never take the audit advisory lock. The log is shaped for volume: one line per HTTP request, one line per WebSocket read, and one line per WebSocket connection. ## Lines per operation | Operation | Lines | |---|---| | HTTP request on a matched route | 1 (`http`), except the skip list below | | HTTP request matching no route | 1 (`http`, `route="<unmatched>"`) at status 400 and above, otherwise none | | WebSocket connection | 1 (`close`, carrying the login) | | WebSocket `REQ` / `COUNT` / NIP-AR sync | 1 each (`req` / `count` / `sync`), plus the `buzz_ws_reads_total` counter | | WebSocket NIP-42 login | 1 (`auth`) per decided or cut-off attempt, capped after the decision as below | | Audio login | 1 (`auth`, at its final admission check) | | WebSocket `EVENT` | 1 (`Event ingested`, not on the access target) | ## HTTP requests One middleware, `http_access_log`, writes every HTTP line. It sits between `track_metrics` and `http_trace_layer`, so rebuilt errors, body-limit refusals and localhost refusals all reach it. It writes the line when the response is ready, which is not proof the body was delivered. A request dropped before then logs `request=aborted` at WARN with no `status`. - Every line carries `route` (the matched route template, never the raw path), `peer_addr` when the listener has one, `status`, `time_to_response_ms` and `http_method` (the HTTP verb). The verb stays on the line itself because the `http.request` span is absent under an access-only filter and when a pending request is dropped. - Handlers record what they learn on the request's `AccessSlot`: the login (`method`, `pubkey`, `pubkey_verified`, `community`), the decision, the filter summary, and the route's own fields. The decision is `outcome` (`success` / `denied` / `failed` / `challenged`) with a fixed `reason`, or `undecided` when the route recorded none. A refusal is final: a later admission doesn't replace it, so a replayed NIP-98 proof that verifies and is then refused logs `failed` / `nip98_replay`. - `outcome` describes the login. What the route did is in `status` and the route's fields: a rejected `/events` write logs `outcome=success` with `status=400`, `accepted=false` and its `reject_reason`. - Route fields: - `/query` and `/count`: the filter summary and `result_count`. An incremental `/query` response (`{changes:[...]}`) counts its changes, removals included. Artifact reads record their summary before the query runs, so a read dropped mid-query keeps it. - `/events`: `accepted`, `kind`, `event_id`, and on a refused write the parse `category` / `line` / `column` or a fixed `reject_reason` (such as `invalid` or `restricted`), never the ingest message. - Invite mint: `invite_id`, `expires_at` and `max_uses`. Invite claim: `joined`. - Admin feedback attachment reads: `feedback_id` and `attachment_sha256`. With `BUZZ_ADMIN_AUTH=disabled` they log `method=network_trusted` with no key, and Host or Origin refusals log their fixed reason. - Every other route logs only the login fields. - No field can hold an invite code, a token, or a Blossom read hash. Media routes log the `/media/{sha256_ext}` template. - Skip list: NIP-11 and other non-upgrade requests on `/`, `/info`, `/.well-known/nostr.json`, `/health`, `/_liveness`, `/_readiness`, and the SPA and static files the relay serves when enabled (web assets, invite landing pages, git web pages, the admin bundle) at any status. Successful WebSocket upgrades (101) log no `http` line. Refusals before the upgrade on `/` and the audio path log one, with the status actually sent. - Routes that aren't Nostr-key logins (the command-token disconnect endpoint, the workflow webhook, the git server's internal callback) log `outcome=undecided` with no key. - Dev-mode `X-Pubkey` requests log `method=x_pubkey`, `pubkey_verified=false`. - A successful `/query` or `/count` that returned nothing (`result_count=0`) logs at DEBUG: on staging, 93% of `/query` lines were empty polls. Denied, failed, aborted and error reads, reads with no `result_count`, writes and logins keep their levels. - Other successful lines and git's credential-less first request (`challenged`, with its 401) are INFO. Refusals, statuses of 400 and above, and aborted requests are WARN. In Enforce mode the line records the remapped status (for example 403). Denial bodies and headers are unchanged. ## WebSocket reads - `REQ` / `COUNT` / `sync` lines carry `pubkey`, `conn_id`, `sub_id`, the filter summary, the outcome (`served` / `denied` / `aborted` / `error` plus a fixed reason), and `events_queued` (REQ and sync) or `count` (a COUNT that counted). A `REQ` is `served` only once its EOSE is queued to the client. A read logs WARN only when it ends in `error`. `served`, `denied` and `aborted` are INFO, since `aborted` means the read ended before completing: the client disconnected or sent CLOSE, a same-ID `REQ` replaced it, or the session was cancelled. - A NIP-AR sync subscription writes its `sync` line when its initial set completes, not when live delivery ends. Its summary is the registered live filter (kinds and channels, never artifact predicates), and `events_queued` counts the initial set. - `REQ` lines stay at INFO even when empty at EOSE, because the subscription can still deliver live events later. - Filter summary, shared with HTTP `/query` and `/count`: `kinds`, channel ids, authors and `p` values each list the first 10 distinct values. The exact `kind_count` / `channel_count` / `author_count` / `p_count` (summed per filter) appears only when the list is `*` or shows fewer values than the filters held. A dimension that any filter leaves open logs `*`. `since` / `until` / `limit` take the widest bound and are absent when any filter leaves them open, so no field understates what the read covered. `ids` is a count only. NIP-50 search logs `search: true`, never the text. Only well-formed channel ids and pubkeys are listed. An artifact read's summary uses only the standard dimensions of each raw filter, so artifact predicates and client-defined tags (such as `#1`) are never logged. The summary is skipped when the access target is off. Under `=warn`, every WARN line, HTTP or WebSocket, still carries it. - Per connection, only the first 16 reads that are refused before they run (`auth_required`, quota, concurrency, overload) or denied are logged. The rest are counted as `reads_suppressed` on the close line. A refusal that arrives after the close line has been written, from a read still running when the client left, logs its own line, so every refusal is either logged or counted, never both. Served reads are never capped. ## WebSocket read counter `buzz_ws_reads_total{type="req"|"count"|"sync"}` counts every parsed `REQ`, `COUNT` and NIP-AR sync frame before admission, exported at zero from startup. It is a metric, so it counts at any `RUST_LOG`, including production's current `error`. ## WebSocket logins A NIP-42 login writes one `auth` line when it is decided. A successful login logs INFO `outcome=success` at its commit point, decided under the connection's close-reason lock, so a disconnect racing the login (session expiry, admin disconnect, revocation, ban or roster removal) either wins, and the login logs no success, or loses and closes the connection right after the success line. A refused or failed login writes one WARN `auth` line with `outcome`, a fixed `reason` from the shared `AuthReason` code set, `pubkey` and `pubkey_verified`. Root WebSocket `auth` lines carry `peer_addr`. AUTH frames after the login is decided are capped at 16 per connection, and the rest are counted as `auth_suppressed`. An attempt cut off by close or session expiry logs `aborted`. Audio logins log one `auth` line at their final admission check. ## Connections The close line is the connection's only line. It is owned by the upgraded socket, so it is written exactly once however the connection ends: a normal close, a refusal while checking the community is active (`refused=not_admitted`), the connection limit (`refused=connection_limit`), or the handler being dropped. It carries `conn_id`, `community`, `peer_addr`, `duration_ms`, the login, `events_queued` / `bytes_queued` (frames the writer accepted and flushed, not client delivery), and the two suppression counters. ## Error text - Database errors log as `DbError::kind()` (SQLSTATE or variant), including the command-executor event insert. Search errors, on both read paths and in the search-cancel task, log as `SearchError::kind()` / the same SQLx classifier. Workflow database and webhook errors log as their codes, since webhook URLs can be secrets. - Git NIP-98 failures log a fixed code. Git subprocess stderr moved to DEBUG because hook output can contain message text. ## Operator notes `.env.example` documents the target, its `event` values (`http`, `auth`, `req`, `count`, `sync`, `close`), the skip list, the levels, and how to turn it down: `buzz_relay::access=warn` keeps refusals, errors and aborted HTTP requests, and of WebSocket reads only those that end in `error`, and `=off` drops everything. It also lists the removed lines: `NIP-42 auth successful/failed`, `WebSocket connection established/closed`, `event=connect`, `HTTP bridge request`, `Event ingested via pipeline`, `relay invite minted`, `relay member added via invite`, `admin feedback attachment read`, and `addr` (now `peer_addr`). JSON lines no longer repeat the current span as `span`, since `spans` already ends with it. A line logged under an explicit `parent:` span loses that parent's fields. The default OTel export filter sets `buzz_relay::access=off`, so access lines aren't exported a second time as span events. `peer_addr` is the TCP peer. `BUZZ_CLIENT_IP_HEADER` optionally names a request header (for example `cf-connecting-ip` behind Cloudflare or `x-real-ip` behind nginx, matched case-insensitively). When set, `http`, `auth` (root and audio), `close` and connection-limit lines also log `client_ip`: the rightmost comma-separated entry of the header's last field, only if it parses as an IP address. It never replaces `peer_addr`, and it is only trustworthy when every request arrives through a proxy that overwrites the header. --------- Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
Shu Hui
|
fa7a3bfbda |
feat(relay): push per-community storage usage to kgoose (#8714)
### Why?
To bill hosted communities for storage in Orb, kgoose needs
per-community storage usage. The relay already computes it in a daily
snapshot, but nothing delivered it anywhere.
## Summary
- Adds opt-in `BUZZ_STORAGE_USAGE_URL`. When it is set, one
leader-elected relay pod checks every 5 minutes for a new storage
snapshot and POSTs it to kgoose's `POST /v1/buzz/storage-usage/record`
(squareup/cash-server#144285) in consecutive batches of up to 50
communities until every community is sent.
- Each batch is `{measured_at, communities: [{community_host,
storage_bytes, storage_objects}]}`. Active communities with no stored
bytes are reported as zero, and re-sends are safe because kgoose bills
each community once per UTC day.
- Retries on 408, 429, 5xx, and transport errors. No migration;
documented in `docs/storage-accounting.md`. kgoose authenticates the
relay as the `buzz` mesh service, so the relay needs mesh egress to
kgoose-builderbot.
### Related issue
None found. Complements #7899 (snapshot history in the DB). Receiving
endpoint: squareup/cash-server#144285.
### Testing
Unit tests cover batching, retry and reject handling, zero-filling, and
a real HTTP delivery that checks the exact kgoose request body. A
Postgres-gated DB test covers the new reads. End to end, a local relay
delivered to a fake endpoint once per new snapshot.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Shu Hui <shu@squareup.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
Salman Mohammed
|
6847882b14 |
fix(relay): stop logging unmapped-host NIP-FI denials as server failures (#8997)
## What this does In enforce mode, a request whose Host maps to no community gets NIP-FI's `authorization_unavailable` 503 (`docs/nips/NIP-FI.md`, Rejection and privacy table). tower-http's default classifier treats every 5xx as a failure, so each of these requests also logs a `tower_http::trace::on_failure` error. That line has no host or path, only `503` and `latency: 0 ms`. When a scanner probes a host this relay doesn't serve, it produces thousands of these lines in a minute, and they trip error-rate alerts even though nothing is down. This PR keeps the wire response the same and stops counting these denials as server failures: - `nip_fi_core::community_denial` builds the same `http_denial` response. When the Host is confirmed unmapped, it adds an internal `UnresolvedHost` response extension. Extensions never reach the wire, so the NIP-FI denial contract ([FI-TRACE-DENIAL-ORACLE]) still holds. - Each site that turns a community-resolution failure into a denial now uses it: - the router guard (`nip_fi_assertion_guard`) - the WebSocket upgrade (`check_nip_fi_at_upgrade`) - HTTP admission, for routes the guard exempts (`Stage::Community`) - The HTTP trace layer now uses `RelayResponseClassifier`. It treats marked responses as expected and keeps tower-http's 5xx-is-failure rule for everything else. Other 503s still log as failures. That includes an unavailable JWKS key source on a mapped host, `deny_protected`, a stale snapshot, and a failed database lookup. ## Why not return 4xx for unmapped hosts? NIP-FI.md maps "community/Host resolution unavailable" to 503, and `nip_fi_enforce_unmapped_host_is_503_before_verification` pins that ruling. Changing the status would be a spec change. This PR only changes how the relay logs these denials. ## Database community source With `BUZZ_NIP_FI_COMMUNITY_SOURCE=database`, a not-found host and a failed lookup (DB error, timeout, shed waiter) return the same denial class. The marker is set only when the host is confirmed unmapped: a missing Host, a non-bindable host, or a live negative-cache entry. Resolver outages stay unmarked and still log as failures. With the `env` source, every resolution failure means the host is unmapped, so it is always marked. ## Testing - `nip_fi_enforce_unmapped_host_is_503_before_verification` (guard, root upgrade, audio upgrade) and `enforce_unmapped_host_is_503_before_nip98_and_verifier` (HTTP admission) now also assert the marker. Mutation check: removing the guard marker turns the router test red. - New `nip_fi_mapped_host_dependency_503_is_not_marked_unresolved`: a mapped-host 503 from an unavailable key source is unmarked and still classified as a failure. - New `relay_response_classifier_skips_only_unresolved_host_denials`: covers marked 503, unmarked 503, 500, 200 and 401. - New `only_confirmed_unmapped_hosts_are_marked_as_expected_denials` (database source, all three sites): a negative-cached host and a non-bindable host are marked; a bound host whose lookup fails is not. Mutation check: marking every database failure turns it red. - `cargo clippy -p buzz-relay --all-targets -D warnings` is clean, and the pre-push `rust-tests` lane passed. - Local relay, before and after. Setup: enforce mode, `RUST_LOG=error`, one mapped host, JWKS unreachable on purpose. Both builds got the same requests: | Request | Status (both builds) | `on_failure` errors before | after | |---|---|---|---| | Unmapped host, 4 protected paths | 503 | 4 | 0 | | Unmapped host, WebSocket upgrade on `/` | 503 | 1 | 0 | | Mapped host, no assertion | 401 | 0 | 0 | | Mapped host, well-formed JWT, JWKS unreachable | 503 | 1 | 1 | | Unmapped host, `/assets/` path and `POST /api/invites/claim` | 404 | 0 | 0 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> Co-authored-by: opus <f038cc09b32a776e7d95b65c267e9d851777ee2b6cdbd46a48edba01b1d10ebd@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
88697f1c59 |
fix(relay): apply kind 5 / 9005 deletions atomically (#8944)
## Summary Kind 5 and 9005 deletions now store the event and soft-delete their targets in one transaction. Before this PR the deletion committed first. The target soft-delete ran afterwards as a best-effort side effect, and a failure there was only logged. Resending the deletion then returned `duplicate:` without retrying. A crash between the two commits had the same result. Either way the relay accepted a deletion while its target stayed live. The bridge `include_aux` closure still compensates for those rows. Its second hop returns deletions of the reactions and edits it found, so clients can cancel them. This PR keeps that hop, because rows already torn this way can't be repaired by a resend: - Ingest rejects events more than 900 seconds from server time. - Clients don't resend deletions that were accepted. - During a rolling deploy, old pods keep writing torn rows until they are replaced. Dropping the hop waits for a follow-up, once existing torn rows have had a verified repair and no old writers remain. ### Changes - **Atomic deletion.** Kind 9005, and kind 5 with `e` targets, go through `Db::insert_deletion_event`, driven by `side_effects::persist_deletion`. The write stores the event and its mentions, soft-deletes the targets and updates their thread counters. It follows the existing `persisted_atomically` path used for workflow deletion and kinds 9000/9002. - A failure stores nothing and rejects the event, so the client can resend. - A resend of a stored deletion inside the ingest window soft-deletes any target still live and is dispatched as inserted. It doesn't repair older rows. - Live 39005, reaction-table cleanup and the 9005 system message still run after the commit and stay best-effort. - Kind 5 deletions of `a` coordinates stay a side effect. - **Closure unchanged.** The channel window and legacy thread-reply paths keep both hops, and the docs (`docs/bridge-channel-window.md`, `docs/nips/NIP-CW.md`) are unchanged. ### What got simpler Deletion no longer has a window where the event is accepted but not applied. `handle_standard_deletion_event` is now only the `a`-coordinate path. Target planning and soft-delete for `e` targets live in one place, `persist_deletion`. ### Follow-ups (not in this PR) - Repair the existing torn rows: soft-delete the live targets of stored kind 5 / 9005 `e` deletions and fix their thread counters. Then drop the closure's deletion-of-aux hop. - Hop 1 still probes every partition once per row id. A partition bound or the indexed `reactions` table would cut that. ### Related issue N/A, found while investigating relay database CPU. ### Testing - `torn_deletions_still_cancel_live_targets_in_both_windows` (bridge Postgres) stores kind 5 and 9005 deletions without soft-deleting their reaction and edit targets. It queries the channel window and the legacy thread path with `include_aux`, without a resend, and asserts each live target comes back with its deletion. - `insert_deletion_event_target_failure_stores_nothing_and_resend_applies` (buzz-db Postgres) uses a trigger to make the target's soft-delete fail. It asserts nothing is stored and the target is unchanged, then that a resend applies both. - `insert_deletion_event_resend_repairs_a_live_target` (buzz-db Postgres): a resend repairs a deletion stored without its target's soft-delete, and a further resend changes nothing. - `failed_target_soft_delete_stores_nothing` and `resent_deletion_repairs_a_live_target` (relay Postgres) cover the same two cases through ingest, for kinds 5 and 9005. - `test_channel_window_rows_overlays_and_exact_multiple_exhaustion` (`e2e_nostr_interop`, Relay E2E lane) deletes its reaction and asserts the reaction leaves the closure. - Blox: fmt, clippy (`buzz-db`, `buzz-relay`, `buzz-test-client`), and the `buzz-db` and `buzz-relay` Postgres lanes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Will Pfleger
|
f6198f6997 |
feat(relay): record NIP-FI enforce denials at parity with shadow (#8974)
## What this does
In enforce mode, NIP-FI denials used to be returned without a reason
counter or a log line. This PR records enforce denials with the same
`WouldDeny` reasons, label sets and log fields as shadow. **Admission
behavior does not change**: statuses, close codes and frame order are
all the same.
- **`buzz_nip_fi_enforce_total{route,stage,outcome,community}`**: the
same shape as `buzz_nip_fi_shadow_total`, including an `admit`/`admit`
stage, so `1 - admit/total` is the enforce denial rate. Each socket or
HTTP request gets exactly one verdict.
-
**`buzz_nip_fi_enforce_session_end_total{route,reason=expired|revoked,community}`**:
once per admitted session.
- **`buzz_nip_fi_enforce_strict_proof_total{route,outcome,community}`**:
the outcome of the proof check that actually decided the request.
Nothing is verified twice.
- **Log lines** `nip-fi enforce deny` and `nip-fi enforce session end`
(info, `mode="enforce"`, from `buzz_relay::nip_fi_shadow`). Shadow's
`nip-fi shadow would-deny` gains `mode="shadow"`. Neither mode logs
admits.
### How each verdict is recorded once
- **Sockets (root, audio).** `EnforceSession` moves Pending →
Admitted(pubkey) → Ended under one std mutex, and records inside the
lock with no awaits.
- Root admits at its AUTH commit; audio admits at its join commit.
- A refusal for a non-NIP-FI reason retires the session without a
record. Audio retires it before rollback.
- A deadline the effect gate sees before the expiry worker is claimed at
that seam.
- **Ends.** Revocation and expiry are recorded only by the terminal
transition that wins (`publish_authorization_denied`), and only while
the socket isn't already cancelled. A community close or cancellation
that wins first records nothing.
- **HTTP.** The router guard scopes each enforced request with one
verdict. An agent token's verdict is deferred to the end of the request:
- the first owner-decided refusal is the verdict, otherwise the request
was admitted;
- membership rechecks, batches, Git success and early returns each count
once.
- **Owner provenance** comes from the decision itself
(`BanVerdict{outcome,source}` from one `restriction_state` read), with
no rereads.
- Reasons: `owner_banned`, `owner_ban_unavailable`, `not_stored_owner`,
`owner_link_unavailable`.
- When the agent's own ban read fails and there's a token owner, the
result is `owner/unavailable`, as shadow records it.
- A ban or failed restriction read that ingest catches on `POST /events`
after membership passed replaces the request's admit with `owner/denied`
when the owner's row decided the ban, or `owner/unavailable` on a failed
read. An agent's own ban is not recorded as its owner's.
## Shadow-to-enforce map
| Shadow signal | Enforce counterpart |
|---|---|
| `buzz_nip_fi_shadow_total` denials: upgrade, guard, HTTP admission,
AUTH pairing, deny set, stale binding | `buzz_nip_fi_enforce_total` with
the same stage/outcome, plus a `nip-fi enforce deny` line |
| `buzz_nip_fi_shadow_total` owner stage: ws, audio, http | the same,
from the original decision: root/audio ban seam and owner link; HTTP
`enforce_membership`, smart Git, Git settings, buzz/v1 ban reread |
| `buzz_nip_fi_shadow_total` `deadline/session_deadline` (pending) |
`EnforceSession::expire`, from the worker's winning terminal or the
effect gate that saw it first |
| `buzz_nip_fi_shadow_total` admit |
`buzz_nip_fi_enforce_total{stage="admit",outcome="admit"}`, one per
socket or request |
| `buzz_nip_fi_shadow_session_end_total{reason=expired\|revoked}` |
`buzz_nip_fi_enforce_session_end_total`, same labels |
| `buzz_nip_fi_shadow_strict_proof_total` |
`buzz_nip_fi_enforce_strict_proof_total`, from the deciding check (a
rejection is the route's 403, or 401 when the proof is missing) |
| `buzz_nip_fi_shadow_disconnect_total{admin,accepted}` |
`buzz_nip_fi_disconnect_total` (`api/nip_fi.rs:160`) |
| `buzz_nip_fi_shadow_disconnect_total{admin,propagation_failure}` |
`buzz_nip_fi_disconnect_propagation_failures_total`
(`api/nip_fi.rs:180–185`) |
| `buzz_nip_fi_shadow_disconnect_total{admin,capacity}` |
`buzz_nip_fi_disconnect_capacity_rejections_total`
(`api/nip_fi.rs:196–201`) |
| `buzz_nip_fi_shadow_disconnect_total{cross_pod,capacity}` |
`buzz_nip_fi_cross_pod_capacity_exceeded_total`
(`api/nip_fi.rs:433–434`) |
| `buzz_nip_fi_shadow_disconnect_total{cross_pod,poison}` |
`buzz_nip_fi_cross_pod_shard_poison_total` (`api/nip_fi.rs:442–443`) |
| `buzz_nip_fi_shadow_disconnect_total{cross_pod,lag}` |
`buzz_nip_fi_disconnect_lag_total` (`main.rs:1596–1602`) |
The enforce series was renamed from `buzz_nip_fi_enforce_denied_total`
to `buzz_nip_fi_enforce_total` when it gained the `admit` stage.
## Tests
Infra-free, in the `test-unit` lane:
- `state::enforce_end_is_recorded_only_by_the_winning_terminal`: root
scan, audio scan and deadline, each against a community close or
cancellation first and FI first.
- `nip_fi_http::enforce_strict_proof_records_the_deciding_check_once`:
pass and reject, and no record when cardinality refuses.
-
`nip_fi_shadow_session::audio_owner_link_refusal_shares_roots_seam_and_reasons`:
calls the shared owner-link helper with route `audio`, not a huddle
socket.
- The existing shadow, upgrade, router and HTTP enforce tests.
Postgres lane:
-
`handlers::auth::enforce_root_auth_records_owner_refusals_from_one_decision`:
clean admit then `expired` through the real expiry worker; banned token
owner, banned stored owner, other stored owner and a failed ban read;
authorization reads counted from the decision only.
-
`handlers::auth::enforce_root_deadline_seen_by_the_gate_first_is_recorded_once`:
both orderings.
- `audio::handler::enforce_join_commit_is_the_audio_admission`: drives
the join-commit seam directly, not a huddle socket or the expiry worker:
committed then expired; archived with an expiry held inside the
rollback; gate-first deadline.
-
`api::bridge::token_owner_reaches_membership_on_every_nip_fi_http_route`:
one verdict per request on every NIP-FI HTTP route, including sidebar,
read-state, Blossom and Git, for an owner ban, a stored-owner conflict
and a clear owner.
-
`api::bridge::token_owner_ban_reread_is_the_fixed_nip_fi_denial_on_http`:
own ban, owner ban and unavailable at the reread, on sidebar and Git
settings.
-
`api::bridge::token_owner_publish_ban_is_the_fixed_nip_fi_denial_on_http`:
a late ban or failed read on `POST /events` at the write, typing,
moderation and relay-admin checks records exactly one owner denial and
no admit when only the owner is banned, and no owner denial when the
agent alone or both are banned.
-
`audio::handler::tests::already_expired_audio_bootstrap_records_the_deadline`:
an already-expired bootstrap records `deadline/session_deadline`, not a
deny-set denial.
Each new test names the mutation that turns it red. These were run red:
- deleting root's `session.admit(pubkey)`
- deleting audio's commit `session.admit(pubkey)`
- deleting `effect_refused`
- deleting enforce from `retire_shadow`
- deleting the commit seam's `expire`
- deleting the cancellation guard
- deleting the own-read-failure provenance
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <wpfleger@block.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
|
||
|
Daniel Rouleau
|
a9aa1f73b0 |
Enable incremental artifact reads on every relay (#8975)
## Why Compatible clients need incremental artifact reads for local collections on every Buzz relay. ## What Expose incremental artifact reads and capability discovery on every Buzz relay. ## How Serve incremental queries through the existing authentication and authorization checks. Keep the existing work budgets and the separate WebSocket sync capability. The HTTP test verifies capability discovery and a successful incremental query. ## Risk The relay starts no additional background work. Clients that use incremental reads consume database capacity. The net database load is not yet measured. Serving writers must maintain the modification timestamps. ## Testing Read-only SQL checks on November 3, 2026 verify the prerequisites below. Each database records successful migrations 76–78. All five required columns have the expected types, defaults, and nullability. All five indexes have the expected definitions and are valid, ready, and live. | Deployment | Schema and indexes | Running timestamp writers | | --- | --- | --- | | Production `bb-block` | Pass | `f6f6d9a` includes PR 8523 | | Production `bb-public` | Pass | `f3a7bab` includes PR 8523 | | Staging `bb-block` | Pass | `b4509de` includes PR 8523 | The checked columns are `artifact_heads.modified_at`, `artifact_revisions.modified_at`, `artifact_revisions.source_channel_id`, `channel_members.sync_modified_at`, and `communities.artifact_policy_modified_at`. The checked indexes are `idx_artifact_revisions_artifact`, `artifact_heads_modified`, `artifact_heads_channel`, `channel_members_sync`, and `artifact_revisions_moves`. At commit `62b26ad`, all 21 NIP-11 tests and the HTTP capability test pass on Linux with PostgreSQL 17 and Redis. The HTTP test checks capability discovery and a successful incremental query. All required push hooks also pass. No manual testing. Human confirmation remains pending. ## Bigger picture [Client PR 1507](https://github.com/block/buzz-app/pull/1507) uses the advertised capability for local artifact sync. Existing deployments receive this behavior when they deploy the relay version that includes this PR. Generated with Codex --------- Signed-off-by: Daniel Rouleau <drouleau@squareup.com> |
||
|
Will Pfleger
|
ce47aefdc3 |
fix(relay): send membership notices on channel delete, moderation kick and moderation DM (#8977)
Related: [buzz-app#2215](https://github.com/block/buzz-app/pull/2215) ## Problem A connected client learns about membership changes from kind:44100/44101 notices and then re-reads its channel list. Three relay paths changed a member's channel list without sending a notice, so clients only caught up on their periodic full read: - **Channel delete (9008):** no member was told. - **Moderation kick:** the kicked member wasn't told, and the channel's 39000/39002 discovery wasn't refreshed, so it still listed them. - **Moderation DM:** a new, existing or hidden moderation DM didn't surface live. ## Change - **Channel delete:** `soft_delete_channel_capturing_members` (buzz-db) reads the channel type and active members under deletion's existing membership lock, in the same transaction. A join either commits first and is included, or waits and is then refused. After the commit, discovery cleanup and cache invalidation, the relay sends a type-tagged 44101 to each captured member in a spawned task, eight at a time, outside any lock. The 9008 OK doesn't wait for them, and the task is started before the `channel_deleted` system message is emitted, so a system-message failure can't drop them. An already-deleted channel sends nothing. `soft_delete_channel` keeps its signature and delegates. - **Moderation kick:** `apply_kick_live_side_effects` refreshes 39000/39002 after the removal fence commits, then sends 44101 to the kicked member. If the refresh fails, it returns the error before the notice, so the moderation driver's existing mutation-marker recovery retries. If the channel was deleted since the kick, there's no roster to refresh: it removes the channel's discovery events and treats that as converged, so the action still finalizes. The removal/re-add fence and the re-added early return are unchanged. - **Moderation DM:** `send_moderation_notice` drops both participants' cached channel sets on every pod (`invalidate_membership`), as opening a DM does in `command_executor::handle_dm_open`, so the read the notice prompts includes the DM. After clearing `hidden_at`, it republishes the recipient's NIP-DV kind:30622 visibility snapshot (clients filter the sidebar on it), then emits discovery, then sends 44100. It sends the notice on every delivery, tagged `moderation_source` with the delivery's source id, so two deliveries in the same second each get their own 44100. The content JSON is unchanged. A failed snapshot or discovery returns an error, so the outbox retries. - **Ordering guarantee:** each notice is sent after its change is committed and readable on the relay instance that sends it. Ordering across relay instances, and other instances' cached channel sets, are best-effort, the same as for existing membership notices. - A failed notice delivery is logged and never undoes the change, like the existing 9001 path. There's no new retry system. ## Tests - `membership_notice_postgres_tests.rs`: one journey per path. A `#[cfg(test)]` barrier at the point a notice is signed parks the producer, keyed by recipient. While it's parked, the test reads the channel list (and, for the DM, the 30622 snapshot) as a client would on receipt. The barrier can also pin a notice's `created_at`. - The DM journey pins both deliveries to the same second, hides the DM through the real kind:41012 path between them, and checks what a client sees: a live 44100 for each delivery, with distinct ids, then a channel-list read that lists the DM and a visibility read that no longer hides it. - `moderation_dm_notice_follows_a_cached_channel_list` warms the recipient's channel set with a read before the DM exists, then checks the read on receipt of the notice lists the DM. - `soft_delete_captures_members_at_its_serialization_point` (buzz-db): a join and then a delete queue behind a held membership lock. The delete must return the joiner, and a later join is refused. - `crash_recovery_finalizes_kick_after_channel_deleted` (admin): commits a kick, deletes the channel, then runs the recovery worker. The action finalizes, and the member's `removed_at` is unchanged, so the removal doesn't run again. Each ordering test fails when its prerequisite is moved or removed: the kick notice ahead of the roster refresh, the delete notices ahead of discovery cleanup, the DM without the snapshot republish or the cache drop, the DM notice without its source tag (the second notice coalesces), and the member capture outside the lock. The recovery test fails if a deleted channel's `ChannelNotFound` propagates. No spec files change. This reads a channel delete as removing every member under the existing `NOSTR.md` notice promise. --------- Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
Luke Tornquist
|
d44031f418 |
perf(relay): cut writer-pool round trips past the 48k relay ceiling (#8846)
## Summary Five changes that take load off the writer pool and the relay's connection bookkeeping. On a single-relay load rig, the clean ceiling moves from 40k to about 48k WebSocket clients. In the A/B, connect p99 at 48k fell from 4.1 s (abort) to 1.65 s. 1. **Ban/mute restriction cache with a 1 s max age.** HTTP API requests, publishes and ephemeral writes read `(community, pubkey)` restriction state through one cache. An entry is served for 1 s from when its read began, matching the open-channel cache's bound. Socket admission, moderation-command actor checks and the `/buzz/v1` ban recheck still read fresh. An admitted socket outlives the 1 s window, so a banned client that reconnects inside it would otherwise keep a socket open indefinitely. A ban drops the target's entry, and each owned agent's entry, on every pod as part of the live-disconnect fan-out, best-effort. Timeouts, untimeouts and unbans drop it on the pod that handles them. 2. **Fewer round trips per event read.** Replica reads skip the per-read `statement_timeout` tighten, because the reader pool installs the deadline at connect. They keep the savepoint, so a failed or cancelled read leaves the session usable and releases its locks. That takes them from 4 round trips to 3. Writer reads send `BEGIN` and the tighten as one message, which also takes them from 4 to 3. 3. **Concurrent open-channel cache misses share one read.** The first miss in a community takes a per-community gate and reads; the others wait for it and use its result. A waiter whose leader failed reads once on its own and does not queue again. A waiter also stops waiting after the cache's 1 s max age and reads on its own, because the leader's set would be stale by then anyway. So a leader whose read hangs stalls only its own request. 4. **`has_admitted_connection` is one map lookup.** A `(community, pubkey) → admitted count` index is maintained under each socket's pubkey lock in `mark_admitted`, `set_authenticated_identity` and `deregister`. It replaces a scan of every connection on each disconnect. 5. **Host → community cache with a 1 s max age.** Only successful lookups are cached; unmapped hosts and errors go to the database every time. ## What got simpler - The ban handler's separate pod-local cache drop is gone; the disconnect every ban already runs covers it on every pod. - One write-restriction gate instead of two. `enforce_cached_write_restriction`, `enforce_relay_membership_ephemeral` and the `cached_ban` parameter chain through `enforce_membership` / `admit_member` / `deny_banned` are deleted. - The ephemeral path's separate 30 s ban cache is gone, so the relay has one staleness rule for restriction state instead of two. ## Notes - Bans reach the restriction cache on every pod. Every ban path (signed command, report action, relay admin) closes the target's and its agents' sockets clusterwide, and each pod drops those entries in the same step. A dropped Redis publish, or a restriction read already in flight when the drop lands (it can refill the old row), falls back to the 1 s bound. Timeouts on other pods also rely on the 1 s bound, and so do unbans and untimeouts, which fail closed. - On the load rig the restriction-cache hit rate was 28%, because HTTP agents poll every 5 s. A longer TTL with Redis invalidation would raise it, but that is a separate change. ## Load test Single relay pinned to 8 cores, native Postgres 16, a seeded fixture with 2,000 open channels, and a 6-shard generator. Steady ramp with 120 s holds; a step aborts at p99 > 3 s. Each step adds 10% HTTP agents. | clients | base connect p99 | this PR connect p99 | base relay cores | this PR relay cores | |---|---|---|---|---| | 32k | 34 ms | 31 ms | 1.71 | 1.66 | | 40k | 45 ms | 26 ms | 2.33 | 2.15 | | 48k | 4,107 ms (abort) | 1,652 ms | 4.7 | 3.0 | | 56k | n/a | 4,535 ms (abort) | n/a | 4.0 | Open-channel re-reads over the run fell from 72k to 1.6k. The tenant cache hit rate was 94%. 48k is this branch's knee. A later run of the same build aborted at 48k: connect p99 2.2 s, but 29% subscribe errors from the concurrent-handler cap. With a 10 s restriction max age, it passed 48k, at connect p99 1.56 s, 60% hit rate and 10% fewer relay cores, and both arms stopped at 56k. ## Testing - `cargo fmt --check`, `cargo clippy -p buzz-relay -p buzz-db --all-targets -D warnings` - `cargo test -p buzz-db --lib` and `-p buzz-relay --lib`, run in parallel. The only failures were 8 DB-seeding tests that need a migrated default database; the Postgres lane covers them. Commits after that run changed the replica savepoint, the `/buzz/v1` recheck revert and some test edits. - Postgres lane at head: `scripts/postgres-test-run.sh -p buzz-relay -p buzz-db --lib --bins --tests`. All 1,183 tests passed. - Mutants. Each one must turn its new test red: - waiters skipping the gate - a no-op index removal - the tenant cache ignoring age - the restriction cache ignoring age - a waiter waiting on the gate with no limit - a disconnect that keeps the cached restriction row - replica reads dropping the savepoint buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
02925ceb6e |
fix(relay): cap bridge searches in flight per user (#8956)
## Summary
Typeahead starts a bridge `/query` search on almost every keystroke. The
client aborts the previous request. But if the abort never reaches the
relay, for example because an intermediary keeps the upstream request
open, every one of those searches keeps running.
Each `(community, pubkey)` may now have **4** bridge searches in flight.
The next search supersedes the oldest one. The superseded request
returns **409** `{"error":"search superseded by a newer search"}`. It is
deliberately not a 429: desktop arms its relay-wide admission gate on
any 429, which would hold every relay request for 10 s, and the CLI
retries a `rate-limited:` 429. Its future is dropped, so #8954's
cancel-on-drop cancels its statement on the server.
- **Newest wins.** The newcomer is never rejected, because the newest
keystroke is the one the user is waiting for.
- **The cap is above 1.** Sessions that share a pubkey keep their
parallel searches: several agent sessions, or people search and message
search running side by side.
- **Bookkeeping lives in process.** `SearchSlots` holds a
`Mutex<HashMap<key, VecDeque<(id, CancellationToken)>>>`. A slot removes
itself by id when its search finishes, fails, is dropped or is
superseded. So a superseded search can never remove the newer search
that replaced it, and a user's entry is removed when their last search
leaves.
- **A search that finishes as it is superseded still returns its
result.** The `select!` is biased toward the search.
The limit is per relay process, not global. That is enough to bound how
much one user can pile onto one process's search pool.
## What got simpler
Nothing was removed. The change adds one small, self-contained module
and one call site. It also wires the new tests into the unit lane
(`Justfile test-unit` and `scripts/run-tests.sh`). The relay's unit lane
is an allowlist, so without that wiring the tests would run nowhere in
CI.
## Tests
`api::bridge::search_slots::tests`:
- `the_search_past_the_cap_supersedes_the_oldest`
- `a_finished_search_frees_its_slot`
- `a_superseded_search_leaves_its_successor_in_place`
- `no_entry_outlives_its_searches`
- `a_superseded_search_is_a_conflict_not_back_pressure`
Each of these mutants made at least one test fail:
- Removing the cap.
- Replacing the by-id removal with `pop_front`.
- Leaving empty user entries in the map.
- Answering supersession with 429.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Luke Tornquist
|
949f61dfab | fix(search): cancel abandoned searches and bound the search pool (#8954) | ||
|
Logan Johnson
|
5f8d0aab98 |
fix(relay): refuse a second listed open channel with the same name (#8782)
## Summary
Two open channels in one community could have the same name, so the
channel list showed two `#square-ios` rows and people could not tell
them apart. Now the relay refuses to create, rename or open up a channel
when that would give it the same name as another channel in the channel
list, ignoring case. The client gets `invalid: an open channel named "…"
already exists`.
The rule follows Slack: an archived channel keeps its name until it is
deleted. Private channels, DMs and unlisted channels may still repeat
names. So a refusal never reveals that a private channel exists.
The PR also adds a script that renames the duplicates that exist today.
It sends each rename through the relay, so relays keep serving while it
runs. The stacked PR https://github.com/block/buzz/pull/8787 then adds a
database index that enforces the rule. This PR is the release that must
be fully deployed before that index.
## Details
**Which channels the rule covers.** A *listed* channel: open, not
unlisted, a stream or forum, not deleted. Archived channels are
included. The check runs on create, rename, private → open and unlisted
→ listed.
**Relay check** (`crates/buzz-db/src/store/channel.rs`,
`crates/buzz-relay/src/handlers/ingest.rs`):
- The check runs after the write, in the same transaction, under a
per-name lock. So two concurrent writers cannot both commit the same
name.
- A refused create or edit (kind:9007 or kind:9002) gets `OK false
"invalid: an open channel named … already exists"`. Nothing from it is
stored.
- The relay gives the same answer when the database index from
https://github.com/block/buzz/pull/8787 refuses the write. That covers
the rollout, and a rollback from that release.
- A channel create with no `h` tag (the new channel's id) is now refused
before it is stored. Before, such a create ran after the event was
stored, so a refusal (for example, a taken name) still answered
"accepted". All first-party clients send `h`.
**Operator rename** (`crates/buzz-relay/src/handlers/side_effects.rs`).
The rename script needs to rename channels it does not own, archived
ones included. So a relay operator may now send a channel edit that
changes only the name of a listed channel. Everything else still needs
the channel's owner or admin.
- "Operator" means `RELAY_OPERATOR_PUBKEYS`, the owner fallback, or an
`operator` row in `relay_operators`. A moderator is not an operator.
- The rename also applies to an archived listed channel, whatever the
operator's role in that channel, and the channel stays archived. Other
edits of an archived channel are still refused.
- Private channels, unlisted channels and DMs still need the channel's
owner or admin. The relay decides "listed" under the channel row lock,
so a channel made private during the rename is not renamed.
- The rename takes the normal edit path: the name check, the stored
event, the channel metadata event and fan-out.
**Rename script**
(`scripts/maintenance/rename_duplicate_listed_channel_names.sh`, with
the read-only plan `plan_duplicate_listed_channel_renames.sql`):
- In each group of listed channels that share a name, the oldest
unarchived channel keeps the name. Each other channel becomes
`<name>-<first 8 chars of id>`, or `<name>-<id>` when that is also
taken. When all the channels in a group are archived, the oldest keeps
the name.
- **Starter channels.** Released desktop and mobile apps create
`#general` and `#welcome-everyone` when a community has no open,
unarchived stream with that name. Under the new rule, the relay refuses
that create while an archived channel has the name, and the apps' setup
fails. So the script also renames `general` and `welcome-everyone` when
every listed channel with that name is archived. That frees the name for
the apps.
- The apps create each starter at a fixed id, derived from the relay's
address. If a channel already has that id, the relay refuses the create
there whatever its name, so a rename cannot help. For that name the
script renames nothing, and an archived starter at its id keeps the
name. The script computes the ids from `BUZZ_RELAY_URL` (or
`BUZZ_STARTER_ORIGIN` when the apps use a different address) and prints
them, so the operator can check them against a client. A relay-side fix
for those clients is planned separately.
- The operator key goes in `BUZZ_PRIVATE_KEY`. It must be a relay
operator and a member of the community. The script stops at the first
auth refusal and says so.
- The script stops before the plan for an unknown host, an archived
community and a community being deleted, and says what to do.
- A failed rename (the new name is now taken, or the relay can't be
reached) is reported, and the script goes on. At the end it reads the
plan again and exits 1 if any rename remains. `--dry-run` prints the
plan.
- The script replaces the earlier direct-SQL repair and `buzz-admin
reconcile-channels --metadata`, which needed every relay pod stopped.
**Duplicates that exist today** (read on 2026-10-30):
`ml-platform-public` (`d41a31f8…`, `7be55f5a…`) and `square-ios`
(`79f8e9d4…` archived, `fe246d3e…` live).
### Rollout
The full procedure, with exact SQL and recovery steps, is in
`docs/channel-name-uniqueness-deployment.md` in
https://github.com/block/buzz/pull/8787.
1. **Deploy this PR.** Wait until no older relay pod serves; check the
image of every pod. An older relay can still create a duplicate, and it
refuses the operator rename.
2. **Rename the existing duplicates in each community**, with relays
serving. The key must be an `operator`, not a moderator, and a member of
the community (`RELAY_URL=wss://<host> buzz-admin add-member --pubkey
<npub>`).
```sh
DATABASE_URL=postgres://... BUZZ_PRIVATE_KEY=<operator key> \
scripts/maintenance/rename_duplicate_listed_channel_names.sh --dry-run
<community host>
# review the plan, then run it again without --dry-run
```
For the duplicates known today, the newer `ml-platform-public` and the
archived `square-ios` (`79f8e9d4…`) get the `-<id8>` suffix.
3. **Confirm that no duplicate remains** in any community (query in the
doc).
4. **Build the index, then deploy
https://github.com/block/buzz/pull/8787.**
Rollback from https://github.com/block/buzz/pull/8787 goes to this
release only, keeping the index, with auto-migration off.
### Related issue
None found.
### Testing
- **Live local relay.** The script ran against a local relay built at
|
||
|
Matt Toohey
|
322fdbf086 |
test(relay): hold the replica's second aux page and move media's DB tests (#8740)
## Summary Two buzz-relay test fixes from the 23–30 Oct flake audit. Both are test-only. This is one of six PRs split from the reviewed `flakey-tests` branch (siblings below). ### `thread_window_bridge_restarts_both_aux_hops_after_replica_failure` (3 runs) - **Runs:** 40799617319, 40924179886 and 40930100994 (#8697, docs only), all on 10-29 in PostgreSQL Tests, attempt 1 (jobs 123208043033, 123632125866, 123651641392). None of the branches touches thread_window, and each run passed on a re-run. Signature: `failure_postgres_tests.rs:46`, "barrier must precede closure completion", left: 2, right: 1. - **Cause:** `pause_after_aux_page` polls the request until a `thread_window_aux` span closes, then expects exactly one. The fixture's 1,001 reactions make two replica pages at `AUX_LIMIT` 1000, and each page is one round trip on the transaction already held. A loaded runner can deschedule the test thread after page 2's query is written. Postgres then answers first, and the same poll reads page 2 too. - **Change:** `test_support` gains a `PausingProxy`, a trimmed copy of buzz-db's private one with only the hold-everything switch. The replica reader pool connects through it. `AuxPages` pauses it when the first aux span closes. That runs inside the request's poll, after page 1's response is consumed and before page 2's query is written, so page 2 stays pending however fast Postgres answers. The test resumes the proxy after it terminates the reader's backend. The held page then sees the termination, and the closure restarts on the writer, as before. The other three `pause_after_aux_page` callers read from the writer and pass `None`. ### `api::media` tests that need Postgres (fail every time Flake Check selects them) - **Run:** 40954304982 (#8658), 10-29, Flake check (Rust), job 123730434146. All six `test_state()` tests panic at the community-seed `expect` after the 30 s sqlx acquire timeout. - **Cause:** when a hunk lands outside a test body, Flake Check selects every test in the file. It skips only `postgres_tests::`, `postgres_*` binaries and `external_infra`, and its job has no database. These six tests seed `relay.example` through Postgres but sat in plain `mod tests` without `#[ignore]`. So any PR that changes non-test code in `media.rs` fails that lane, and at least 8 open PRs touch the file. No other lane ran these tests either. - **Change:** move `test_state`, the `media_get_*` helpers, `media_request` and the six tests into `tests::postgres_tests`. Mark each test `#[ignore = "requires Postgres"]`, and point `test_state` at `test_support::database_url()`, as the module's other states do. The PostgreSQL lane now runs them, and Flake Check skips them. The change also drops imports `mod tests` no longer uses, and updates the Justfile comment that gave `api::media` as the reason the infra-free lane is scoped to `api::admin`. - When `media.rs` is staged, the pre-commit secret scan reads the whole file and flags the module's existing public ES256 test key (`TEST_EC_PKCS8_PEM`). The key is now written on one line with a sadscan marker, as `invites.rs` already does. Its bytes are unchanged. ### Related issue None found. **Conflicts with #8451 and #8444**, which both already conflict with `main`. The one-line `TEST_EC_PKCS8_PEM` rewrite is the only new conflict hunk with each. The test move merges cleanly with both. ### Testing Verified locally: - Each of the four `pause_after_aux_page` tests passes when run alone against local Postgres through `scripts/postgres-test-run.sh`. - A forced check, not committed: after the barrier, the test gave page 2 another 500 ms over ten re-polls and asserted the page count stayed at 1. It passed with the pause and failed with `None`, with the CI signature (left: 2, right: 1). A true same-poll burst needs the OS to preempt between sqlx's write and read, so it was not forced directly. - `check-postgres-test-discovery.py` accepts the touched files and all 621 files under `crates/`, and `scripts/test-postgres-test-discovery.sh` passes. Each of the six media tests passes when run alone against local Postgres. - `cargo fmt --check` and `cargo clippy -p buzz-relay --all-targets -- -D warnings` are clean under 1.98.1. Only hosted CI can show the full lanes, which were not run locally. The six media tests have never run in CI, so this PR's PostgreSQL lane is their first hosted run. Pushed with `git push --no-verify`, because the pre-push suites fail when the laptop is under load. ### Readiness An agent review of the combined `flakey-tests` branch (b432453af) found no blockers. `buzz-review-completed` is not added, because the human test step (AGENTS.md "Before opening a PR", step 3) hasn't happened yet. ### Siblings - #8739 ci: Hermit fetch deadline and sccache fallback - #8741 test(mobile): RunnerTests title fade and alert wait - #8742 fix(desktop): deep-linked message hold (product change) - #8743 test(desktop): four specs racing scroll settles and focus - #8744 test(desktop): five settle classes and remote-owned-mentions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Matt Toohey
|
cde6c1bf5b |
feat(relay): auto-join new community members to default channels (#8594)
## Summary Adds **default channels**: relay owners and admins can flag channels that every new community member is automatically joined into. **Setting (contract)** - New `channels.auto_join` column (migration `0080`, constant default `false`, so no table rewrite). - kind:9002 accepts `["auto_join", "true"|"false"]`. Authorization comes from the actor's **relay** roster role (owner/admin), not a channel role, so the actor doesn't need to be a channel member. API tokens need `AdminChannels`. - Only open, unarchived `stream` channels qualify. This is checked under the channel row lock, so one event can open and flag a channel in one step, and a concurrent archive can't slip past the check. - kind:39000 carries a bare `auto_join` tag only while the setting is effective (on, open and unarchived). Every change posts one `auto_join_changed` kind:40099 notice. - SDK `build_update_channel(.., auto_join)`, `buzz channels update --auto-join <true|false>`, a NOSTR.md "Default channels" section, and a NIP-CL tag-table row. **Behaviour** - kind:9021's body is pulled out into `join_channel_as_self`, so auto-join uses the same path as a self-join: the channel's `join_role` applies, and it emits `member_joined` 40099, 39000/39002 and 44100. - `auto_join_default_channels` runs inline right after a member is added at three sites: the v2 invite claim, the v1 invite claim, and kind:9030 add. It does not run on login or reconnect, so a later kind:9022 leave stands. - Best effort: each failure is logged and counted (`buzz_auto_join_failures_total`) but never retried, because the roster membership has already committed. This deliberately departs from Review-Proven Rule 1 and is documented in NOSTR.md. `buzz-admin add-member` writes the roster row only. **Communities without a membership roster** (`BUZZ_REQUIRE_RELAY_MEMBERSHIP=false`) - These have no invite claim or kind:9030 to hook, so the three sites above join nobody there. Instead, after the AUTH `OK`, the relay joins a person to every effective default channel they have **never had a membership row in** (`auto_join_never_joined_channels`, spawned so admission never waits on it). - The channel's own membership history is the once-only record, so this adds no new table, column or migration. A kind:9022 leave or a kind:9001 removal keeps the row with `removed_at`, so the person stays out. A new device or another relay pod finds the row and does nothing. `put_first_member` checks the row again under the `put_member` locks, so two admissions racing each other join once. - A channel flagged later is joined on each person's next admission. A failed membership write leaves no row, so the next admission retries it. - Each channel's join holds the session's NIP-FI effect permit, so once the socket closes or its admission expires, no further channels are joined. - Agents are skipped: any socket with an admitted owner (a NIP-OA tag, a NIP-FI agent token, or a stored `agent_owner_pubkey`). - The kind:9021 side effects moved into `announce_self_join`, so both join paths emit the same `member_joined`, 39000/39002 and 44100. **Review follow-ups** - `get_accessible_channels` now selects `auto_join`. Without it, every call failed with `ColumnNotFound`. - The final authorization branch of the 9002 validator now refuses by default. `list_auto_join_channels` filters to `stream` channels itself. ## Testing - Postgres-lane journeys: - `edit_metadata_postgres_tests`: who may set `auto_join`, the 40099 notice, the effective-state tag, and the refusals. - `claim_auto_joins_default_channels_once`: v2 and v1 claims, and leave then re-claim. - `adding_a_member_auto_joins_default_channels`: kind:9030 add. - Removing the auto-join call at any one of the three sites fails the matching test (checked by mutation). - `root_auth_joins_people_to_default_channels_without_a_roster`: through `handle_auth` on a roster-less community. A person is joined, a NIP-OA agent is not, and a leave holds when the person is admitted again. Mutations checked: removing the spawn, and dropping the agent check. Each check waits for that admission's own auto-join outcome (a skip reports through the same test hook as a finished task) and fails if none arrives. A third mutation drops the agent check and delays the task 3s; it still fails. - `admission_auto_join_stops_when_the_session_ends`: closing the socket before the first or second channel stops the remaining joins; a later admission joins what was skipped. Mutation checked: dropping the effect permit. - `first_member_put_joins_once_and_never_undoes_a_leave` (buzz-db): the person is joined once with the channel's `join_role` (`guest` on an announce channel), a second put does nothing, after a leave the channel is neither listed nor joined, and two racing puts write exactly once. Mutations checked: removing the row check, removing the channel lock, and removing the `NOT EXISTS`. - e2e `test_invite_claim_auto_joins_default_channel` against a local relay. - Unit coverage in the SDK and CLI builders and `group_metadata_tags`. ## What got simpler The roster-less path adds no schema and no new concept: it reuses the channel's membership history as its record. It also shares the kind:9021 join code. It replaces the per-device client workarounds, [buzz-app#2029](https://github.com/block/buzz-app/pull/2029) and [buzz#8795](https://github.com/block/buzz/pull/8795), which are now closed. ## Related - Client half (buzz-app): https://github.com/block/buzz-app/pull/1624 ## Demo Recorded against a local relay built from this branch with throwaway identities (Alice = owner, Bob = new member); no real accounts or hosted relays involved. Relay at 141aa01d4, client at buzz-app bb0d97d86 (browser dev build). **Owner turns on "Starter channel for new members" on two channels.** Alice creates `#general` and `#welcome`, then flips the switch on each (~0:25 and ~0:41). The switch greys out while the kind 9002 is in flight and only turns on once the republished 39000 carries the `auto_join` tag. https://github.com/user-attachments/assets/e590d343-11d3-4385-af43-2cfaf19f8468 **A brand-new member joins and lands in a starter channel.** Bob claims Alice's invite with empty storage; from ~0:18 he is reading `#welcome` with both `#general` and `#welcome` in the sidebar, and at 0:27 opens `#general` as a member with no Join prompt. The relay logged one `auto-joined default channel` line per flagged channel. https://github.com/user-attachments/assets/037a0aaa-b25d-474d-a316-8d25e5e54f89 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com> Signed-off-by: tornquist <tornquist@squareup.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> |
||
|
fsola-sq
|
7d00c70805 |
feat(relay): publish NIP-FI verified profiles (BUZZ-433) (#8774)
## Summary Publish historical verified names from valid human NIP-FI assertions as relay-signed kind 30382 events. Keep profile publication optional and separate from access decisions. Disconnects and access changes initiate asynchronous cleanup without retracting profiles verified after the original trigger time. Publish `display_name` as the chosen short display handle and `username` as the account handle. Carry optional `full_name` in the signed assertion into a matching kind 30382 tag. Adding, changing, or removing the full name updates the profile immediately; retractions remove every name field. Older profiles without `full_name` remain valid. Adds opt-in issuer labels, discovery, and schema migration 0079. Deploy and drain all access writers before enabling publication; use a stable relay key and a shared publication-policy revision. Update clients before enabling assertions with `full_name`. ### Related issue Implements the Relay portion of [the verified-profile specification](https://github.com/block/buzz/pull/8745) at `500829a9` (BUZZ-433). This is the closest existing PR found. Companion client: https://github.com/block/buzz-app/pull/2005. Companion issuer: [KGoose/bb-identity #144773](https://github.com/squareup/cash-server/pull/144773), which includes `nostr_public_profile` in valid human assertions. ### Testing Passed the lifecycle test against a disposable local PostgreSQL database, including full-name addition, change, and removal: `cargo test -p buzz-db store::verified_profiles::postgres_tests::profile_lifecycle_retains_assertion_ordering --lib -- --ignored --exact` with `BUZZ_TEST_DATABASE_URL` set to that database. Independent agent review found no remaining blockers. No live Relay test or human verification yet. This remains a draft and does not attest the repository's human-testing checklist. For end-to-end testing, enable publication for a test issuer, admit a member with a signed profile claim, and query kind 30382. Verify short display handles, optional full names, changed names, missing-claim retraction, disconnect cleanup, and a later admission surviving delayed cleanup. Check the companion app against the same relay. Generated with Codex --------- Signed-off-by: Franco Sola <fsola@squareup.com> Signed-off-by: Bradley Axen <baxen@squareup.com> Co-authored-by: Bradley Axen <baxen@squareup.com> Co-authored-by: mbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com> |
||
|
Logan Johnson
|
a56966b777 |
feat(relay): NIP-AR sync subscriptions (#8522)
🤖 ## Summary Clients that show a list of artifacts (tasks, config entries and similar records) had no way to get the complete list and then stay current. They could only page through query results, and a page is not proof that the list is complete. This PR adds **sync**. A client sends one subscription and gets every matching artifact as one consistent snapshot, then a signal that the set is complete. After that it gets the current version of each artifact that changes. When the relay cannot guarantee an exact copy, it closes the subscription with a `resync` reason, so the client never misses a change without knowing it. It also adds a **head lookup**. It returns the current version of named artifacts, deleted ones included, so that "create or restore" flows can retry safely. This implements the Sync section of NIP-AR (the artifact protocol spec), which is the first commit of this PR. ## Details - **Request.** A normal `REQ` whose only filter has `"artifact": "current"` and at least one channel in `#h`. This is the same current-state filter as HTTP `/query`; on a `REQ` it becomes a sync, because a `REQ` already means "stored set, `EOSE`, then live". The client does not page: the relay streams the whole set until `EOSE`, so `limit` and `offset` are refused. Other artifact views (`history`, `head`) stay HTTP-only, as on main. Other clients and other `REQ` shapes do not change. - **Initial set.** The relay first registers for live changes and waits until Redis confirms that registration. Then it reads every matching current version in pages ordered by artifact ID, all inside one read-only `REPEATABLE READ` transaction. So all pages come from the same database snapshot. `EOSE` comes only after a complete read. A failed read, an unreadable channel, or a read that takes too long ends with `CLOSED` and no `EOSE`. - **Live changes.** Fan-out never sends the published event to a sync. It only records which artifact changed. The sync then reads that artifact's current version and sends it, one artifact at a time. So a client never gets an older version after a newer one. Changes that arrive during the initial read are held and sent after `EOSE`. Changes are signalled per channel, so a sync skips an artifact whose `type` or `d` does not match its filter: those never change, so it was never in the set. A board that syncs three record types in the same channels no longer gets each change three times. - **When the relay sends `resync`.** The Redis link drops, reconnects, or skips a message it cannot decode. Local fan-out lags. The connection's send queue is full. A moderator redacts an event in the channel. The reader's channel access changes (remove, leave, metadata edit, channel delete). An access change ends the whole sync, also when the sync names other channels. A sync cannot drop one channel and still claim an exact set. - **Publish retry.** When a Redis publish fails for an event kind that syncs use, the relay retries it in the background until it succeeds. - **Relay info (NIP-11).** The relay now lists `sync` (`max_events: 4096`), adds the `head` mode, and sets `websocket_query_extensions: true`. - **No migration.** Sync reads the existing `artifact_heads` table. - **Next to incremental reads (#8523).** #8523 added a separate HTTP protocol, `artifacts.incremental` / `"artifact":"incremental"`. It keeps a saved store up to date after a disconnect, but it does not prove a complete set. The two protocols use different NIP-11 keys and filter values, so they do not collide. They share one copy of the current-state access rule: `artifact_query::predicates`, which #8523 reads through, now applies the view join and calls `push_scope`, and the sync snapshot and head lookup call `push_scope` too. - **A failed access lookup in fan-out resyncs.** The access gate fails closed when a visibility or membership lookup fails. For a sync, that would drop a change with no signal, so the gate also reports those recipients and each one gets `resync: access check failed`. ### Known cost To find out which redactions remove an artifact that a sync covers, the relay would need an extra database read. So every redaction in a synced channel resyncs that channel's syncs, and so does every channel metadata edit. Each resync reads the set again, up to 4,096 artifacts. This is correct, but it adds load to channels where people often delete chat messages. We can narrow it later if it becomes a problem. ### Related issue None found. The design was reviewed as NIP-AR Sync r28. ### Testing - New Postgres tests: keyset pages see one snapshot, and an artifact created after the snapshot opens is in no page; the head view includes deletions; a private or unknown channel is refused. - New relay end-to-end tests on real Postgres and Redis, using the real `REQ` handler, ingest and fan-out: initial set in order, then `EOSE`, then current versions for an update and a delete; a chat message causes no frame; `CLOSE` stops the sync. Other tests: removing the reader from one channel resyncs the whole sync; an unreadable or unknown channel gets `CLOSED restricted` with no `EOSE`; a node-wide loss resyncs. - Unit tests for the parser, request routing, fan-out diversion, the held-change limit, and access-change signals that must arrive before the access filter. - Lifecycle tests from an adversarial review: every frame is checked against the subscription owner, also when a `CLOSE` or a replacing `REQ` arrives while the frame waits for queue room or while a live head read is in progress. A resync recorded while `EOSE` waits ends the sync with no `EOSE`. The held list refuses a change before it stores it (4096 entries, 4 MiB of held removals). - End to end: a relay container built from this branch, with the buzz-app #1523 browser build and the Work plugin. An edit from another client showed in the open board 4 to 25 ms after the write returned, with no refresh. - `cargo clippy --all-targets -D warnings` is clean for the four changed crates. All `buzz-relay` lib tests pass except `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`. That test returns 504 on this machine. It is a mesh networking test, and this PR does not touch that code. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Daniel Rouleau
|
a57349b237 |
feat(relay): add incremental artifact reads (#8523)
## Summary Buzz clients need to recover task changes after a disconnect. This adds incremental reads of changed artifacts with database modification timestamps and a 120-second overlap. Each result contains the current revision or an explicit removal. A separate read of known IDs confirms accepted edits and restores access state. Pages use a timestamp and a stable ID to handle equal timestamps. Access grants include older records. Access policy changes require discovery and revalidation of saved IDs. The reader does not load a list of all readable channels. Ordinary artifact writes update their own rows. ### Incremental reads and WebSocket Sync This HTTP protocol maintains an eventually consistent local store. It uses `artifacts.incremental` and `"artifact":"incremental"`. It does not prove a complete set at one database snapshot. Absence never proves deletion or loss of access. [WebSocket Sync](https://github.com/block/buzz/pull/8522) serves a separate use case: a complete set at `EOSE`, followed by live updates. It owns `artifacts.sync`. Both protocols can coexist. The second PR to merge must reuse the shared access helper in `artifact_query.rs` when it resolves the overlapping edits. The gate is `BUZZ_ARTIFACT_INCREMENTAL_ENABLED`. Errors use the `incremental_` prefix. Unlisted open channels require membership or an explicit `#h` scope. Listing changes reset incremental progress. Incremental reads exclude TTL artifacts. Known-ID reads return `remove` for them. The `#h` scope limits candidates before pagination. Initial reads omit deleted heads. Later reads retain deletion removals. ### Why the new timestamps are needed `created_at` comes from the event signer. `received_at` records event arrival. Neither advances when access changes or an event is redacted. Sync needs database timestamps for those changes: | Column | Why sync needs it | | --- | --- | | `artifact_heads.modified_at` | Finds current records that change through an edit, deletion, or redaction. | | `artifact_revisions.modified_at` | Dates retained move records. This lets sync remove a task from its former channel when the reader cannot access its new channel. | | `channel_members.sync_modified_at` | Finds grants, rejoins, and removals. A grant must include older tasks whose own timestamps do not change. | | `communities.artifact_policy_modified_at` | Requests recovery after a relevant channel visibility, TTL, or deletion change. Clients revalidate saved IDs. Ordinary task writes do not update it. | The database sets these timestamps with `clock_timestamp()`. The 120-second overlap covers the accepted delay between timestamp assignment and commit visibility. `BUZZ_ARTIFACT_INCREMENTAL_ENABLED` defaults to off. Apply migrations `0076`–`0078` and upgrade all writers before enablement. The 120-second delay bound is an operating assumption. [Deployment steps](https://github.com/block/buzz/blob/drouleau/artifact-snapshots/docs/artifact-sync-deployment.md) cover index prebuilds and rollback. ### Related issue No duplicate found. Related work: [read permissions](https://github.com/block/buzz/pull/8338), [statement limits](https://github.com/block/buzz/pull/8461), and [atomic channel settings](https://github.com/block/buzz/pull/8516). ### Testing Signed HTTP E2E passes on `f3d6ac852e7ce4e2cc53bd891cef87832a868d72` after CI passes. It includes main through `ef4130da4`. A standalone local relay uses a fresh PostgreSQL 17 database, Redis, and signed NIP-98 HTTP requests. All 78 migrations apply through relay startup. All behavioral writes and reads use HTTP. The tests cover create, edit, delta reads, known-ID recovery, private moves, access grants, removals, rejoins, visibility resets, redaction, and deletion. Unlisted tests cover hidden broad discovery, named scope, known-ID reads, member access, and policy resets. They also cover bootstrap across two pages, overlap replay, and 101 move removals across pages. Decoded cursors contain no private revision IDs. With the incremental gate off, `artifacts.incremental` is absent and a signed incremental request returns `503 incremental_unavailable`. Unsigned requests are rejected. Signed tests also cover TTL exclusion, current-ID removals for TTL artifacts, moves into TTL channels, deleted initial heads, and scoped pages with exactly 100 and 3 upserts. A separate local scale test on `babcdb766` reads 10,050 tasks across 60,002 memberships. It uses seeded fixtures and a second relay state with the same signing key. The initial read takes 8.31 seconds over 101 pages with 4,923-byte events. A read with no changes takes 5.70 milliseconds. The candidate keeps the same database query. These are local measurements, not production limits. Cold pages repeat candidate scans and access checks. Writer load needs validation before fleet enablement. ```sh cargo test -p buzz-relay sync_downloads_ten_thousand_tasks_with_sixty_thousand_memberships -- --ignored --nocapture ``` The behavioral run uses `cargo build -p buzz-relay --bin buzz-relay`, then `cargo run -p buzz-relay --example artifact_incremental_final_oct30_f3`. The temporary harness drives the built relay and is removed after the run. The worktree is clean at the tested SHA. It does not exercise a native client. Git and S3 behavior are outside this validation. The [protocol](https://github.com/block/buzz/blob/drouleau/artifact-snapshots/docs/nips/NIP-AR.md#incremental-reads) defines progress, removals, limits, and recovery. Relay verification covers the signed HTTP path. Native client verification belongs to [the client PR](https://github.com/block/buzz-app/pull/1507). Generated with Codex --------- Signed-off-by: Daniel Rouleau <drouleau@squareup.com> |
||
|
Luke Tornquist
|
cc04cdca1a | perf(relay): route author- and #p-scoped global subscriptions by selector (#8666) | ||
|
Luke Tornquist
|
0e8fb49bd0 | test(relay): fix main's build after the test-state helper rename (#8718) | ||
|
Matt Toohey
|
ef4130da4b |
fix(db): keep lease releases durable and deflake PostgreSQL lane tests (#8676)
## Summary
When the user-deletion worker dropped a lease heartbeat, its pooled
connection could be left inside an open transaction. The worker's final
lease `release` then reported success but never committed, and a rerun
saw the request as "leased by another worker" for up to 60 s. The
heartbeat is now one autocommit `UPDATE`.
The PR also fixes six PostgreSQL-lane test flakes from the 21–28 Oct
2026 CI audit. It moves one relay test off the 100 ms-acquire test
helper before that test starts flaking the same way.
This is one of seven PRs split out of the `flakey-tests` branch
(siblings below). It touches only `buzz-db`, `buzz-deletion` and
`buzz-relay`. Every relay file except `connection.rs` and
`boot_lifecycle.rs` changes only for the helper rename in the last row.
### Related issue
None found. Nothing on `main` (
|
||
|
Luke Tornquist
|
efcdf00101 | relay: open the search pool lazily so a dead reader can't block boot (#8662) | ||
|
Tom Brow
|
4b21a10c4b |
fix: don't notify workflow owners through their attribution tag (#8694)
A workflow message names its owner in a marked p tag, ["p", owner, "", "owner"], for attribution. #p live, history and push matching, operator listeners, desktop, mobile and the iOS NSE treat that tag as attribution rather than a mention. A workflow that mentions its owner adds a plain p, so the owner is notified only when actually mentioned. NOSTR.md records the #p exception. Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz> Signed-off-by: Tom Brow <tomb@squareup.com> |
||
|
Luke Tornquist
|
1f16f621d8 |
perf(relay): cut connect-burst CPU in log filtering and open-channel lookup (#8670)
## Summary
A cold relay taking a burst of new connections spent most of its CPU on
two things that don't scale with useful work. This PR removes both.
**Log filtering held a contended lock.** `EnvFilter` (tracing-subscriber
0.3.23) takes a shared `RwLock` read on every span enter and exit, even
when no directive names a span. Every poll of an instrumented future
enters its span, and the relay installs two per-layer `EnvFilter`s (logs
and OTEL), so every worker contends on those locks. Plain `target=level`
directive lists now filter through
`tracing_subscriber::filter::Targets`, which keeps no per-span state.
Directives that name spans or fields still use `EnvFilter`, which is the
only filter that understands them.
**Each reader re-read the community's open channels.** A reader's
accessible channels are its member channels plus the community's listed
open channels, and the open half is the same for every reader. The
per-reader cache stored the union, so every miss fetched every open
channel row again. The lookup is now two cached halves:
- member channels, per reader, cached as before;
- open channels, once per community.
The shared half is used for a bounded time. Cross-pod invalidation is
best-effort, so a pod can miss the drop that follows a channel becoming
private, unlisted or deleted elsewhere. Each cached open set records
when its database read began, and readers use it only for
`OPEN_CHANNELS_MAX_AGE` (1 s) after that. After that the next reader
re-reads it. **Accepted trade-off:** if an invalidation is lost, any
reader on that pod can be granted a just-revoked open channel, history
included, for at most 1 s. That includes a reader with no cache entry,
which on `main` always read the database. This is a new exposure for
those readers. With no bound, the shared set could have served that
channel for its 10 s TTL. Readers with a cached entry already had a 10 s
window on `main`, so for them the bound is tighter than before. An exact
per-community version check would remove the window but needs a schema
change, so it's tracked as a follow-up.
Invalidations still drop the halves. A community-wide accessible-channel
or channel-deleted invalidation drops both halves, and a membership
invalidation drops only that reader's member half. The age bound counts
from the start of the read, so open-set fills need no fence: a fill that
raced a drop is still at most 1 s stale. Uncached callers of
`get_accessible_channel_ids` get the same set as before, now built from
the two halves. Open-set reads are counted by
`buzz_open_channels_cache_hits_total` and
`buzz_open_channels_cache_misses_total{reason="absent"|"expired"}`.
**Metric meaning change:** `buzz_accessible_channels_cache_hits_total`
and `buzz_accessible_channels_cache_misses_total` keep their names but
now count only the per-reader member half. Dashboards that read them as
whole-lookup hit rates should add the open-set counters above.
Empty levels (`target=`) stay on `EnvFilter`. `Targets` reads an empty
level as ERROR while `EnvFilter` reads it as TRACE, so only
`target=level` directives with a non-empty level take the `Targets`
path.
## Results
Measured with the relay load generator (#8636) on one Linux machine. The
relay ran as a release build on 8 pinned cores, with local Postgres,
Redis and MinIO; the generator ran on separate cores. The fixture is one
community with 2,000 open channels. Clients connect in a burst against a
cold relay: 48k WebSocket clients and 4.8k HTTP agents.
Each arm restored the same database snapshot and ran the same generator
binary and scenario. Arms were rotated (base/before/after, then
before/after/base, then after/base/before), and the table shows medians
of three runs per arm. Both columns after base also carry #8666, so the
comparison isolates this PR's design.
- **before** is this PR's first revision, which had an unbounded shared
open set.
- **after** is this head, where the open set is bounded to 1 s.
| | base | before | after |
|---|---|---|---|
| relay cores (mean) | 6.37 | 2.39 | 2.45 |
| connect p99 | 3,870 ms | 40 ms | 42 ms |
| subscribe EOSE p99 | 12,952 ms | 110 ms | 112 ms |
| history EOSE p99 | 1,252 ms | 68 ms | 72 ms |
| live delivery p99 | 26,421 ms | 54 ms | 55 ms |
| HTTP query p99 | 16,426 ms | 44 ms | 43 ms |
| DB tuples returned/s | 1.04M | 43k | 49k |
| DB transactions/s | 4,397 | 4,065 | 4,062 |
| relay RSS peak | 12.6 GB | 10.7 GB | 10.7 GB |
| errors | 16, plus 241k throttled | 0 | 0 |
The 1 s bound costs about 1,400 extra open-set reads per run, within
run-to-run noise of the unbounded version. One run in each of the two
right-hand arms had a slow tail (connect p99 0.8–1.3 s), which the
medians absorb.
An earlier revision bound the shared set to a per-reader digest instead.
That was exact, but in the same kind of A/B its per-miss digest query
returned about 28 times the tuples per second of the unbounded version.
It gave back the database savings, so this revision uses the time bound.
The open-channel gain scales with how many listed open channels a
community has. The filter gain doesn't depend on the fixture.
## Rollout
There's no flag, so the change takes effect on deploy. There's no schema
or data change, so reverting the commit is the rollback.
1. Deploy to staging and compare relay CPU and connect and EOSE p99
under a connection burst.
2. Watch `buzz_open_channels_cache_misses_total{reason="expired"}`.
Expect at most about one per second per active community per pod.
3. Confirm the deployed `RUST_LOG` and `BUZZ_OTEL_FILTER` are plain
`target=level` lists, so they take the `Targets` path. Other values fall
back to `EnvFilter` and keep today's cost.
## What got simpler
- `invalidate_channel_deleted_local` now reuses the community-scoped
accessible-channel drop instead of repeating it.
- Tests seed accessible channels through one `seed_accessible_channels`
helper instead of writing cache internals.
## Tests
- `telemetry::tests::only_plain_target_level_directives_use_targets`
checks which directive strings are plain. It covers span directives,
field directives, whitespace, empty segments, empty targets (`=debug`)
and bad levels.
- The rewritten `env_filter_tests` assert on captured log output rather
than `Display` strings or cached callsite interest, so they are stable
when run in parallel. `span_directives_keep_env_filter_semantics` proves
that `[conn]=debug` still enables events only inside the span.
-
`state::tests::postgres_tests::a_lost_open_set_invalidation_grants_a_revoked_channel_only_until_max_age`
runs two relay states on one database, with no pub/sub between them. For
each revocation (private, deleted, unlisted), it:
1. primes the reader pod;
2. revokes the channel in SQL and invalidates only the writer pod;
3. shows that a new reader is still granted the channel within the
window;
4. ages the cached set past the limit;
5. asserts that a new reader and an earlier cached reader are both
denied.
Dropping the age check fails it. Never caching the open set fails its
within-window control.
- `env_filter_tests::directive_filters_log_what_env_filter_logs` emits
events at every level for six targets, including prefix lookalikes. It
asserts that the filter built for each of 17 directive lists logs
exactly what `EnvFilter` logs for that list. Routing `buzz_relay=` to
`Targets` fails it.
- `state::tests::accessible_channels_join_member_and_open_halves` checks
the merge. The two existing scoped-invalidation tests now also assert
the open half.
- These state tests, and the two existing ones, were not in the `just
test-unit` allowlist. They are infra-free, so they are now selected by
exact name, and `scripts/run-tests.sh` mirrors that.
Identified debt, not fixed here: the per-reader member half keeps the
pre-existing unfenced read-then-insert (a raced fill only reaches that
reader, as before).
`handlers::req::tests::expiry_during_stalled_{history,search}_read_releases_permit_and_claim`
are not selected by any CI lane.
buzz-review-completed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Brad Seiler
|
1fe120165f |
BUZZ-272: Deny workflow runs for banned owners (#8242)
## Summary A workflow whose saved owner is banned from the community can no longer create runs through any trigger: webhook, channel-event, cron, or signed command trigger. Every one of those doors already calls `WorkflowEngine::check_owner_authority`, which re-checks the owner's channel role. That gate now also reads the owner's authoritative restriction state. A ban denies the run, and a failed lookup fails closed. Unbanning restores admission. On the webhook door the denial is the same generic 404 as before, and no run is queued. On the event and cron paths, a ban denial is logged at WARN at most once per workflow every ten minutes, and at DEBUG in between. A failed lookup and every other denial still warn every time. The handler-local ban lookup in `api/bridge.rs` is removed because the shared gate covers it. This is a point-in-time check: a ban committed after the lookup can race with run creation. It does not cancel existing runs. Community timeouts do not block runs in this change. This PR also carries the shared mesh receive fix needed for independent validation (see Shared carry). ### Related BUZZ-272. - Invitations: [#8237](https://github.com/block/buzz/pull/8237). - Route coverage: [#8239](https://github.com/block/buzz/pull/8239). - Live-session revalidation: [#8241](https://github.com/block/buzz/pull/8241). - Staging runbook: [#8235](https://github.com/block/buzz/pull/8235). ### Testing Postgres lane: - `on_event_denies_run_while_owner_banned`: a banned owner's channel-event trigger creates no run, and after unban it creates one. - `cron_tick_skips_banned_owners_without_consuming_the_claim`: one tick skips a banned owner and an agent whose human owner is banned, while an unbanned workflow on the same tick still fires. After unban, a tick for the same scheduled instant fires both, so the denial did not consume their scheduled-fire claim. - `on_event_denies_run_when_restriction_lookup_fails`: the event path fails closed when the restriction read errors, and fires once the read succeeds. - `webhook_admission_enforces_owner_ban_tenant_scope_and_lookup_failure`: the HTTP door returns the same generic 404 for a wrong secret, a banned owner, a forbidden role and a failed restriction lookup, with no run created. It also covers another tenant's allowed control and unban. Unit lane: - `ban_denial_warns_once_per_interval_while_other_denials_always_warn`. Results: - At `81c367f48`, CI passed: Unit Tests 572/572, PostgreSQL Tests 1169/1169 and Relay E2E. A local-relay E2E also passed: - unbanned webhook: 202 and one run; - banned webhook: generic 404 and no run; - banned event trigger: no run; - after unban: one run. - At `72c8d8bbc` (merged with main), CI passed: Unit Tests (including all 171 buzz-workflow unit tests), PostgreSQL Tests 1227/1227 (all four ban tests listed by name), and both `Build (linux/amd64|arm64)` image jobs, which build with `--locked`. A Blox run at the same head also passed `cargo metadata --locked`, fmt and clippy, and the buzz-workflow unit tests passed there with no database reachable. - Not covered: the signed trigger has no dedicated test. It calls the same gate, which was confirmed by reading the code. ### Shared carry #8237, #8239, #8241 and #8242 each carry the same mesh-receive fix, boot and scraper regressions, and push-lease fixture correction. Whichever lands first owns them, and the others drop them on their next main merge. ### What got simpler The ban check now lives in the one gate every run-creation door already shares, so there is one fewer handler-local branch. Cron, event and signed triggers get the same protection without per-door code. The cron loop body is now `cron_tick(now)`, so a test can drive one tick without the 60-second sleep. buzz-workflow's infra-free unit tests ran in no CI lane. `just test-unit` and its cargo fallback in `scripts/run-tests.sh` now run them. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
76cd4294f6 |
fix(relay): close a REQ whose history read failed instead of sending EOSE (#8692)
## Summary **Merge order:** this lands and deploys **before** [#8688](https://github.com/block/buzz/pull/8688). That PR's buzz-acp resume cursor counts on a failed history read never ending in EOSE. Until #8688 ships, buzz-acp re-sends a REQ as soon as it gets an error CLOSED. The relay's REQ rate limit bounds that loop, and buzz-acp already parks on a rate-limited CLOSED. #8688 replaces this with a per-subscription backoff. When a historical query fails for a reason other than a statement timeout, the relay logs the error and still sends `EOSE`. EOSE means "you have every stored event" (`ARCHITECTURE.md` §5, NIP-01). A client that resumes from its last-seen point therefore treats the failed read as complete, advances past it, and never replays that history. This is RPR #1: a failure must not become an authoritative empty result. The history task now returns the CLOSED reason to send: - **Statement timeout:** `["CLOSED", sub_id, "error: query timed out"]`, unchanged. - **Any other DB error:** `["CLOSED", sub_id, "error: database error"]`. - **Cleanup:** CLOSED goes through the existing `close_if_owner` path, so the subscription, its sub-registry filters and its topic refcount are released, the same as on a timeout. - **Docs:** `ARCHITECTURE.md` §5 now states this contract. **Clients don't hot-loop on it.** Checked at `main` |
||
|
ravarora2
|
8caf8351c1 |
Add standalone owned-community deletion handlers (#8645)
## Summary
User deletion needs to delegate owned communities to the existing
community deletion process and report what is still waiting. This adds
the standalone components for that flow; wiring them into the user
workflow is a later change.
The owned-community handler checks for an existing community deletion
request first. If none exists, it calls `submit_and_prepare`, the same
function used by the community CLI. The handler saves the child request
ID, last observed status, and operator instructions on the existing user
stage. Community approval and execution remain separate. Repeated or
concurrent calls reuse the existing child without resetting its approval
or execution state. The user stage follows the current request for its
approved community ID. If an operator aborts A and submits B for that
community, the next invocation saves B on the same stage row.
An unfinished child leaves the user stage waiting. A later handler
invocation refreshes its observation and completes the stage when the
child reaches logical completion (`retention_pending` with a completion
timestamp). Retained-copy obligations remain visible in the saved
result. Lease owner and generation checks protect stage updates.
The separate summary handler reads only the user request, approval, and
stage tables. It lists unfinished stages, saved external waits and
operator actions, completed results, and inventory communities not yet
reached. It never queries or refreshes community deletion state, so
child statuses are explicitly labeled as last observed.
### Scope and rollout
- No migrations, new HTTP endpoints, CLI commands, or worker dispatch
changes. Owned-community intake remains disabled.
- Waiting uses the existing stage status plus `metadata_json.waiting`.
This slice does not advance past waiting stages. Integration must add an
explicit resume or bounded revisit cadence before enabling automatic
scheduling.
- The CLI and user handler share the existing database submission,
inventory building, and inventory freezing operations. There is no API
wrapper class or second insertion path. Preparation failures leave a
durable child request and a retry record on the user stage; subsequent
lookup exposes instructions for the operator to finish preparation.
Guidance distinguishes operator-origin preparation, automatic
owner-origin preparation, blocked requests that need remediation and
`unblock`, and aborted requests that can be replaced.
- Existing submission accepts a host for the CLI or an immutable
community ID for the parent. A transaction protects target resolution
and admission. ID-pinned calls never replace an existing child, even an
aborted one. Inventory helpers take their store and media dependencies
directly. Successful stage persistence clears any saved waiting marker.
- These components become available to callers after release but are not
scheduled by the relay or user worker. No data migration is needed to
roll back this change.
See [the implementation and testing
guide](
|
||
|
Brad Seiler
|
d6196a16c1 | fix(relay): revoke invitations and restrict claims (#8237) | ||
|
Luke Tornquist
|
0a362eef15 |
perf(db,relay): bound thread participant scans and coalesce live summaries (#8635)
## Summary A thread summary's `participants` (up to 10 recent authors, used for the facepile) was computed by aggregating **every** reply in the thread. The live `kind:39005` emit runs that aggregate on the writer after every reply insert and every threaded delete. Each summary therefore cost O(thread), recomputed after every reply, so one very large thread could saturate the writer. ### Scale and impact A single long-running thread can grow to hundreds of thousands of replies and take several new replies per second. Each of those replies then pays a whole-thread aggregate on the writer, so writer CPU follows the reply rate of the largest active threads, and the read-side participant queries (channel window, thread root summaries) slow down the same way. Synthetic reproduction (PostgreSQL 17, the repo's `schema/schema.sql`, default `shared_buffers`): one channel with 200k background messages and three threads of 450k, 6k and 300 replies from a few authors with ~800-byte bodies. `EXPLAIN (ANALYZE, BUFFERS)` of the participant query, warm cache (runs 2–3 of 3): | Thread | Before (whole-thread aggregate) | After (newest 1000 replies) | |---|---|---| | 450k replies | ~7.6 s, ~117k buffers read, ~16k temp blocks spilled | ~6.8 ms, ~4k buffers | | 6k replies | ~25 ms, ~24k buffers | ~7 ms, ~4k buffers | | 300 replies | ~2.6 ms, ~1.2k buffers | ~2.6 ms, ~1.2k buffers | Cost now stops growing at 1000 replies, and small threads cost the same as before. With coalescing, a busy thread runs this query at most about once per second per pod instead of once per reply. ### Change 1. **Bound the participant scan to the newest 1000 replies.** All three participant readers now share one SQL fragment (`recent_thread_participants_sql!`): - `get_thread_summary` (the live 39005 emit) - the channel-window batch - `resolve_thread_root_summaries` The fragment walks `idx_thread_metadata_window` newest-first with `LIMIT 1000`, then joins `events` only for those rows, to get `pubkey` and drop deleted replies. Deleted replies keep their `thread_metadata` row, so they count toward the 1000 but contribute no author. It aggregates by author and returns the 10 most recently active. Cost no longer grows with thread size. Desktop and mobile show the 3 most recent participants and the forum view shows up to 10, so an author who hasn't posted in the last 1000 replies is not visible in any client today. `NIP-CW`, `docs/bridge-channel-window.md` and the `ThreadSummary::participants` doc now state the bound, deleted replies included. 2. **Coalesce live summary emits per thread root.** `emit_live_thread_summary` goes through a new `CoalescedRefresh` keyed by `(community, root)`: - A quiet thread emits immediately, as before. - On a busy thread, emits are spaced at least 1s apart per pod. A trailing emit covers the rest of the burst and reads state committed after its last mutation. - Because emits are at least 1s apart, consecutive overlays from one pod always get strictly increasing `created_at`, so the client's newest-wins replacement is never tied. - A panicking or cancelled refresh releases its key, including a task dropped before its first poll: the release guard is built before the spawn. Each run is bounded by a 30s timeout so a hung refresh can't wedge it. ### What got simpler - Three hand-written copies of the participant aggregate are now one fragment. - `get_thread_summary` is one statement instead of two round trips. - The channel-window batch dropped its `ROW_NUMBER()` window function and the `GROUP BY root, pubkey` over every reply of every root on the page. ### Related issue None found. ### Testing - `participant_summaries_read_only_the_newest_replies` (buzz-db, Postgres) seeds a thread with 1001 replies: - an early author, pushed just outside the window - a filler author - a deleted reply - a latest author All three readers must return `[latest, filler]`. Mutant check: removing the `LIMIT` (whole-thread aggregate) turns it red. - `coalesced_refresh` unit tests (paused tokio clock): - A 100-request burst runs once immediately and once trailing, then frees the key. - Separate keys run independently. - A panicking refresh doesn't wedge its key. - A hung refresh times out and releases its key. Mutant check: removing the timeout turns it red. - A request that arrives while a run is in flight gets one trailing run, and that run reads the state written mid-run. Mutant check: dropping the rerun flag turns it red. - A task dropped before its first poll (runtime shut down) still releases its key. Mutant check: building the guard inside the task turns it red. These run in `just test-unit` (added to the buzz-relay `--lib` filter and `scripts/run-tests.sh`). - Blox gate: `cargo fmt --check`, clippy `-D warnings` (buzz-db, buzz-relay), and the full buzz-db and buzz-relay Postgres nextest lanes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Logan Johnson
|
646ef8abf9 |
Stop requiring the 0032 roster trigger at relay startup (#8611)
🤖 ## Summary When a relay starts, it checks that a database trigger from migration 0032 is installed and works. If the trigger is missing, the relay refuses to start. This PR removes that startup check, so a later migration can drop the trigger. The trigger blocks a stale channel member list from being saved. Since #8316, the relay runs the same check in Rust before every save of a member list. The trigger is now only a duplicate backstop. Removing it lets #8494 compute the `member`/`bot` label when it reads members, instead of storing `bot` in the database. This PR does not drop the trigger. The trigger stays installed and keeps working. Only the requirement that it exists at startup is removed. ### Related issue Prepares the trigger retirement that #8494 needs (review thread on #8494 about storing `bot`). No duplicate PRs found. ### Rollout `docs/database.md` requires this sequence for retiring a trigger: 1. Ship the replacement in the app. This was #8316. 2. Fully deploy a release in which no relay requires the trigger. **This PR.** 3. Drop the trigger in a separate migration, and name the step-2 release as a required stop in its release notes. A relay from before this PR refuses to start after the trigger is dropped. So every environment, staging included, must run this release before the drop ships. ## Details - Removed the startup step `channel_roster_fence`, which ran the catalog check and a behavior probe (a test insert that must be rejected, then rolled back). - Removed the catalog check that ran after migrations in `run_migrations_locked`. - The catalog check function stays, but only tests use it. - Removed four tests about the startup gate. Added `migrations_run_without_the_roster_trigger`: it drops the trigger in a scratch database and runs the migrations. If the post-migration check comes back, the test fails. - `observability_source` lists the event writers that may skip the guard. The probe was one of them, so it is removed from those lists, and the expected writer count goes from 9 to 8. - The Helm chart README now says that newer relays do not require the trigger. ### Testing - `cargo clippy -p buzz-db -p buzz-relay --all-targets -D warnings` and `cargo fmt --check` pass. - `observability_source` (21) and `startup_steps` (6) pass. - `migrations_run_without_the_roster_trigger` passes. With the post-migration catalog check restored, it fails with "channel roster fence trigger missing". 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
78b5dae1f6 |
fix(relay): store a kind:9000 and its membership writes in one transaction (#8606)
Follow-up to #8575. ## Problem Today a kind:9000 is stored first and the membership write runs afterwards, on its own. If that write fails, the relay logs it and still returns an accepted OK. A resend of the same event then returns `duplicate:`, so the person is never added. This is the catch-log-and-accept failure that Review-Proven Rules #1 and #5 describe. ## Change kind:9000 now uses the same atomic path as kind:9002 (`persisted_atomically` in ingest). - **`buzz-db`:** new `put_members_with_event`, which runs one transaction: 1. Take the channel's membership lock, before the event insert, the TTL lock and the channel row. This matches the order the other membership writers use, so it adds no new lock cycle. 2. Store the event. If it's already stored, roll back and return `Duplicate`. 3. Run the existing `put_members` decisions and upsert. These now live in a shared `put_members_on` step on the transaction. 4. Commit. - **Failure:** nothing is stored and the client gets an `error:` rejection, so it can resend the same event and have it applied. - **Everyone refused:** if the decisions under the lock refuse every person, nothing is stored and the event is rejected with the same text validation gives. Before, a single-person add refused at that point was accepted with an empty message. - **After commit:** the live effects stay as they were. They're best-effort and read the committed state: cache invalidation, `member_joined` notices, member-added notifications and 39002 discovery. - **`NOSTR.md`:** now says a failed kind:9000 can be resent as is. ## What got simpler - kind:9000 no longer has its own store-then-apply path in ingest. It shares the atomic branch with 9002 and workflow deletion. - `apply_put_user`'s "write failed, report it as a per-person rejection inside an accepted OK" branch is gone. - The `Db::put_members` wrapper is deleted. Its only caller was the kind:9000 path. ## Identified debt (not in this PR) kind:9001 (remove), 9021 (join) and 9022 (leave) have the same gap. Each stores the event, then runs the membership write in `handle_side_effects`, and a failure is only logged. kind:9008 and 9005 follow the same store-then-apply pattern. These can move onto the same atomic shape in a follow-up. ## Tests - `failed_put_user_stores_nothing` (relay, Postgres): a test trigger fails the second person's row in a two-person add, after the event row and the first person were already written in the same transaction. It checks: - Both the send and a resend are rejected with `error:`. - No kind:9000 is stored, nobody is added, and no `member_joined` notice is posted. - With the trigger dropped, the same signed event adds both people. - One more resend returns `duplicate:` and posts no second notice. - `put_members_with_event_stores_the_event_with_its_writes` (buzz-db): checks the three outcomes: - Refused: nothing is stored. - Applied: the event is stored and the person is written. - Duplicate: the roster is left alone. - `put_user_refused_under_the_lock_stores_nothing` (relay, Postgres): a test trigger removes the sender from a private channel inside the write transaction, so validation passes and the decision under the lock refuses. For a single add and a batch, it checks the `invalid:` rejection, that nothing is stored and no notice is posted, and that the same event applies once the trigger is gone. - Mutation checks, run in Blox gates: - At `d18e88fd`: committing the refused event fails the buzz-db test; applying a duplicate fails the first two tests; storing the event in its own transaction fails the first two tests. - At `3b106623`: turning the refusal under the lock into `error:` fails the new refusal test, and so does committing the refused event. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Will Pfleger
|
24a3706060 |
feat(relay): log who and why for NIP-FI shadow would-deny (#8634)
In shadow mode, the relay's `nip-fi shadow would-deny` line now says which client enforce would refuse and why, not just the coarse public outcome. - The line moves from `debug` to `info`. Admits still write no line. - It gains `reason`, one value from a fixed snake_case set naming the step's private condition: `community_unresolved` (community resolution failed or was unavailable), `community_stale`, `session_deadline`, `duplicate_authorization`, `missing_proof`, `invalid_proof`, `missing_assertion`, `malformed_assertion`, `verifier_unavailable`, the verifier's own error code without its `nip_fi_` prefix (for example `expired`, `unknown_issuer`, `invalid_signature_or_claims`, `token_type_rejected`), `key_mismatch`, `no_asserted_key`, `deny_set`, `owner_banned`, `not_stored_owner`, `owner_ban_unavailable`, `owner_link_unavailable` and `tenant_unavailable`. - It gains `pubkey`, the hex Nostr key of the would-be-refused client, or `absent`. On HTTP it is the verified NIP-98 proof key, for every step including community and cardinality. On WebSocket and audio it is the key NIP-42 AUTH proved, or for deny-set and deadline the key a verified token asserts. A router-guard or upgrade would-deny precedes any proof, so its line names the key the client later proves: the request's verified NIP-98 key, or the connection's first verified NIP-42 AUTH key. Each such line is written exactly once. On HTTP it is written when the handler verifies its NIP-98 proof, or as `absent` when the request ends without one. On WebSocket and audio it is written at the first verified NIP-42 AUTH, while the connection stays open, or as `absent` when an unproven connection's read loop ends (close or auth timeout), before any cleanup that can block. Its counter still increments at once. An unsigned `X-Pubkey` never counts. - `buzz_nip_fi_shadow_total` keeps exactly its `route`, `stage`, `outcome` and `community` labels, so cardinality stays bounded. No token, assertion, issuer, subject or other claim reaches the line; verifier reasons are static strings. The denial classes and response bodies clients see are unchanged. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
Logan Johnson
|
403855d788 |
test(relay): give group-state Postgres tests a normal acquire timeout (#8564)
🤖 ## Summary The relay's PostgreSQL CI job sometimes fails in the channel group-state tests with `ensure community: Sqlx(PoolTimedOut)`. The tests themselves are fine. They connect to the test database through a test helper that waits only 100 ms for a connection, and on a busy CI runner the first connection can take longer than that. One example is `group_state_events_carry_type_then_creator` in [this #8522 run](https://github.com/block/buzz/actions/runs/40013074144/job/120623293420): it failed after 0.57 s, while its other 1110 tests passed. This PR gives these tests a pool with sqlx's normal acquire timeout. No product code changes. ## Details - `test_state_with_database_url` keeps its 100 ms timeout on purpose. Its other callers point at a closed port (`127.0.0.1:1`) to test fail-closed paths, and the short timeout makes them fail fast. - `group_state_postgres_tests::state()` is the only caller that passes it the real test database. It now builds a normal lazy pool and passes it to `test_state_with_database_pool`. The same `state()` also serves `create_with_members_postgres_tests`, `edit_metadata_postgres_tests`, and `system_channel_postgres_tests`. ### Related issue N/A. I searched open PRs for `PoolTimedOut` and `acquire_timeout` and found none for this. ### Testing To reproduce a slow first connection, I ran the tests through a local TCP proxy that waits 250 ms before it connects to Postgres: - Before this change: `group_state_postgres_tests` had 13 of 14 tests fail with `PoolTimedOut`. - After this change: all 14 pass. `create_with_members_postgres_tests` (3), `edit_metadata_postgres_tests` (7), and `system_channel_postgres_tests` (9) also pass through the same proxy. - Without the proxy, all 14 pass before and after. - `cargo clippy -p buzz-relay --all-targets -D warnings` and `just test-unit` pass. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Matt Toohey
|
8216db1c71 |
chore(toolchain): bump Rust to 1.98.1 for macOS 27 proc-macro loading (#8598)
## Summary
Bump the pinned Rust toolchain from 1.95.0 to 1.98.1 so release builds
work on macOS 27 hosts.
Rust ≤ 1.97 strips Apple-target artifacts with its bundled
`llvm-objcopy`, which leaves the Mach-O `LC_SYMTAB` string pool
(`__LINKEDIT`) only 4-byte aligned. macOS 27's dyld refuses to `dlopen`
such images (`mis-aligned LINKEDIT string pool`). Because cargo applies
the release profile's default `strip = "debuginfo"` to proc-macro dylibs
too, `cargo build --release` of any crate that uses sqlx macros
(`buzz-db`, `buzz-relay`, `buzz-admin`, …) fails on macOS 27: rustc
cannot load `libsqlx_macros-*.dylib`. Dev builds pass because the dev
profile does not strip, which is why this only shows up as a
release/packaging failure.
The fix (llvm/llvm-project#203680, pulled into rustc by
rust-lang/rust#158410 for rust-lang/rust#157750) first shipped in Rust
1.98.0 and was never backported to 1.97.x. `block/buzz-app` pinned
1.98.1 for the same failure in block/buzz-app#529. With this bump the
`CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_STRIP=none` / custom-linker
workarounds people have been using locally are no longer needed.
### What changes
Two commits: the toolchain bump (with the desktop clippy fixes), then
the workspace clippy fixes.
| Location | Change |
|---|---|
| `rust-toolchain.toml` | `channel = "1.95.0"` → `"1.98.1"` (profile
unchanged) |
| `Dockerfile`, `Dockerfile.push-gateway` | `ARG RUST_VERSION=1.95` →
`1.98` |
| `Dockerfile.sprig` | `rust:1.98-alpine3.22` with the manifest-list
digest verified on Docker Hub (shared by the `1.98` and `1.98.1` tags) |
| `.github/workflows/release.yml` | `dtolnay/rust-toolchain` pinned to
the tip of its `1.98.1` branch (`18f04874`) for the Windows desktop
release job |
| `benchmarks/harbor-buzz-orchestra` | `RUST_IMAGE = rust:1.98-alpine` |
| `_ci-rust.yml`, `windows-canary.yml`, `app_state_tests.rs` | stale
"1.95" comments reworded; the `unknown_lints` allow stays with a
version-agnostic rationale |
| `desktop/src-tauri` | lints clippy 1.98 newly raises under `-D
warnings`: `chunks_exact_to_as_chunks` at five fixed-width PCM/UTF-16
decoders, `needless_borrows_for_generic_args` on the `Copy` `norm`
closure in `config_bridge/reader.rs`, and a redundant `Engine as _`
import in `hpke_key_backup_tests.rs` that `use super::*` already
provides |
| Workspace crates (9) | the 30 lints clippy 1.98 newly raises in `cargo
clippy --workspace --all-targets -- -D warnings`, listed below |
Workspace lint fixes:
| Lint | Sites | Fix |
|---|---|---|
| `result_large_err` (20) | `buzz-relay` `api/git/transport.rs` (8),
`api/git/settings.rs` (6), `api/gifs.rs`, `api/workflows.rs`,
`api/profiles.rs`; `buzz-push-gateway` `verify_installation_assertion` |
Clippy 1.98 now checks `async fn`s, so every `async fn … -> Result<_,
axum::response::Response>` trips it. These follow the existing allow
pattern in `api/media.rs` and `nip_fi_http.rs` ("Response is the natural
error type for axum handlers"): module-level in the two git files,
per-function elsewhere. Boxing `Response` would touch every handler for
no runtime benefit. |
| `chunks_exact_to_as_chunks` (3) | `buzz-media` MP4 brand scan;
`buzz-voice` npy f32 reader and stereo test fixture | `chunks_exact(N)`
→ `as_chunks::<N>().0` |
| `question_mark` (3) | `buzz-acp` queue flush fallback and
slash-command mention skip; `buzz-agent` `find_git_root` | `match … {
None => return None }` → `?` |
| `useless_format` (2) | `buzz-auth` nip_fi startup test policies |
`format!("…")` → `"…".to_string()` |
| `useless_borrows_in_formatting` (1) | `buzz-pairing-cli` secret print
| drop the borrow |
| `manual_noop_waker` (1) | `buzz-sdk` broker test `block_on` |
`Waker::noop()` (stable since 1.85) |
All `as_chunks` rewrites keep `chunks_exact`'s remainder-dropping
semantics, and `as_chunks` is stable since 1.88, which is the MSRV.
One change in `api/git/transport.rs` is not a lint fix. Staging that
file trips the pre-commit secret scanner on its existing test-only ES256
fixture (`GIT_TEST_EC_PEM`), and the scanner only honours a suppression
marker on the flagged line. So the `\`-continued string literal is
rewritten as a byte-identical `concat!` whose `BEGIN` line carries
`sadscan:disable kingfisher.privkey.2`.
Left as is on purpose: `Cargo.toml` `rust-version = "1.88.0"` (MSRV
floor, a separate concern), the README/CONTRIBUTING "1.88+" text, Hermit
(no Rust packages there), the lockfiles, and the `(1.95.0)` comment in
`scripts/start-isolated-test-relay.sh` (the pre-commit secret scanner
flags that file's pre-existing dev-only `DATABASE_URL` whenever it is
staged).
### Related issue
Fixes #8074
### Testing
All under rustc/clippy 1.98.1 on a macOS 27 host (Darwin 27.0.0,
aarch64), with no `RUSTFLAGS` / `SDKROOT` / strip overrides in the
environment:
- `cargo build --release -p buzz-db` (the #8074 repro: the release
profile strips the proc-macro dylib): **builds**. The resulting
`libsqlx_macros-*.dylib` (14 MB) has `LC_SYMTAB stroff % 8 == 0` per
`otool -l`. The same build on 1.95.0 died with `mis-aligned LINKEDIT
string pool`.
- `just clippy` (workspace, `-D warnings`) and `cargo fmt --check`
(workspace and `desktop/src-tauri`): pass
- `just desktop-tauri-clippy` (default and `mesh-llm` feature sets):
pass
- `just test-unit`: every crate passes. The `buzz-acp` binary failed
only under pre-push lane contention; rerun solo it is 1060 passed, 0
failed.
- `just desktop-tauri-test`: 3422 passed, 0 failed
- After the lint fixes: lib/bin unit tests for `buzz-acp`, `buzz-agent`,
`buzz-auth`, `buzz-media`, `buzz-voice`, `buzz-sdk`, `buzz-pairing-cli`,
and `buzz-relay`'s `api:
|
||
|
Luke Tornquist
|
0e36dbc823 |
fix(db): keep the reader pool warm and drop its pre-acquire ping (#8495)
## Summary
The reader pool's 150ms `acquire_timeout` covers the whole acquire: the
pre-acquire idle-connection ping **and** any new-connection dial (TCP +
TLS + SCRAM). When readers slow down for a moment, the timeout:
- abandons in-flight dials mid-handshake (Postgres logs `could not
accept SSL connection: EOF detected`),
- closes healthy idle connections whose ping overran instead of
returning them to the pool, and
- falls every miss back to the writer, which then exhausts its own pool.
Across a fleet of relay pods this shows up as a burst of abandoned
handshakes on the readers that starts on nearly every pod at once,
followed seconds later by writer pool-acquire timeouts. The database
itself reports no statement, lock, or connection-limit errors before the
burst: the client is tearing down its own reader connections.
This PR removes that amplifier. It does not change the request budget,
writer fallback, or the writer pool.
### Change (`crates/buzz-db/src/runtime/mod.rs`)
- **Warm floor:** reader `min_connections` is now `max / 5`, at least 1,
derived from the existing reader pool size (no new config knob). sqlx
re-dials the floor in background maintenance, outside the request
budget, from its reaper and whenever a connection is closed instead of
released. The floor is per relay process, so standing reader connections
are the floor times the number of relay processes; size the reader's
`max_connections` for it.
- **Floor fills at boot:** with `max_lifetime`/`idle_timeout` set, a
lazy sqlx 0.9 pool does not dial `min_connections` at construction. The
existing boot ping now closes its probe connection instead of returning
it, which triggers sqlx's floor maintenance. Probe acquires that overrun
150ms retry with bounded backoff (250ms, 500ms, 1s, 2s). The doc comment
that said sqlx dials the minimum eagerly was wrong for this config and
is corrected.
- **`test_before_acquire(false)` on the reader:** the routed read's
`BEGIN … READ ONLY` already proves the connection and fails closed to
the writer (`reader_validation_error`). This removes one round trip per
checkout and the warm-connection churn.
- **Bounded fence proof:** with no pre-acquire ping, the proof (Aurora
capability probe, `BEGIN`, heartbeat observation) is the first round
trip on a checkout. It now runs under `READER_PROOF_TIMEOUT` (1s), so a
reader that stops answering without closing its sockets (partition, host
loss, failover) fails closed to the writer as the new route reason
`reader_proof_timeout` instead of hanging the request until TCP gives
up. 1s is well above a normal round trip, so a brief reader stall waits
on the reader rather than flooding the writer.
- **Bounded release:** every connection returned to the reader pool
first passes a release check bounded at 5s (`Db::read_pool_options`).
sqlx's own release ping has no timeout, so without this a socket whose
reader went silent would hold its slot until the kernel gives up; with
every slot held that way, the pool cannot dial the reader that replaced
it, and all routed reads fall back to the writer. A check that times out
or errors closes the socket and frees the slot, and floor maintenance
dials a replacement. A connection still inside a transaction is closed
instead of returned: a checkout dropped while its `BEGIN` was in flight
(a timed-out proof, a cancelled caller) has no `ROLLBACK` queued,
because sqlx counts the transaction only once `BEGIN` is acknowledged.
Closing it ends the session, and the server rolls it back. An expired
proof is therefore simply dropped. Maintenance checkouts (fleet usage
collection) also keep their close-on-drop: when one is abandoned at its
deadline, its socket is closed under sqlx's 5s close bound rather than
going through the release path.
- **Pinned isolation default:** reader sessions set
`default_transaction_isolation=read committed` in the startup packet,
which takes precedence over database and role defaults. The release
check depends on this, because it detects an open routed-read
transaction by its `REPEATABLE READ` isolation.
- **Metric:** each close is counted in
`buzz_db_reader_release_closed_total{reason}` (`timeout`, `error`,
`open_transaction`). Operator notes for the floor,
`reader_proof_timeout`, and this counter are in
`deploy/charts/buzz/README.md`.
### What got simpler
- The reader checkout path does one less round trip and no longer closes
healthy connections on a slow ping.
- Boot warm-up reuses sqlx's own `min_connections` maintenance rather
than re-implementing it.
- One release rule now covers every reader checkout, so the
open-transaction leak on cancelled reads no longer needs special
handling per call site.
- No new config knob.
- The reader test proxy now lives once in `test_support` instead of
inside `store/usage.rs`.
- Tests build their reader pools from the production options rather than
hand-copied ones.
### Tradeoffs
- The reader is now dialed at boot (previously "no reader dial at
boot"). If the reader is down at boot, this only produces log noise, and
reads still fail closed to the writer.
- The release check is one trivial statement (`SELECT
current_setting(...)`) per released reader connection. It runs in sqlx's
release task, after the response, so request latency is unchanged.
- The check detects an open transaction by comparing the current
isolation level with the session default. That is only sound because the
reader session default is pinned to `READ COMMITTED` and routed reads
begin `REPEATABLE READ`.
- **Residual window:** after the release check passes, sqlx sends its
own release ping (one more round trip), and that ping has no timeout. A
serving-read connection whose endpoint goes silent between those two
round trips still holds its slot until TCP gives up. Only releases
already mid-ping at that instant are exposed: later releases fail the
bounded check first and are closed. sqlx behaves the same way for every
pool on `main`. Maintenance checkouts avoid it through close-on-drop.
Bounding it for serving reads would mean replacing sqlx's release path
or putting a pooler in front of the reader, so it is left as a
follow-up.
- Boot fill is best effort. Each boot-probe retry still uses the 150ms
serving acquire budget, so if cold handshakes consistently take longer
than that, all five attempts fail and the floor is not filled at boot.
It fills the first time any reader connection is closed, which the
reaper and the release check both do, and until then reads dial as
before.
- This also closes pre-existing debt: a caller that dropped a routed
read mid-`BEGIN` (client disconnect) could leave a connection `idle in
transaction`. The release check covers that path too.
### Related issue
N/A (production incident follow-up; none found).
### Testing
- Postgres tests (`runtime/tests/reader_pool_postgres_tests.rs`):
- the boot ping leaves the reader pool at exactly its floor, all idle
(base leaves one connection);
- a warm reader whose heartbeat read blocks (exclusive lock) falls back
to the writer as `reader_proof_timeout` within one proof budget;
- a warm reader connection terminated server-side falls back as
`reader_validation_error` with no client error;
- a proof that times out mid-`BEGIN` on a paused reader (a test proxy
that holds bytes, like `docker pause`) falls back as
`reader_proof_timeout`. The reader database default is set to
`REPEATABLE READ` for this test. Once the reader resumes, it leaves no
`idle in transaction` reader backend, and the close is counted as
`open_transaction`. The test fails if the release check is made blind to
open transactions, or if the isolation pin is removed.
- Failover: every established reader session is stalled while new
sessions get through. Each stalled warm connection costs one
`reader_proof_timeout`, then routed reads reach the reader again through
new connections within 8s, and the closes are counted as `timeout`. The
test fails with the release bound removed.
- `store/usage.rs`: a fleet collection abandoned at its deadline on a
paused reader frees its slot within 3s while the reader is still paused,
which is faster than the 5s release check. The test fails with
maintenance close-on-drop removed.
- Unit tests: the min-connections mapping, the reader pool options, and
that construction still dials nothing.
- Blox, at `f3e3bf158`: `just fmt-check` and `just clippy` are clean,
and `buzz-db` unit tests are 191/191.
- Blox, at `01745b21e`: fmt is clean, the targeted reader tests passed 3
of 3 runs, and the release-bound and transaction-blind mutations both
turn the targeted tests red.
- Blox, at `2f8e482fb` (before the rebase onto `main`): fmt and clippy
are clean, the targeted reader tests passed 3 of 3 runs, `buzz-db` unit
tests pass, and each of four mutations turns its targeted test red:
release bound, transaction-blind check, isolation pin, and maintenance
close-on-drop.
- Blox, at the rebased head `e80bc2bed`: the same four mutations each
turn their targeted test red, and fmt, clippy, the targeted reader tests
(3 of 3 runs) and `buzz-db` unit tests are clean.
- CI Postgres lane at `e80bc2bed`: 1130/1130 passed, including every new
reader-pool test.
- Live check at `e80bc2bed`: a relay with a two-slot reader pool behind
a proxy, which was then frozen for 18s. All 246 history reads succeeded,
and the slowest took 1.0s. During the freeze, 2 proof timeouts and 2
release-timeout closes were counted. Reads returned to the reader within
36ms of the resume, and no reader backend was left `idle in
transaction`.
- Rebase note: `main` added a reader `after_connect` hook that tightens
`statement_timeout`. It now lives in `Db::read_pool_options`, so the
tests' reader pools get it too.
---------
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Bradley Axen
|
a3a47e6294 |
feat(relay): add an unlisted channel setting (#8579)
🤖 ## Summary Sharing a Solo chat with a large channel today means adding every person as a member, one at a time, and the app blocks the share past 100 people. The plan is to share it the way Google Docs does with "anyone at the company with the link". Buzz's `open` visibility already gives that kind of access: anyone in the community can read and join an open channel. The problem is that an open channel also shows up in everyone's channel list and search, and a private chat shouldn't appear there just because its link was posted somewhere. This PR adds a channel setting, `unlisted`. An **unlisted open channel** can be read and joined by anyone who names it by id (from a shared link). For non-members, it is left out of every read that does not name it. Members see it as before. This is the relay side only; the buzz-app change comes later. #8575 (batch adds) remains the path for small audiences. Unlisted is not an access control. Anyone who has the id can read the channel, and that is intended. ## Solo flows this supports Each user action is one event. The buzz-app side comes later; the relay supports all of these now. | Flow | Event | Result | | --- | --- | --- | | New Solo chat | kind:9007 `visibility private` + `["unlisted", "true"]` | The chat starts unlisted, so it stays unlisted when any client later opens it. | | Share | One kind:9002: `visibility open` + `unlisted true` + `join_role guest\|member`. Then post the link. | Anyone with the link can read the chat by id, both history and live. Nobody is added as a member, so the chat isn't in anyone's channel list. A non-member can't post; `join_role` decides whether someone who joins can. | | View | A REQ that names the chat by `#h`, or by `#d` for its metadata | A non-member can read it. It is never listed for them. | | Reply | kind:9021 join (the app sends it before a viewer's first reply) | Joining makes them a member with the role from `join_role`, so with `member` they can post. A kind:9022 leave makes them a viewer again; the chat stays readable by id. | | Unshare | kind:9002 `visibility private` | The existing open→private eviction closes non-members' live subscriptions. New reads by non-members are refused. | | Convert to a channel | One kind:9002: `["t", ""]` + `unlisted false`, plus `visibility` | Clears the Solo label. Non-members can now find it in discovery. | **Sidebar pins are not relay state.** Pin and unpin live in the app's existing kind:30078 sidebar preferences, which sync across devices and don't touch membership. Don't map them to join/leave: someone a chat was shared with by member add is already a member, so a leave would remove their access. Your own Solo chats keep showing up, because you're their owner member, and members see unlisted channels everywhere. Existing Solo chats aren't backfilled. They get the flag when they're shared, because the share event sets it. **Open question, not built here:** someone who only views a shared chat isn't a member. A mention of them doesn't reach their inbox (`#p:[me]` without `#h`) and sends them no push. ## Behavior | Reader | Reads that don't name the channel (kind:39000 without `#d`, `#P` creator queries, messages, COUNT, NIP-50 search, artifact queries, channel list) | Reads that name it (`#h`, or `#d` on 39000–39003) | kind:9021 join | | --- | --- | --- | --- | | Member | Listed | Readable | n/a | | Non-member, unlisted open channel | **Not listed** | Readable | Works; `join_role` still decides the role | | Non-member, private channel (unlisted or not) | Not listed | Not readable | Refused | - **Setting it:** kind:9002 `["unlisted", "true"]` / `["unlisted", "false"]`. Owner or admin only, like `visibility`. A missing tag means no change. It can be sent in the same event as `["visibility", "open"]`, so a private chat becomes open and unlisted in one step. Any other value is refused before the event is stored. The relay posts an `unlisted_changed` system message and clears the accessible-channels cache across nodes. Nobody loses access when a channel is unlisted, so no subscriptions are evicted. - **At creation:** kind:9007 also takes `["unlisted", "true"|"false"]`. A missing tag means listed. Any other value is refused before the event is stored, as on kind:9002. It is stored with the channel row in the create's transaction, on both the client-UUID path and the relay-chosen-UUID path. - **kind:39000** carries a bare `["unlisted"]` flag while the setting is on. It does not add NIP-29 `hidden`, so the metadata stays readable by id. - **NIP-11** advertises `buzz-unlisted-channels` in `supported_extensions`. A relay without this PR would apply `["visibility", "open"]` and ignore `unlisted`, which would list the chat publicly. So buzz-app must check for the extension before it sends that pair. - **CLI:** `buzz channels update --channel <id> --unlisted true|false` and `buzz channels create ... --unlisted`. `buzz_sdk::build_update_channel` takes a new `unlisted: Option<bool>` argument, and `buzz_sdk::build_create_channel` takes `unlisted: bool`, which emits the tag only when true. ### Open question: does `unlisted` only mean something when the channel is open? **Decision: store it on its own; it only has an effect while the channel is open.** Private channels are already invisible to non-members, so on a private channel the flag does nothing. It is still stored and published, for three reasons: - a client can mark a Solo chat unlisted before opening it, or in the same event that opens it, so the chat is never listed even for a moment; - switching private → open → private → open keeps the setting; - there is no cross-field rule in kind:9002 to validate or keep in sync. kind:39000 publishes the flag as stored, whatever the visibility, so clients can show the setting. ## How The relay already has a "readable only when named" path for system channels (NIP-CL). Unlisted open channels reuse it: - **`get_accessible_channel_ids`**: the open half (the implicit scope of a REQ that doesn't name a channel) adds `AND NOT unlisted`. The member half is unchanged, so members still have the channel. - **`get_readable_named_only_channels`**: also returns unlisted open channels of the legacy types. Every reader of that set (REQ, COUNT, NIP-50 search, HTTP `/query` and `/count`, thread windows, workflow run reads) adds a named-only channel only to a filter that names it, through `filter_channel_scope`. An unlisted channel's type is `stream`/`forum`, so the `#t:["system"]`/`#P` system opt-in never adds it. - **`get_accessible_channels`** (the channel list): open channels are listed to non-members only when not unlisted. - **`query_artifacts`**: an artifact in an unlisted open channel reaches a non-member only when the query's `#h` names that channel. - **Live fan-out:** no change was needed. A global subscription never receives channel-scoped events, kind:39000 included (`fan_out_scoped`). A channel-scoped subscription has to name the channel, and that goes through the same named-only authorization. **Left alone on purpose:** access checks that read a named channel or event keep treating an unlisted channel as open. These are `handle_join_request` (kind:9021), the ingest write gate, operator-listener mention delivery (`load_deliverable_event`), personal read-state writes, and push matching. Push matching still requires membership, so non-members get no pushes from an unlisted channel. **Named reads checked:** these callers use the implicit channel set (`get_accessible_channel_ids`, its cache, `get_accessible_channels`). Every one that reads a channel the request names also adds the named-only set, so a non-member can still read an unlisted open chat by id. That covers WS REQ (history, live registration, NIP-50 search), WS COUNT, HTTP `/query` and `/count`, thread windows and workflow run reads. Other named reads don't use the implicit set: they check visibility or membership, so an unlisted open chat counts as open. Those are live fan-out, the ingest write gate, typing, kind:9021 join, huddle audio join, operator-listener delivery, the workflow sink and read-state writes. Feed queries (mentions, needs-action, activity) and the `/buzz/v1/me/sidebar` accessory API are lists, so leaving out chats you only view is intended. Media reads don't check channels. `buzz-auth`'s `ChannelAccessChecker` has no production implementation. ## Database (docs/database.md checklist) - **Phase:** expand only. The migration adds a column with a constant default, and the same release ships the code that reads and writes it. No backfill or contract step follows. - No trigger, function, procedure, rule or foreign key is added. - No migration changes existing rows. `ADD COLUMN ... BOOLEAN NOT NULL DEFAULT FALSE` stores the constant default in the catalog, so the table is not rewritten. - Every location in Rule 4 is updated: `migrations/0069_channel_unlisted.sql`, `schema/tables/public/channels.sql`, and the migration count. No seeds, types or reconcile changes are needed. `admin_schema_parity_between_desired_state_and_migrations` now compares `channels` columns between the pgschema desired state and the migrated database. I failed it once by removing the column from `channels.sql`. Only columns are compared for `channels`: on current `main`, pgschema already re-emits `idx_channels_id_live` without its `INCLUDE (community_id)`, an index drift that predates this PR and is noted in the test. - **Currently deployed relay against the new schema:** it never selects or writes `unlisted`, and every insert gets the default. All channel-record queries name their columns. Nobody can set the flag until this relay ships, because the old relay refuses a kind:9002 whose only tag is `unlisted`. - The migration starts with `SET LOCAL lock_timeout = '5s'`. No index is added: the new predicates ride the existing `(community_id, visibility)` index. ## Rollout 1. Deploy. Every channel stays listed, so nothing changes. 2. **Don't unlist any channel while an old relay is still running.** An old relay ignores the column and lists the channel to everyone. The NIP-11 extension lets clients gate on this. Unlisted isn't an access boundary, so a mixed fleet only exposes the channel in listings for a while. 3. **Rollback:** an older relay ignores the column, so unlisted channels show up as ordinary open channels again. Nothing breaks. ## Tests Postgres (`buzz-db`, discovered by the Postgres lane): - `unlisted_open_channel_is_named_only_for_non_members`: sets the flag through `update_channel`. Then checks that a non-member doesn't get it from `get_accessible_channel_ids` or `get_accessible_channels`, and does get it from `get_readable_named_only_channels`; that a member still lists it; that `put_member` (the kind:9021 path) lets the non-member join, after which they list it; and that listing it again shows it to a new reader. - `unlisted_private_channel_stays_private_and_opens_unlisted`: a private unlisted channel is not readable by name, and opening it keeps the flag. - `group_metadata_publishes_the_stored_unlisted_flag`, plus the unit test `publishes_the_unlisted_flag_as_stored`. - `unlisted_channel_artifacts_are_read_by_name_for_non_members`. - `channel_created_unlisted_is_never_listed`: creating through `create_channel_with_event` (the kind:9007 path) and `create_labeled_channel` stores the flag, and the channel is never listed to a non-member. - Mutation-checked: each of `NOT unlisted` in the implicit scope, `NOT c.unlisted` in the channel list, the unlisted branch of the named-only query, its `visibility = 'open'` guard, and the artifact condition was removed one at a time, and each removal failed at least one of these tests. Relay E2E (`test_nip29_unlisted_channel_wire`, added to the CI relay E2E step), against a local relay built from this branch: - A non-member or an unknown value is refused. - kind:39000 by `#d` carries `unlisted` and `public`, and not `hidden`. - kind:39000 discovery without `#d`: non-members don't see the channel, members do. - History and NIP-50 search: unnamed, a non-member doesn't see the channel's messages; with `#h`, they do; a member sees them unnamed. - A kind:9021 join works, and the joiner then lists the channel. - Listing it again shows it to everyone. - A reader whose channel set was cached before each change sees the change at once. I ran this against a relay with the cache invalidation removed, and it failed. Relay E2E (`test_nip29_solo_share_and_convert_wire`, added to the CI relay E2E step), each flow as a single event: - kind:9007 with an unknown `unlisted` value is refused. A chat created with `unlisted true` publishes the flag, and stays unlisted after a plain `visibility open`. - Share (open + unlisted + `join_role guest` in one kind:9002): a non-member doesn't list the chat. They read it by `#h`, both history and a live message, and can't post. A kind:9021 join lists it for them as a read-only guest. A kind:9022 leave unlists it again, and it stays readable by `#h`. - Unshare (`visibility private`): the viewer's live subscription gets CLOSED, and a new `#h` read is refused. - Convert (`["t", ""]` + `unlisted false`): the Solo label is cleared, and a non-member finds the chat in discovery. - I ran this against a relay that dropped the kind:9007 flag, and it failed. Also run: `cargo test` for buzz-db, buzz-sdk, buzz-cli, buzz-core, buzz-relay and buzz-test-client; the Postgres lane for buzz-db and buzz-relay (1045 tests); and the existing unlisted, announce, departure, guest, role-change and NIP-43 snapshot wire tests. In the lane, two tests failed on pool timeouts (`resend_validated_after_the_first_commit_is_a_duplicate`, `edit_replaces_and_clears_labels`) and passed when rerun alone. An earlier run had the same kind of flake: `routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold` (a timing budget) and `resent_edit_is_a_duplicate` (a pool timeout). In a full local `e2e_relay` run, three tests outside CI's list also failed (`test_invite_*` twice, `test_unarchive_emits_member_added_notification`). They fail the same way on a relay built from `main`. ## Not in this PR - The buzz-app follow-up: support open Solo chats, the join-from-card view, the large-audience prompt, and gating on `buzz-unlisted-channels`. - A NIP-CL update. The behavior is documented in NOSTR.md ("Unlisted channels"). buzz-review-completed --------- Signed-off-by: Bradley Axen <baxen@squareup.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: ART <ea39c1c8acf57180161ef4ba4ed68abf8589f5291956b909a5aa991fc66664ca@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz> Co-authored-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> |
||
|
Tom Brow
|
b8b1ab5b3c |
perf(push): avoid empty push delivery claims (#8524)
Each enabled push delivery worker currently opens a claim transaction for every active community, even when its queue is empty. Discover eligible communities in bounded pages first, then use the existing claim and delivery path. No schema or index change is needed. Per-delivery optimizations (sharing serving-write leases across a batch and combining revalidation or membership checks), enqueue batching, and outbox pruning are out of scope and are being investigated separately, including their deletion, revocation, and stale-worker safety guarantees. ### Related issue No matching existing issue or PR found. ### Validation - 25 focused PostgreSQL tests passed, covering real-worker delivery, eligibility filtering, pagination fairness, concurrent claims, expired-claim recovery, and stale-worker fencing. The [full PostgreSQL CI suite](https://github.com/block/buzz/actions/runs/39675698081/job/119490943190) passed all 1,090 tests, including all seven new regressions. - Discovery was benchmarked with **43,000 communities**, existing indexes, and 30 warm executions per scenario. Empty queues required no community lookups; eight ready communities spread across the ID range required eight community index lookups. Results and the alternative-query comparison are below. - In the separate real-worker test with 1,000 empty communities, the original worker made 1,401 claim admissions in 1.8 seconds; the fixed worker made zero. One and four workers made 3 and 12 discovery reads respectively. - In the delivery test with 65 ready communities plus 1,000 empty communities, all 81 notifications reached an HTTP test gateway in 0.83 seconds versus 1.55 seconds with the original loop. - Independent standards and specification reviews, plus [Codex review](https://github.com/block/buzz/pull/8524#issuecomment-6326568721), found no remaining blockers. Final Rust linting, the full local unit/integration run, desktop JavaScript tests/build, web build, and mobile analysis/tests passed. The full local desktop test run had one unrelated timing failure in `reaps_descendant_on_timeout`; its isolated recheck passed. [CI](https://github.com/block/buzz/actions/runs/39675698081) passed, including the Windows retry after an unrelated subprocess timeout. These are isolated local measurements, not production latency guarantees or a simulation of full production concurrency. Production delivery validation remains part of the release process; this PR does not re-enable push or deploy the change. ### Alternative considered An outbox-first query selects due pending jobs and expired sending jobs with `UNION ALL`, applies the same eligibility checks, then returns distinct community IDs. This could avoid scanning communities with no ready work. However, the current query's `EXISTS` clause does not force PostgreSQL to probe the outbox once per community: the measured plans already avoid that work when queues are empty. Both query shapes were tested against the same **43,000-community** dataset using existing indexes. Median database execution times over 30 warm runs: | Scenario | Current query | Outbox-first query | | --- | ---: | ---: | | Empty outbox | 0.038 ms | 0.041 ms | | No ready work, 27,336 retained jobs | 0.060 ms | 0.066 ms | | Eight ready communities spread across the ID range, including expired sending jobs, plus retained history | 0.109 ms | 0.126 ms | | Eight ready communities, 128 invalid jobs, and 27,200 terminal jobs | 0.225 ms | 0.335 ms | Both queries returned identical community IDs in every scenario. Forced generic prepared plans produced similar results. The current query was retained because the alternative did not reduce measured execution time or eliminate a community scan present in the empty-queue plan. No query rewrite or additional index was justified by these measurements. --------- Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Will Pfleger
|
369ffa0380 |
feat(relay): verify NIP-FI agent tokens and admit agents through owners (#8401)
The relay side of NIP-FI agent tokens. An agent token has its own type, `nip-fi-agent+jwt`, and carries its human owner's pubkey in `nostr_owner_pubkey`. This PR verifies that claim, makes denying or disconnecting the owner cut off every agent that owner authorized, and makes the verified owner decide community admission. A relay without agent support refuses the unknown type, so nothing needs to advertise or configure agent support. The `NIP-FI.md` text for the agent type is included. Builds on #8372 and #8377 (both merged). ### Owner claim and owner deny - **Token type binds the owner claim (`buzz-auth`).** A `DedicatedNipFi` issuer policy accepts `nip-fi+jwt` (human) and `nip-fi-agent+jwt` (agent); `AccessTokenAtJwt` policies stay `at+jwt`-only with their `client_id` and subject rules unchanged, and unknown types are refused. An agent token must carry a lowercase-hex `nostr_owner_pubkey` distinct from its `nostr_pubkey`; a human or `at+jwt` token carrying the claim with any value, including null, is `evidence_rejected`. The verified owner is exposed as `VerifiedAssertion::owner()`. - **No agent switch.** There is no relay setting, NIP-11 field or operator binding field for agent support. - **Session owner.** Each connection records its NIP-FI issuer and the agent's owner (`NipFiSession`) on the root WebSocket manager and the audio registry. `disconnect_nip_fi(iss, target)` closes a connection when either its own key or its recorded owner equals the target. - **Owner deny check.** `nip_fi_core::assertion_denied` checks the deny set for both `(iss, k)` and `(iss, owner)`, and every existing deny check uses it: WebSocket before upgrade and after registration, HTTP, audio, and the shadow observer, so shadow and enforce make the same decisions. The shadow disconnect observer matches a session's key or owner under its issuer with the same rule, so an owner's disconnect records a would-close for its agents. ### Shadow owner check In shadow mode, WebSocket AUTH, audio join and HTTP admission record a `Stage::Owner` verdict when an admitted agent token's owner is banned (`owner/denied`), conflicts with the agent's stored owner (`owner/denied`), or cannot be checked because a read failed (`owner/unavailable`). These are reads only; shadow admission is unchanged. Shadow applies the same stored-owner rule as enforce but never writes the link, so it can't predict a later owner-link write failure, or which of two racing first links wins. ### Admission through the verified owner - **Owner bans, unconditionally.** `community_ban_outcome` takes the token owner and checks its ban in place of a NIP-OA owner. That applies whether the agent is a member itself, the relay is open, or the agent is admitted through its owner. - **Owner-derived membership.** In `api::relay_members::check_membership`, an agent that is a member itself is still admitted first. Otherwise, when a verified assertion names an owner, the owner's membership decides (`MembershipDecision::ViaOwner`), and no NIP-OA is needed on the connection. The issuer has already passed this community's NIP-FI binding check at that point. When a token owner is present it is the only owner considered. - **Stored-owner consistency.** `link_agent_owner` replaces the boolean `materialize_nip_oa_owner` result with `OwnerLink::{Linked, Conflict, Unavailable}`. An unset stored owner is recorded, so relay bans and roster removal of the owner reach the agent through `revoke_live_access`. Shadow and enforce judge the stored owner through one read, `stored_owner_link`, and `link_agent_owner` writes (atomically, first write wins) only when that read finds no owner; a failed read fails closed. If the stored owner differs from the token owner, the request is refused with `authorization denied` on HTTP, WS and audio. A NIP-OA owner conflict keeps its existing link-error refusal. - **One decision on every surface.** Root WS AUTH, audio registration and every NIP-98 HTTP route (bridge, `buzz/v1`, git transport and settings, invites, media, workflows, moderation reads) pass `NipFiAdmission::token_owner()` / `VerifiedAssertion::owner()` into the same ban, membership and owner-link steps. The final socket check rereads bans and writer-side membership; it doesn't reread the owner link, which is written once from NULL. HTTP membership still goes through the bounded replica read. With a verified agent token, every HTTP admission refusal, including the fresh ban rereads in `buzz/v1` and Git settings, is NIP-FI's fixed `authorization_denied` (or `authorization_unavailable` when a dependency fails), and audio checks bans before writing the owner link. `enforce_relay_membership` now takes the `TenantContext`, because it writes the owner link. - **Publishing.** A ban caught while publishing over `POST /events` after the shared membership check is also an admission refusal. Ingest reports it as a typed `IngestError::Ban` (banned, or ban lookup unavailable) at the write gate, the HTTP typing indicator's cached gate, and the moderation and relay-admin command rereads. `IngestError::Ban` carries the legacy error. The bridge turns it into the fixed `authorization denied` / `authorization unavailable` only when the request carries a verified agent token. WebSocket and token-less HTTP responses are byte-for-byte unchanged, and timeouts, scopes, permissions and every other ingest error keep their existing responses. ### Rollout - **Deploy the relay before kgoose.** An older relay refuses `nip-fi-agent+jwt`, so agents are locked out until both are deployed. Humans are unaffected. - **There is no agent-only switch.** Setting NIP-FI to shadow or off turns it off for humans too. - **Rollback is safe for owner links.** Stored owner links survive a rollback, and the older code still honors them. - **Shadow signals:** in `buzz_nip_fi_shadow_total`, `stage="owner"` means an owner would-deny (`owner/denied`) or an owner read that was unavailable (`owner/unavailable`), not a success. A clean agent admission shows as `stage="admit", outcome="admit"`. - **The relay checks that the owner is a member, not which issuer the owner logged in through.** In a community that trusts several issuers, any trusted issuer can name any member as owner. That matches the spec: the issuer is trusted to vouch for the owner, and kgoose requires the owner's attestation. Agent tokens are only issued to relays whose NIP-FI binding trusts the issuer, and NIP-FI itself is off unless configured. No deployed config or secrets change. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
Bradley Axen
|
d0dee1cb50 |
feat(relay): add several people with one kind:9000 (#8575)
## What
A NIP-29 kind:9000 (put-user) can now name up to 100 people, one `p` tag
each. A client sharing a channel with many people can do it in one
signed write instead of one event per person.
- **Role per person:** NIP-29 style `["p", pubkey, role]`. When the
third element is absent or empty, the event's `["role", r]` tag applies.
- **Same checks, per person:** each person gets the checks a single add
does, in the same order: `decide_put_user`, then the agent's
`channel_add_policy`, then the admission rules again under the channel's
membership lock. One refused person no longer refuses the others.
- **Per-person results:** when the event names two or more people, the
OK message reports each result. Only non-empty lists appear, and
refusals are grouped by reason so 100 people stay well under a client's
16 KB receipt limit:
```
members:
{"added":[hex…],"updated":[hex…],"already_member":[hex…],"rejected":[{"reason":"…","pubkeys":[hex…]}]}
```
- **When the event is refused:** if it is malformed (bad `p` hex,
unknown role, more than 100 people, missing `h`, archived channel), or
if nobody it names may be added. With one person the message keeps
today's `invalid: <reason>`. With several it is `invalid: members: {…}`.
- **One-person events don't change:** a successful single add still
returns an empty OK message, which an existing e2e test pins.
- **Re-adding someone at the role they already have** no longer posts a
second `member_joined` timeline notice. The member-added notification
(kind:44100) still goes out, so an agent that missed its first one is
still re-signaled.
- **Group-state events (39000–39002)** are now published once per event
instead of once per person.
## Cost of a batch
The database work for one event does not grow with the number of people:
- **One membership transaction.** `buzz-db` gains `put_members`: one
transaction, one membership lock, one channel read, one read of every
involved role, the owner count only if a demotion needs it, and one
`UNNEST` upsert with `RETURNING`. `put_member` is now `put_members` with
one person, so there is one implementation of the admission and
last-owner rules, and a single add makes two fewer round trips than
before.
- **Same answers as one write per person.** People are decided in tag
order, and each decision sees the roles earlier people were given,
including the actor's own (an owner who steps down mid-batch can't then
grant admin), and the owners they add or remove. A new buzz-db test pins
this.
- **One policy query.** Validation reads every third-party target's
`channel_add_policy` with one `= ANY($2)` query
(`get_agent_channel_policies`) instead of one per person.
- **One channel-type read** for all the member-added notifications,
instead of one per notification.
What still scales with people is what each person must receive: their
`member_joined` timeline notice and their kind:44100 notification, one
stored event each.
## Why
This is for Solo chat sharing in buzz-app.
[block/buzz-app#1605](https://github.com/block/buzz-app/pull/1605)
routes every share through one helper, `shareChatsWithChannel`, which
today sends one kind:9000 per missing member. A small follow-up there
switches it to one batch, then re-reads the roster and adds anyone still
missing one at a time. That keeps it correct on relays without this
change, which accept the event but ignore every `p` after the first.
The batch stays a signed NIP-29 event rather than a new HTTP endpoint,
per VISION.md and AGENTS.md ("Prefer Nostr events over new HTTP
endpoints").
## Wire changes for existing senders
- Any malformed `p` tag now refuses the event. Before, malformed `p`
tags were skipped and the first valid one was used.
- Extra `p` tags are now targets. Before, they were ignored.
- A third `p` element that isn't a role, such as a NIP-01 relay hint,
refuses the event with `invalid role: …`. No sender in this repo does
that: `build_add_member` sends one bare `p`.
## Tests
**buzz-db (`channel_members` Postgres tests):** new
`put_members_decides_each_person_in_order`: the sole owner can't step
down before an heir is named in the same call, but the heir is still
added; with the heir counted the owner can step down, and the admin
grant after it is then refused; a no-op re-add reports its prior role; a
repeated pubkey refuses the call. The other `channel_members` Postgres
tests pass, including the announce-admission and concurrent `join_role`
tests that exercise `put_member`.
**buzz-relay (`put_user_postgres_tests.rs`), 5 tests:**
- mixed results in one event: added, already a member, an agent with
`nobody` policy rejected, a per-`p` `bot` role, a duplicate `p`, a
single roster publish
- `updated` for a role change, with the event `role` tag as fallback
- a plain member may add someone, but may not grant an elevated role on
a `p`
- a single add keeps its empty OK message
- an all-refused event: single text and grouped JSON
- a malformed `p` tag
- the 100-person cap: 101 refused, 100 accepted
Related relay Postgres groups (`group_state`, `create_with_members`,
`edit_metadata`, `membership`) pass.
**Live relay:** `e2e_relay` against a local relay at 9c27a0f15: 58 of 61
passed, including `test_nip29_batch_put_user_wire` (added to the CI
Relay E2E list) and every NIP-29 put-user, announce, guest and departure
test in that list. The 3 failures don't touch this change: the two
invite tests assume the relay's seeded host, and this relay ran on
`:3030`. `test_unarchive_emits_member_added_notification` timed out once
and passed when re-run, as it did on the first commit.
**Local caveats:** the local `channel_members` roster-fence test and the
deletion tests fail on a shared local database because other tests leave
its fence trigger disabled, so the PostgreSQL Tests job in CI is the
authority for the full suite. The pre-push `rust-tests` hook passed.
`desktop-tauri-checks` failed on `chunks_exact` lints in desktop files
this PR doesn't touch, so the push used `--no-verify`. `cargo clippy
--all-targets` (buzz-db, buzz-relay) and `cargo fmt --check` are clean.
---------
Signed-off-by: Bradley Axen <baxen@squareup.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: ART <ea39c1c8acf57180161ef4ba4ed68abf8589f5291956b909a5aa991fc66664ca@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: claude00 <fb8b76a01257932b8ca35060862c29bd09b2137638573731c2d5074fc02c5385@buzz.block.builderlab.xyz>
Co-authored-by: mbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz>
Co-authored-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
|
||
|
Luke Tornquist
|
7ce422417b |
fix(mesh): expire dead peer records and instrument mesh boot (#8503)
## Summary
Relay pods can stall in the `mesh_boot` startup step, and the mesh peer
table grows without limit. One prod pod held 7,793 peer records. Most of
those records belong to pods that no longer exist.
**Cause.** Every boot gets a fresh runtime id, so a restarted pod leaves
its old record behind. Nothing ever removed records, and scuttlebutt
gossip kept spreading the dead ones around the fleet. Each reconcile
pass, including the first one inside `boot_mesh`, then dials every
record one at a time.
This PR makes dead records expire. It also adds the measurements we need
to see the next stall directly.
### Peer-table lifetime (`crates/buzz-relay-mesh/src/membership.rs`)
- A peer record expires 5 minutes after its last heartbeat
(`DEFAULT_PEER_RECORD_TTL`). For comparison, live runtimes refresh every
2s, and the Redis ready registry expires a silent runtime after 45s.
- What happens to expired records:
- They are **rejected on arrival**, so peers that haven't been upgraded
can't re-add them.
- They are **left out** of digests, deltas, and dial candidates.
- They are **pruned** at the start of each reconcile pass.
- Between prunes, they are not admitted (`has_peer`) and not routed
(`peers`, `peer_has_capability`).
- An expired entry that hasn't been pruned yet counts as absent, so any
fresh record replaces it whatever its version. That lets a live peer's
v1 Redis hint re-admit it before the next prune.
- **The TTL never prunes or hides a peer with a live connection**, even
if its heartbeat is stale. Its gossip can stop arriving while the
transport stays up: a dead control stream, or a clock behind by more
than the TTL. The connection proves the runtime is alive. Its stale
record is still kept out of digests and dials. (The existing phi filter
in `peers()` still drops a connected peer once its gossip stalls; this
PR doesn't change that.)
- **`Connected` tracks the transport map.** Because the TTL exemption
trusts that flag, `install_peer` and `remove_peer` write it while
holding the peers lock, so a connection that closes during install
always ends `Disconnected`. A dial's `Connecting`/`Disconnected` writes
are skipped when a connection is already installed, so a failed outbound
dial can't overwrite a live inbound connection. Each reconcile pass also
re-syncs the state from the transport map.
- **Gossip no longer sets connection state.** Before, any gossiped
record was marked `Connected`, even if this pod had never dialed it.
That inflated the `/_mesh` counts. Now a record learned from gossip or
the ready registry enters as `Disconnected`. Only the real connect and
disconnect paths change the state.
- Before building its candidate list, reconcile now skips peers that are
already connected. A pass dials serially, and a dial to a dead runtime
can take about a minute, so `dial_peer` re-reads the record before every
address attempt. It stops once the record has expired or started
draining, or once a connection has landed.
### Instrumentation
- **`mesh_boot` sub-steps** use the existing startup-step gauges:
`mesh_bind`, `mesh_publish`, and `mesh_initial_reconcile`. A pod stuck
in `mesh_boot` now also shows which await it is stuck on.
- **New metrics** (`crates/buzz-relay-mesh/src/telemetry.rs`):
| Metric | Shows |
|---|---|
| `buzz_mesh_dial_total{outcome}` / `buzz_mesh_dial_duration_seconds` |
Cost of each dial |
| `buzz_mesh_reconcile_pass_seconds` | Full pass time |
| `buzz_mesh_reconcile_candidates` | Dials planned for this pass (set
after the registry scan) |
| `buzz_mesh_peer_records{state,draining}` | Table size |
| `buzz_mesh_peer_record_max_age_seconds` | Oldest heartbeat |
| `buzz_mesh_peer_records_expired_total{reason=pruned\|rejected}` |
Expiry activity |
- The dial and pass histograms get buckets up to 600s. The default
`_seconds` buckets stop at 5s, which would hide exactly the stall these
metrics are meant to show.
### CI
The `buzz-relay-mesh` tests and the `buzz-relay` `mesh_boot::tests` did
not run in any CI lane. This PR adds both to `just test-unit`, and adds
the matching stanzas to `scripts/run-tests.sh`.
### What got simpler
- The peer table now has a defined size, about one record per live pod,
where before it grew forever.
- One wrong state write was deleted (gossip marking records
`Connected`).
- Two test suites that silently never ran now run.
### Rollout notes
- Rejecting expired records works one pod at a time. During a rollout,
pods that haven't been upgraded keep gossiping dead records. Upgraded
pods reject them, which shows up as `reason="rejected"` on the expired
counter.
- A pod whose clock runs more than 5 minutes behind has all of its
gossip rejected. It stays reachable through its Redis ready record and
any live connection. A `reason="rejected"` count that keeps climbing
outside a rollout points to that skew.
- Upgraded pods leave expired records out of their digests. Pods that
haven't been upgraded therefore resend the whole ghost set (about 7.8k
records, roughly 1 MB) on every 2s gossip tick, on each old↔new
connection. So during the rollout, `reason="rejected"` will climb by
about the ghost count per tick per connection. This is expected and
stops when the rollout finishes.
### Not in this PR
The boot dial itself is still unbounded: it runs serially, has no
per-dial timeout, and dials gossip-learned records. Bounding it changes
how a pod joins the mesh, so it goes in a separate PR, and these metrics
will measure it.
### Related issue
None found.
### Testing
- New tests:
- `membership::tests`: expired gossip is rejected; prune drops only
expired records; prune keeps a connected peer with a stale heartbeat;
expired unconnected records are neither admitted nor routed before a
prune; a fresh record replaces an expired, unpruned entry whatever its
version; expired records are neither dialed nor gossiped before a prune;
gossip never sets connection state; table stats.
- `runtime::tests::reconcile_pass_publishes_dial_and_peer_table_metrics`
- `runtime::tests::reconcile_pass_prunes_expired_peer_records` (binds
the prune to the reconcile pass)
-
`runtime::tests::dial_skips_candidate_that_expired_after_the_pass_snapshot`
-
`runtime::tests::finishing_dial_does_not_overwrite_a_connection_that_landed`
-
`runtime::tests::connection_closed_during_install_ends_disconnected_and_expires`
-
`runtime::tests::reconcile_pass_resyncs_connection_state_from_transport`
- `mesh_boot::tests::mesh_boot_sub_steps_name_the_step_that_failed`
- Blox at `e06f55a4` (tree `866add1c`): `just fmt-check`, `just clippy`,
`buzz-relay-mesh` nextest 3x, and the `buzz-relay`
`mesh_boot`/`startup_steps`/`metrics` tests all pass. Three of the new
tests fail against a mutant that removes the fix they pin (dial ignores
expiry; dial state writes ignore the transport map; no reconcile
resync).
---------
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
|
||
|
Logan Johnson
|
a2d28eb6c4 |
fix(relay): apply channel setting changes atomically and lock every membership writer (#8516)
🤖 ## Summary This PR fixes two ways the relay could report a channel change that did not fully happen. **1. Channel settings could be saved only in part, while the app was told that they all worked.** When an owner or admin changed a channel's settings (name, description, topic, public or private, message lifetime, archived or not, who can join), the relay saved the request first and then applied each setting with its own database write. If one write failed, the relay only logged the error. It skipped the remaining settings and still reported success. Sending the request again did nothing, because the relay saw it as already saved. Example: a switch from public to private could fail while the app showed success, and the channel stayed public. Now saving the request, applying every setting and saving the "settings changed" messages in the channel happen in one database transaction. Either all of it is saved or none of it is: - If anything fails, nothing is saved and the client gets an error. It can send the same request again. - A repeat of a request that already succeeded is reported as a duplicate and changes nothing. This is also true when the repeat arrives while the first copy is still being saved. - A request to restore a channel that is not archived, or to change an archived channel without restoring it, is refused and nothing is saved. Before, the relay saved the request and a "settings changed" message, and then the restore failed. - Live updates go out only after the save commits. These are: removing access for people who lose it, and pushing the new messages and channel info to clients. **2. A published member list could include someone who had just been removed.** For some channels the relay publishes a list of the channel's members. While it builds that list, it holds a per-channel membership lock. Kicks by a relay admin and the automatic kick after a deploy did not take that lock. A kick could commit after the list was read, so the published list still showed the removed member. Channel deletion also did not wait for the lock. Kicks, channel deletion and the automatic add when someone joins a Huddle (voice call) now all take the lock first. The published list then includes the change, or the change waits until the list is saved. ## Details - **One lock order for all channel settings writes.** Before, each writer took its own locks. Review kept finding writers that took them in a different order or forgot one. `lock_channel_for_settings_write` (`crates/buzz-db/src/store/channel.rs`) is now the only place that takes them. The order is: 1. the membership lock, only when the public/private setting or the "who can join" setting (`join_role`) changes. Without it, a member add could read the old rule while the change commits. 2. the message-lifetime (TTL) lock: exclusive for a lifetime change, shared otherwise. 3. the channel row (`FOR NO KEY UPDATE`). This does not block the key lock that a member insert takes. The row lock also orders two copies of the same request, so the second copy sees the first as already stored. - The function returns a `ChannelSettingsWriter`. The writer borrows the transaction that took the locks, so the compiler stops it from being used with another transaction or after commit. The settings writes (`update_channel_on`, `set_topic_on`, `set_purpose_on`, `set_archived_on`) require the writer. They refuse a change its locks do not cover. - `Db::apply_channel_metadata_event` stores the channel-edit event (kind:9002) and its mentions, applies the settings and stores the "settings changed" messages (kind:40099) in one transaction. The older single-purpose writes (`update_channel`, `set_topic`, `set_purpose`, `archive_channel`, `unarchive_channel`) use the same helpers and behave as before. - `ingest.rs` and `validate_admin_event`: channel edits no longer check archived/active state before the write. The transaction checks it after its duplicate check, under the row lock. A refusal there goes back to the client as `invalid:`. Channel edits have their own save path and skip the general post-save step. `handle_side_effects` refuses them, so no caller can go around the transaction. - `side_effects.rs`: `emit_system_message` is split into signing and sending. The messages are signed before the transaction and sent after it. All "settings changed" messages from one edit share one timestamp and come in a fixed order (topic, purpose, visibility, lifetime, archive). - A change to "who can join" (`join_role`) is applied in the same transaction and announced with its own `join_role_changed` message. - **Membership lock outside settings** (`relay_admin_actions.rs`, `channel_members.rs`, `audio/handler.rs`): both kick paths (`execute_kick_with_marker`, `deploy_kick_member`) take the membership lock before they remove the member. The marker kick still undoes the removal if its lease check fails after waiting. The Huddle join takes its locks through a new `ChannelMembershipLocks` token, and the auto-add is a method on that token, so it cannot run without the lock. `soft_delete_channel` takes the full settings lock order. - **Limit of the membership lock:** it covers member changes, public/private and "who can join" changes, and channel deletion. It does not cover archiving: archive writes and the cleanup of expired temporary channels do not take it. The cleanup keeps its single UPDATE. A comment in `channel.rs` explains why that is safe and why it must not be split into a read and a write. ### Related issue Follow-up to the review of #8143 (which added the "who can join" setting). That review found that channel edits could be applied in part. No separate issue. ### Testing New PostgreSQL tests in `crates/buzz-relay/src/handlers/edit_metadata_postgres_tests.rs`: - `failed_edit_stores_nothing`: a test trigger makes one write fail partway through the transaction. Nothing is stored or changed, on the first send or a resend. Without the trigger, the same signed request applies in full. - `resent_edit_is_a_duplicate`, `resent_archive_and_unarchive_are_duplicates`: a resend after success is a duplicate and adds nothing. - `resend_validated_after_the_first_commit_is_a_duplicate`: a resend validated after its first copy committed is a duplicate, with no second change and no second message. - `concurrent_copies_of_a_restore_apply_once`: two copies of one restore, held at the channel lock until both are waiting, apply once. The other copy is a duplicate. - `unarchive_of_an_active_channel_is_refused`, `join_role_edit_is_applied_and_announced`. Lock tests in buzz-db: each one holds a member list in progress and checks that the writer waits for it. They cover a "who can join" change (`channel_members.rs`), the deploy kick and the marker kick (`relay_admin_actions.rs`), and channel deletion (`channel.rs`). The kick and delete tests fail when the new locks are removed. These tests and the existing channel-edit tests in `group_state_postgres_tests` pass locally against a migrated PostgreSQL database. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> |
||
|
Luke Tornquist
|
2f2f6e2774 |
fix(db): bound every statement and the #p mention scan (#8461)
## Problem An agent `#p` + `#h` + `since` REQ (the `require_mention` subscription in `buzz-acp`) ran for over 45 minutes on a production writer and held `AccessShare` on every `events` partition. There were two causes: 1. **The mention side of the `#p` join was unbounded.** `build_query_events_sql` joined `event_mentions` on `(community_id, event_id)` only. The planner picked a nested loop whose outer side was every mention of the pubkey, ever, and whose inner side range-scanned the channel index from `since` on each partition. Cost ≈ total mentions × channel rows since `since`. 2. **Nothing cancelled it.** The writer `statement_timeout` defaulted to `0`. Only `#e` reads had a deadline (the 20s cap from #7854), and read-replica sessions had none at all. `pg_stat_statements` shows this shape has a multi-thousand-second max going back months, so it is chronic and was not introduced by a recent change. ## Change **`#p` mention bound** (`store/event.rs`, `push_mention_join`, shared by the page SELECT and COUNT) - Join on the full event key, `AND e.created_at = m.event_created_at`. Mention rows copy the event's `created_at` at insert (migration 0007 already relies on this equality), so the join can prune `events` partitions by time. - Repeat `since`/`until` on `m.event_created_at`, so the outer scan is a range on the existing `idx_event_mentions_pubkey_created`. `until` is a superset bound for the keyset cursor; the exact `(created_at, id)` cursor test stays on `e`. - Repeat `kinds` on `m.event_kind` when the filter has kinds. Every production mention writer binds the event kind (`insert_mentions` in `store/runtime/mod.rs`, and the replaceable-event path in `store/replaceable.rs`, back to the first Postgres schema), and no migration inserts mention rows, so this drops no rows written by the relay. Rows imported out of band with a NULL `event_kind` would no longer match a kinds-filtered `#p` read; a read-only production count on 2026-10-20 found 0 such rows on both production reader databases. The pushdown makes `idx_event_mentions_pubkey_kind_created` available to the planner for the `require_mention` kind-9 shape; its use has not been observed (see below). An E2E fixture with a hand-set NULL `event_kind` row confirmed the caveat: base returns 8 events, head 7. - The feed's mentions and needs-action queries (`store/feed.rs`, `build_mentions_query` / `build_needs_action_query`) join `event_mentions` the same way, so each joined row probes one `events` partition instead of all of them. Their `ORDER BY m.event_created_at ... LIMIT` already kept them from running away; this removes the per-row cross-partition probe. - Observed plan (E2E at `9cb6638b1`, before the `kinds` pushdown; 60,000 historical mentions + 2,500 recent decoys, stock planner, `EXPLAIN (ANALYZE, BUFFERS)` of the captured production SQL): the mention side is a bounded range on `idx_event_mentions_pubkey_created` (13 rows), then a hash join with a scan of the recent `events` partition. Base scanned 2,509 recent events and probed `event_mentions_pkey` 2,509 times. Head: **1.1 ms / 319 shared hits**; base: **8.5 ms / 10,049 shared hits**. Not measured against the production replica. A fresh `EXPLAIN (ANALYZE, BUFFERS)` replay at `a7065c367`, after the pushdown, ran on a smaller fixture where the planner chose a sequential mention scan, so it neither confirms nor replaces these numbers. **Reads: 20s deadline on every generic event read** - `query_events_on` and `count_events_on` always run under the transaction-local 20s deadline that `#e` reads used (`fetch_with_read_deadline`, formerly `fetch_with_e_tag_deadline`). It only tightens: shorter operator or caller caps are kept. - Read-replica sessions get a 20s session default through `after_connect`, using the same tighten-only SQL. - Cost: every generic REQ history read and COUNT now pays BEGIN + `set_config` + ROLLBACK (three extra round trips; before, only `#e` reads did). In the local E2E the end-to-end history read was still faster at head (median 4.1 ms vs 10.2 ms at base, 9 samples each, warm cache), so the round trips are dominated by the bounded plan; this does not isolate them or establish production p99. The round trips also lengthen how long each such read holds its writer connection; that hold time has not been measured against production pool sizing. - A replica read cancelled at its deadline (57014) is returned, not re-run on the writer. Usually the cancel means expensive work the writer would only repeat. It can also mean the replica waited on a lock only it holds (a replayed `ACCESS EXCLUSIVE` from primary DDL), when the writer would have answered quickly; that miss is the accepted price of keeping runaway reads off the primary. The rationale now lives once on `DbError::is_statement_cancelled` instead of in a dozen copied comments. The generic routed reads and `ReadSession` already did this; the eight routed reads with their own SQL (feed mentions, needs-action and activity; thread replies; channel window; relay membership; thread window and its auxiliary scan) now do too. A cancelled auxiliary scan also keeps its replica session instead of degrading it to the writer for the rest of the request. Other replica failures (e.g. hot-standby recovery conflicts) still fall back to the writer. - Paths with their own tighter `SET LOCAL` (artifact 2–5s, thread window 4s, personal reads, usage 5–15s, serving lease 1.5s) are unchanged. **Writes: every writer session is bounded** - The default `statement_timeout` changes from `0` to 120s (`DEFAULT_STATEMENT_TIMEOUT_MS`). - `BUZZ_DB_STATEMENT_TIMEOUT_MS` still overrides it. `0` or any value above 300s installs 300s (`MAX_STATEMENT_TIMEOUT_MS`), so a session can no longer disable it, and a clamped value logs a `warn!` when the pool is built. Only the operator-approved, fenced destruction paths below lift it, per transaction. **Long-running writer paths, audited** | Path | Behavior | |---|---| | Migrations / schema destruction (`with_exclusive_schema_destruction_lock`) | Unchanged: these lift lock and statement timeouts on their own detached connection | | Partition maintenance | Unchanged: own 5s `statement_timeout` | | Partition audit (`buzz-admin`) | Unchanged: 30s | | Community deletion inventory, purge and logical verification (`inventory_schema` whole-tenant counts, `purge_postgres` whole-community `DELETE` per table in one tx, `verify_postgres_logically_deleted` whole-tenant `EXISTS` per table right after the purge) | Lifted (`statement_timeout = 0`) transaction-locally by `lift_statement_timeout`, like migrations; lock and idle timeouts still bound them. Inventory counts each table in its own lifted transaction, so each table's lock and snapshot are released as soon as its count finishes, as the autocommit counts did. A cancelled purge would retry from scratch and eventually block, so a cap here would turn large-tenant deletion into an operator override | | Audit appends (`buzz-audit`, audit pool inherits the writer cap) | Waits on the audit-chain advisory lock. With `BUZZ_DB_LOCK_TIMEOUT_MS=0` (or at least the statement cap), `statement_timeout` (57014) cancels the wait instead of `lock_timeout` (55P03). The relay's audit worker now retries both with the same backoff instead of dropping the entry on 57014; admission refusal stays terminal | On HTTP routes a statement timeout already maps to the distinct retryable 503 (`api/mod.rs`, `statement_timeout_maps_to_distinct_503`). A cancelled REQ fails its history read instead of hanging. ## Tests - `p_tag_query_bounds_mention_side_by_time` (unit): page and COUNT SQL carry the `created_at` join and the `since`/`until`/`kinds` mention bounds, and the bounds are absent when the filter has none. - `p_tag_read_with_mention_bounds_keeps_results` (Postgres): through the production mention insert, unbounded, `since`, window and keyset-cursor reads return exactly the mentioned rows, and COUNT matches. Every bound sits exactly on a mentioned row's second and the cursor page includes a same-second sibling after the cursor id, so an off-by-one on either mention-side bound fails it. - `plain_read_is_cancelled_at_deadline_when_timeout_disabled` / `count_events_cancels_at_read_deadline_when_timeout_disabled` (Postgres): the production `#p` + `#h` + `since` shape and a plain COUNT are cancelled at 20s with the session timeout disabled. Previously these covered `#e` only. - `writer_statement_timeout_is_always_bounded` (unit) and `session_statement_timeouts_bound_writer_and_reader_sessions` (Postgres): through `Db::new`, writer sessions install `2min` / `5min` (for `0` and for 600s) / `45s`, and replica sessions install `20s`. - `deletion_whole_tenant_stages_outlive_the_writer_statement_timeout` (Postgres): on a scratch desired-state database with a 500ms writer cap, the real `inventory_schema`, `purge_postgres` and `verify_postgres_logically_deleted` each wait behind a held `ACCESS EXCLUSIVE` lock on `events` for over 1s and still complete, a plain statement is cancelled (57014), and pooled sessions still read `500ms` afterwards. A lifted transaction reads `statement_timeout = 0`, so a bounded lift (e.g. `'300000'`) fails it, as does removing any of the three `lift_statement_timeout` calls. - `mentions_query_is_tenant_scoped_and_joins_mentions_by_composite_key` / `needs_action_query_is_tenant_scoped_and_joins_mentions_by_composite_key` (unit): both feed queries carry the `e.created_at = m.event_created_at` join. - `replica_statement_cancel_propagates_without_writer_rerun` (Postgres): with the replica locked past its cap, each of the six routed reads above returns 57014 instead of the writer's copy. Removing the cancel arm in any of them fails it. - `thread_window_statement_cancel_propagates_without_writer_rerun` (Postgres): a thread-window read parked behind a replica lock and cancelled (57014; the window's own 1s `lock_timeout` would otherwise fire first) returns the cancel instead of the unlocked writer's window, and an auxiliary scan timed out on a replica session returns 57014 and leaves the session on the replica. Removing either guard fails it. - `audit_worker_retries_statement_timeout_until_original_entry_is_appended_once` (Postgres, `buzz-relay`): the production audit worker, with `lock_timeout` disabled and a 100ms statement cap, keeps the entry through repeated 57014 cancels while the chain lock is held and appends it exactly once after release. Treating only 55P03 as retryable fails it. - The existing writer and audit-pool GUC tests now expect `5min` for a disabled statement timeout. ## After deploy, watch - `buzz_db_route_decision{decision="replica",reason="statement_cancelled"}`: replica reads that now fail instead of falling back to the writer, including the replica-only lock case above. - `buzz_db_pool_acquire_wait_seconds` and `buzz_db_pool_waiters` on the writer pool: every generic REQ history read and COUNT now holds its connection for three extra round trips. ## Database checklist (`docs/database.md` § Review checklist) This PR changes schema-dependent `buzz-db` queries (the `#p` join and its `m.event_kind` pushdown). It does not change the schema. - **Phase:** none. No schema change, so this is not an expand, backfill or contract step. Nothing before or after it is required. - **Triggers, functions, procedures, rules, foreign keys:** none added. - **Row mutations / backfill:** no migration, and no row is mutated or backfilled. The one data-shape dependency is mention rows with a NULL `event_kind`: a kinds-filtered `#p` read no longer matches them (see the `kinds` bullet under Change). Relay writers always bind `event_kind`, so only rows imported out of band are affected. A read-only production count on 2026-10-20 found 0 such rows on both production reader databases. - **Contract migration:** N/A. - **Rule 4 locations:** none touched. No table, index or desired-state file changes, so no parity test applies. The new bounds use the existing `idx_event_mentions_pubkey_created` and `idx_event_mentions_pubkey_kind_created`. - **Deployed relay vs. new schema:** there is no new schema. The currently deployed relay and this relay run against the same tables. - **Lock and statement timeouts:** this PR is about them. Generic reads get a 20s deadline, replica sessions a 20s default, and writer sessions a 120s default capped at 300s. Community-deletion transactions lift `statement_timeout` per transaction; migrations and schema destruction are unchanged (see "Long-running writer paths, audited"). No index builds and no operator prebuild step. ## What got simpler - One read-deadline path instead of an `#e`-only branch in `query_events_on` and `count_events_on`. - The `#p` join SQL that was duplicated between SELECT and COUNT now lives in one `push_mention_join`. - The deletion tests share one scratch desired-state database helper instead of an inline copy. - "Is this query bounded?" now has one answer: yes, unless it runs in a migration, schema destruction, or community-deletion transaction. ## Identified debt (not in this PR) - `buzz-acp` re-sends the same filter after a timeout `CLOSED` (no backoff or narrowing). With the `#p` bound the `require_mention` read should finish well inside 20s, so this is a follow-up, not a blocker. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> |
