mirror of
https://github.com/block/buzz.git
synced 2026-11-06 17:31:19 +00:00
a8dd685bdebc197db9b5f69e1379752f7296eaef
894 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Brad Seiler
|
a8dd685bde |
test(auth): match Git controls to handler response contracts
Co-authored-by: Sarnow <1894dcbafaa4df1ac607a9bf879ac0b7d77fbccb0e0f4619ecc252c8adbad8a2@seiler-test.communities.public.builderlabstaging.xyz> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
5285894bd3 |
test(auth): bind route controls to exact outcomes and merged routers
Co-authored-by: Sarnow <1894dcbafaa4df1ac607a9bf879ac0b7d77fbccb0e0f4619ecc252c8adbad8a2@seiler-test.communities.public.builderlabstaging.xyz> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Luke Tornquist
|
cf8b6b26d7 |
fix(test): consume MembershipWrite in ban-matrix seeds
add_relay_member returns a #[must_use] MembershipWrite on main, so clippy -D warnings rejected the three ban-matrix seed calls. Map to the outcome before expect, as the neighboring tests do. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Luke Tornquist
|
bb5b0343f9 |
fix(test): pass relay_keypair to add_relay_member in ban-matrix tests
Main (#8482) added a relay_keypair parameter to Db::add_relay_member.
The merge in
|
||
|
Luke Tornquist
|
6565a0f107 |
fix(test): fail the route inventory on every shape it cannot classify
The scanner keyed `&str` route constants by bare name, so main's `api::profiles::SEARCH_PATH` collided with `api::gifs::SEARCH_PATH` and reported a false duplicate. Key constants by module and resolve qualified references against that module. Unclassified registrations are now hard errors instead of silent passes: `.route_service`, `.nest_service`, and `.fallback_service` (except the allow-listed SPA fallback in `build_router`), `.nest` outside `router::build_router`, and nest targets that do not resolve to exactly one router builder. A fixture self-test pins each shape and runs in `just test-unit` and `scripts/run-tests.sh unit`. Classify the three routes main added: `POST /profiles/search` (member gate, now in the HTTP ban matrix) and the operator-plane `GET /operator/communities/joined` and `GET /operator/communities/nip-fi-binding`. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
b52d8f6033 |
test(relay): cover current community entry points
Keep invitation and webhook-owner gaps explicit until their implementations are integrated. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
seiler
|
9855d86430 |
test(relay): align timeout admin regression
Signed-off-by: seiler <seiler@squareup.com> Co-authored-by: Codex <noreply@openai.com> (cherry picked from commit 31968772ebe1b30c2862966e46b75a9ad4b967b6) Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
seiler
|
d402ba6f42 |
test(relay): map community ban routes and regressions
Signed-off-by: seiler <seiler@squareup.com> Co-authored-by: Codex <noreply@openai.com> (cherry picked from commit dcb83eaec669ed044ab2f16c8b7612bd54ed1d9c) Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
7c29360d7d |
test(db): warm the reader floor before stalling new dials
Follow production boot order so the partial warm-pass assertion does not depend on three cold debug-build authentications completing in 150ms. Keep the production budget and exact stalled-dial accounting. Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz> Co-authored-by: McKeon <f87db83c2e79e370e2349ac850d9f04afe307b90f25a2830f50e825ba75e3cc5@seiler-test.communities.public.builderlabstaging.xyz> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
393d144b5c |
fix(relay): stagger authorization outage disconnects
Spread failed-lookup closes across scans with a stable per-socket offset, keeping the conservative revocation bound at 30 seconds. Share the database query deadline and verify sibling sockets close across three sweeps. Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
3bf280a668 |
fix(db): bound live authorization statements and cancellation
Apply a transaction-local server deadline below the caller timeout, preserve tighter operator limits, and close cancelled or failed checkouts. Verify blocked-query cancellation, pool recovery, and writer timeout restoration on the same backend. Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
548952b87d |
fix(relay): preserve refusal accounting across live admission
Restore audio stored-owner failure accounting and keep live policy revalidation separate from deny-set revocation. Preserve denied-only priority over weak removal claims; cover denied and unavailable root closes. Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
9e798cf498 |
fix(relay): preserve token-owner admission during revalidation
Keep NIP-FI owner delegation independent of the NIP-OA flag. Adapt membership fixtures and session identities to current main, and cover final admission through owner removal for both socket routes. Co-authored-by: Sarnow <1894dcbafaa4df1ac607a9bf879ac0b7d77fbccb0e0f4619ecc252c8adbad8a2@seiler-test.communities.public.builderlabstaging.xyz> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
seiler
|
c5e74c9de1 |
fix(auth): log captured owner membership failures
Signed-off-by: seiler <seiler@squareup.com> Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
seiler
|
690f9da0ce |
fix(relay): prioritize stale authorization targets
Signed-off-by: seiler <seiler@squareup.com> Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
40500a2fd3 |
test(relay): admit live agents through owners before revalidation
Keep a clear audio participant present through policy eviction so the huddle remains active for the database-failure phase. Reject bare closes with a labeled diagnostic. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
b3870a6fe3 |
test(relay): classify live revalidation as external network infrastructure
Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
ab8aecd687 |
test(relay): bound live workflow handshake and control reads
Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
0ca5a04447 |
test(relay): measure live revocation without Redis delivery
Exercise signed root and audio admission against an isolated running relay, durable direct-member and owner bans, a responsive bystander, and bounded database-failure closure. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
a5e5661010 |
test(relay): import root fixture shared ownership type
Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
8382528486 |
test(relay): retain root lifecycle registration during revalidation
Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
1b1f5a23f4 |
test(relay): bind durable revocation to socket admission
Authenticate through root and audio handlers before a durable-only ban. Removing either production enrollment call must fail its regression. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
seiler
|
4aaf747ed2 |
refactor(relay): address revalidation review
Restore CommunityDisconnectReason near the top of state.rs and document scan deadline and reconnect-load limitations. Authorization behavior is unchanged. Signed-off-by: seiler <seiler@squareup.com> Co-authored-by: Codex <noreply@openai.com> (cherry picked from commit a4d1ab288d761d44eba25cc62d25b7d2858788dc) Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
seiler
|
1b81b01b01 |
Add durable live member authorization checks
Signed-off-by: seiler <seiler@squareup.com> Co-authored-by: Codex <noreply@openai.com> (cherry picked from commit 92f13229ed3038d63f2a2d9c889bfd4c212e38e2) Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
c3f4c01946 |
Add invitation history repair as a separate maintenance change
Revoke historical legacy candidates and clear examined admission markers under the deletion/admission locks. Require separate approval before core deployment and a controlled rollout that prevents new timeout/mute history. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> Signed-off-by: seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com> |
||
|
Brad Seiler
|
a1f79e3aba |
feat(relay): persist NIP-FI upgrades (BUZZ-510) (#8985)
## Summary Record that a community requires NIP-FI and provide an idempotent operator API to enable it and recover the original receipt. Acceptance checks the current owner, host, lifecycle and existing issuer trust in one transaction. It never creates trust. Receipts survive community deletion so an operation UUID cannot be reused. New acceptances default to disabled through `BUZZ_NIP_FI_UPGRADES_ENABLED`. Disabled calls return HTTP 503 `upgrades_disabled`; identical retries and authenticated readback still recover existing receipts. Policy is permanent: stopping new calls or rolling back code does not clear it. Last-issuer removal is refused for required communities, and bulk backfill leaves missing bindings for explicit operator repair. Apply additive migration 0082 before code. Existing communities remain optional. This layer records acceptance and reports only `pending` or `unavailable`; deploy and verify [per-request enforcement](https://github.com/block/buzz/pull/8988) and [session activation](https://github.com/block/buzz/pull/8991) before enabling acceptance. The global enforce floor remains. See [the operator contract](https://github.com/block/buzz/blob/seiler/BUZZ-510/relay-auth-upgrade/docs/community-auth-upgrades.md) for signing, recovery and containment. ### Related issue [BUZZ-510](https://linear.app/squareup/issue/BUZZ-510/relay-persist-community-nip-fi-requirements-and-expose-idempotent). Base of native stack `8989`; retains trunk's enforcement telemetry from [#8974](https://github.com/block/buzz/pull/8974). Includes a separate commit that repairs an observer timing flake in a desktop test required by the repository gate. ### Testing Controlled local regressions detect the old disabled error code and removal of the last issuer with `allow_last=true`; intended source is restored. Full PostgreSQL validation and the updated live relay/admin exercise remain pending. Human acceptance remains open. No deployment or production activation. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz> |
||
|
Luke Tornquist
|
4813b8cd15 |
relay: time every WebSocket read and every stored EVENT outcome (#9052)
## Summary
Two gaps in relay telemetry: WebSocket reads (`REQ`, NIP-AR sync,
`COUNT`) have no latency or outcome metric, and
`buzz_event_processing_seconds` only measures stored EVENTs that
succeed.
**What this does**
- **`buzz_ws_read_seconds{type, outcome, reason}`** (new histogram).
It's recorded from `access::ReadLog::drop`. Every WebSocket read already
finishes through that one seam, whether it ends `served`, `denied`,
`error`, or `aborted` (future dropped mid-read), so this adds no new
exit paths and changes no handler. Duration runs from the moment the
handler takes the frame until `EOSE` is queued (`REQ`), the initial set
ends (sync), or the total is sent (`COUNT`). `reason` is the existing
fixed denial/error code (`query_timed_out`, `database_error`,
`not_channel_member`, …), or `none`. The histogram's `_count` doubles as
the outcome counter, so one metric gives both read latency and read
failure rate.
- It's recorded *before* the refusal budget check, because the budget
only exists to suppress log lines. When `refused_read` is over budget,
it records the sample without building the log line, so its cost under a
refusal flood stays the same.
- **`buzz_event_processing_seconds{outcome}`**. Before this change it
was recorded only when `ingest_event` returned `Ok`. A store that was
slow *and failing* (statement timeouts, pool exhaustion) produced no
samples, so latency looked better while the store was degrading. It's
now recorded on every persistent-ingest outcome: `accepted`,
`not_accepted` (for example a duplicate), `invalid`, `auth`, `error`.
Queries that sum across all outcomes still work.
**Cardinality.** Every label value is a `&'static str` from a
code-defined set: 3 types × 4 outcomes, with about 25 reasons that only
appear alongside `denied`/`error`. On purpose, there's no `kind` label:
`bounded_kind_label` still admits around 200 values, which is too many
to cross with histogram buckets.
**What got simpler:** `ReadOutcome::parts()` replaces the inline
outcome→label match in `Drop`, so the log line and the metric share one
mapping. There's no new instrumentation seam: the read metric reuses the
access log's lifecycle.
### Related issue
None found. This came out of a monitoring-gap review: without these
signals, alerts can't catch a slow or failing WebSocket read path, or a
failing EVENT store.
### Testing
-
`access::tests::every_read_is_measured_even_when_its_line_is_suppressed`
(unit). It drives the real `ReadLog` constructors and `Drop`: 20
admission refusals against a 16-line budget, a handler denial past the
budget, a `COUNT` error, a served sync and an aborted `REQ`. It asserts
the exact `(type, outcome, reason, samples)` rows. Two mutants each fail
it: removing the over-budget branch in `refused_read`, and moving the
measurement after the budget check in `Drop`.
-
`handlers::event::tests::postgres_tests::event_processing_is_timed_for_accepted_and_refused_ingest`
(Postgres lane). It sends one accepted text note, then a refused one
from a cached-ban sender, through `handle_event`, and asserts one sample
each under `accepted` and `auth` (a ban maps to the `auth` class).
Removing the `Err`-arm measurement fails it.
- Blox gate at `b314df43f2` (tree `4ca9ebc1`): `cargo fmt --check`
clean. The new Postgres test passes, and removing the `Err`-arm
measurement fails it. buzz-relay unit lane (`cargo nextest run -p
buzz-relay --lib --bins`): 1708/1708 passed.
- At `539796bd51` (the same code; only the second commit's test
assertion differs): workspace clippy (`--all-targets -D warnings`)
passed. The new unit test passed, and each of its two mutants failed it.
buzz-relay Postgres lane: 604/605 passed. The one failure was this
test's old `invalid` expectation, which `b314df43f2` corrects.
- CI at `b314df43f2`: both new tests appear in the CI logs as passing
(Rust / Unit Tests, PostgreSQL Domain / PostgreSQL Tests).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Luke Tornquist
|
25abfc4117 |
fix(loadgen): bound desktop reads at stop, keep timely EOSEs, model presence and probes (#9016)
## Summary Follow-up to #9005, addressing the review findings that merged unaddressed. None of them affect a relay. `buzz-ws-client` gains one method. Each one skews what the desktop client measures when the relay is slow, which is the case the load generator exists for. - **Reads stop at the stop.** A desktop HTTP read (including NIP-11) now races the run's stop and is abandoned there without being recorded, as EOSE waits already were. Before, a read that outlived the 15 s shutdown grace got its client cancelled and failed an otherwise healthy run. `Fetcher::finish` also keeps the read task's handle until the task ends, so a client cancelled during the join still aborts the task instead of detaching it. - **Root lookups are bounded.** The desktop's seeded-root lookups, for thread recovery and replies, now run under the desktop's 30 s read timeout and stop, not reqwest's 60 s. `roots()` takes an optional bound, and the basic human passes none. If the stop cuts a reply's lookup short, the reply isn't published. That applies to the basic human too: a reply whose root lookup finishes after the stop is dropped. - **An EOSE that arrives in time stays in time.** While a publish waits for its OK, `NostrWsConnection` buffers every other frame. The session loop used to judge EOSE deadlines before reading that buffer, and read one frame per pass. It now handles every frame already in hand (at most 256 per pass) before judging deadlines. Each frame is judged by when it arrived, not when it is read. `NostrWsConnection` keeps each buffered frame's arrival time, and the new `next_event_received` returns it. So an EOSE that arrived after its deadline while an OK wait held it counts as a timeout, and so does a frame that answers the probe late. Past that cap, a pass defers due deadlines, for at most one timeout from the first pass that defers them. While it is deferring, the pass only drains, so another OK wait can't expire an EOSE that's already in hand, and sustained traffic can't postpone a missing EOSE forever. - **Presence and the silence probe.** Both modes now open the app's live-only `presence` route (`{"kinds":[20001],"authors":[…],"since":…,"limit":0}`, up to 256 co-members) alongside the profile and membership routes. As in the app, presence takes no part in establishment: it isn't resumed or carried, and if it fails it doesn't cut a renewal's carry for the other routes. After a tick with no frame since the previous one, they send the app's probe, `REQ probe-N {"kinds":[44100],"#p":[viewer],"limit":0}`, which shows up as `desktop_probe` and is closed at its EOSE. Only frames actually received count, including a publish's OK when it arrives before the probe's deadline. A reaction with no target or a publish whose OK timed out doesn't count. Expiry is judged after the drain against the probe still outstanding. If no frame arrives within the EOSE deadline, the session ends as failed and the client reconnects with backoff. Before this, desktop runs left out presence fan-out and idle-client probes, so they understated load. - **Seed log.** "memberships added (N people)" now reads "memberships ensured", since a rerun adds nobody. - **Docs.** `perf/LOADGEN.md` notes that the six-read cap and flat 30 s timeout are simplifications (the app has neither), and that `legacy` is a lower bound. The pre-fix app also opened a per-channel route for each open or non-member channel and re-sent every route each tick, and the model leaves both out. Not done: an explicit count of REQs abandoned at a renewal (report polish; planned cancellations still aren't errors). ### What got simpler Reads abandoned at the stop now go through one helper, `run::bounded`, which the desktop reads and the root lookup share. Before, each read had its own timeout and its own stop check. `global_reqs` returns the routes the client actually opens, not a fixed pair. The probe and presence routes add one `Route` variant, one op and about 40 lines of session code. `Route::establishes` is now the one place that says which routes take part in establishment, replacing the two `Replay` special cases. "Heard" used to be inferred by comparing activity schedules before and after each step. It now comes from frames only, so `publish` reports whether the relay answered and that comparison is gone. ## Testing On a Blox workstation, at the tree of this PR's head: - `cargo fmt --all -- --check`, `cargo clippy -p buzz-test-client -p buzz-ws-client -p buzz-cli --all-targets -- -D warnings` - `cargo test -p buzz-test-client --lib` (parallel) and `cargo test -p buzz-ws-client`, plus the `loadgen::desktop` tests five more times New tests, each confirmed to fail when its fix is reverted: - `a_read_in_flight_at_the_stop_is_abandoned_not_waited_for`: an 8 s roster read at the stop. Without the stop race the client outlives `drive`'s 5 s stop limit. - `a_cancelled_finish_still_aborts_the_read_task`: aborting `finish()` mid-join ends the read task. With the old `take()`, the task is detached. - `an_eose_in_time_is_not_expired_behind_a_slow_ok`: EOSEs at 1.2 s, publish OKs at 3.5 s, deadline 2.5 s. Without the drain, all four routes time out. - `a_silent_tick_probes_and_an_unanswered_probe_reconnects`: the first connection never answers its probe, so the client reconnects and its next probe is answered. - `a_probe_answered_behind_a_late_ok_does_not_reconnect`: the probe's EOSE arrives in time, but it sits behind a heartbeat whose OK comes after the probe's deadline. With expiry judged against a deadline captured before the drain, the session fails and reconnects. - `a_publish_ok_in_time_answers_an_outstanding_probe`: no probe EOSE ever arrives, and heartbeat OKs answer each probe. - `a_reaction_with_no_target_is_not_heard_as_a_frame`: targetless reactions several times a second still leave the tick silent, so it probes. - `a_reply_whose_root_lookup_the_stop_cut_short_is_not_published`: the stop cuts an 8 s root search short, and OKs are withheld. No `kind 9` EVENT reaches the relay, and the client returns promptly. - `a_backlog_defers_deadlines_for_at_most_one_timeout`: the deferral bound and its reset. - `a_long_backlog_behind_slow_oks_keeps_its_timely_eoses`: 600 replayed events per REQ behind 3.5 s OKs, with a 1.5 s deadline and heartbeats every second. The test fails if the deferral is timed from the REQ instead of from the first pass that defers, and it fails if a second OK wait can run between the deferred passes. - `presence_is_a_live_only_global_only_while_someone_is_watched`: the route's order and its live-only filter in both modes, and that it's absent with nobody watched. - `a_late_eose_is_not_answered_behind_a_slow_ok`: the timely case above with EOSEs at 3 s instead, so all four routes time out, not succeed. - `a_probe_answered_late_behind_a_slow_ok_reconnects`: a probe's EOSE arrives 0.2 s after its deadline while a heartbeat waits for its OK, so the session fails and reconnects. The test also checks that a heartbeat followed that probe on the same connection, so the test can't pass on a probe that simply went unanswered. - `an_answered_probe_is_closed_at_its_eose`: `CLOSE probe-1` goes out before any next probe. - `routes_establish_once_and_a_renewal_holds_them_until_a_cut` gains a case: a renewal where presence times out keeps the other routes carried and establishes nothing. Each new test fails when its fix is reverted (seven mutants in the first round and six in the review round, run on Blox). The test harness now has a tick override that can go below the scenario's 1 s floor, and records EVENT frames by kind. The fake relay sends a REQ's replayed events after the REQ's delay, not before, and the fake API's slow reads sleep in `block_in_place`. No existing test changed behavior. Existing expectations moved for the presence route (one more live EOSE per client) and for reads now abandoned at the stop. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
John Matthew Tennant
|
f9ef721e68 |
fix(media): keep transparency in image thumbnails with WebP (#9003)
## Summary
This fixes transparent images showing a black background. Image
thumbnails were always JPEG, which can't store transparency, so a
transparent PNG or GIF upload, such as an agent avatar, rendered on
black wherever a client showed its thumbnail.
The relay now writes a lossy WebP thumbnail at `{sha256}.thumb.webp`,
which keeps transparency, alongside the existing `{sha256}.thumb.jpg`:
- **Encoding:** the WebP comes from libwebp through the `webp` crate,
built from vendored C source, so the relay needs no system library. A
GIF uses its first frame.
- **Upload responses:** the upload descriptor's `thumb`, and its
per-tenant rewrite, point to the WebP.
- **JPEG kept:** installed clients derive `.thumb.jpg` themselves, so
the relay keeps writing it until clients move to WebP with a JPEG
fallback. Existing images still have only their JPEG thumbnail.
- **Serving:** `/media/` serves both variants, each with its own content
type. Other `.thumb.*` extensions still return 404.
- **imeta:** a `thumb` may name either variant, and the relay checks
that the named object exists.
- **Storage sweep:** the bucket taxonomy counts both variants as the
image's thumbnail bytes instead of classifying `.thumb.webp` as unknown.
| 320px thumbnail | JPEG | PNG | lossy WebP (q80) |
|---|---|---|---|
| Transparent avatar (starter-team `fizz.png`) | 11.3 KB, black
background | 65.1 KB | 13.8 KB |
| Opaque avatar (`sprout-agent.png`) | 23.6 KB | 156.5 KB | 12.3 KB |
| Screenshot (`channel-thread.png`) | 13.4 KB | 62.5 KB | 6.4 KB |
Encoding a WebP thumbnail takes 3 to 20 ms.
### Related issue
Supersedes #8772, which served a PNG at the `.thumb.jpg` path.
### Testing
Red/green against a local relay on `:3000`, built from the branch under
test, with Postgres, Redis and MinIO from `docker compose`:
```bash
cargo test -p buzz-test-client --test e2e_media test_transparent_png_thumbnail_is_webp_with_alpha -- --ignored
cargo test -p buzz-test-client --test e2e_media_extended test_ws_imeta_webp_thumb -- --ignored
```
- **On `main`:** both tests fail. The descriptor's `thumb` is
`.thumb.jpg`, and the WebSocket send is rejected with `invalid: imeta
thumb must be a local .thumb.jpg path`.
- **On this branch:** both tests pass:
- Uploading a half-transparent PNG returns a `thumb` ending
`.thumb.webp`.
- HEAD and GET on that URL return `Content-Type: image/webp`.
- The decoded thumbnail keeps alpha 0 on the transparent half and alpha
255 on the opaque half.
- `.thumb.jpg` still returns `image/jpeg`.
- A kind 9 message whose imeta `thumb` names the `.thumb.webp` is
accepted.
## Screenshots
The starter-team `fizz.png` avatar uploaded to a relay built from this
branch. Both thumbnails are shown as round avatars beside the original
PNG. `.thumb.jpg` fills the transparent area with a solid disc, and
`.thumb.webp` keeps the transparency.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: John Matthew Tennant <jtennant@squareup.com>
Co-authored-by: Simone <0828be588d5c9c4d092c5b1cf01761d5aef63d6b104807fd8e8284d517dd1a99@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Luke Tornquist
|
d21f0c0fdb |
perf(relay): keep 1 s read caches warm when the pool is saturated (#9012)
## What The relay's 1-second read caches (restriction state, tenant host, open channels) now date each entry from when the read checked out its pool connection, not from when the cache miss began. Concurrent misses on the restriction and tenant-host caches now share one database read. The open-channels cache already did this. ## Why Each of these caches serves an entry for 1 s after "the read that produced it began". Today that start time is taken before waiting for a pool connection. Once pool waits reach about a second (for example, many clients reconnecting at once), every entry is already expired by the time it lands. The cache hit rate falls toward zero exactly when the pool is saturated, so every request does its own read. Those reads queue for the same connections, which stretches the waits further. Requests that time out in the pool then fail closed (`internal error checking restriction state`, `community active check failed; refusing the socket`), and their clients reconnect and add more load. In a local synthetic reconnect load test with a 50-connection pool: - the restriction cache served fewer than one request in five from cache; - the tenant-host cache, which has a single key for the whole deployment, missed more often than it hit; - the open-channels cache missed hundreds of thousands of times on expired entries. ## How - **Stamp at checkout.** `buzz_db::ReadAt<T>` carries a value plus the instant its connection was checked out. The statement's snapshot is no older than that instant, so serving the value for 1 s after it keeps the documented staleness bound, however long the read queued. Three `*_read_at` methods return it: `moderation_restriction_state_read_at`, `lookup_community_by_host_read_at` and `get_listed_open_channel_ids_read_at`. The existing methods now call these and drop the value's timestamp. Each operation keeps one datastore span under its existing name, carried by the `*_read_at` method. - **Share concurrent misses.** `SharedReads<K>` is the per-key gate the open-channels cache already used, moved into its own module. The open-channels cache, the restriction cache and the tenant-host cache now all use it. The first miss reads; the others wait for it, then recheck the cache. A waiter waits for at most the writer pool's acquire timeout plus `SHARED_READ_STATEMENT_WAIT` (1 s). That covers the leader's own wait for a connection, which is what grows in a storm, so waiters keep sharing its read instead of queueing duplicate reads behind it. A failed or cancelled leader releases its waiters at once. A leader whose statement hangs stalls them for at most 1 s past the acquire timeout, never the statement timeout. - **Reject hosts no community can have.** `bind_community` rejects a host longer than `communities.host VARCHAR(255)` can hold (`buzz_core::tenant::MAX_HOST_LEN`, counted in characters) as unmapped, next to the empty-host check. An unauthenticated caller's arbitrarily long host therefore never reaches a resolver, cache, gate or the database. No staleness bound changes: every entry is still served for at most 1 s after its read's snapshot. Invalidation paths are unchanged. A read in flight at an invalidation can still refill the old row for at most that bound, as documented today. ## What got simpler The open-channels gate's hand-rolled `try_lock`/`timeout` block is replaced by `SharedReads::turn`. One gate implementation now serves three caches instead of one implementation that only one cache had. The gate's wait is set once per gate rather than passed at each call site. Provisioning and host binding share one `MAX_HOST_LEN` instead of a private constant. ## Tests - `shared_reads::tests`: one caller reads while the rest wait; a waiter gives up on a hung read after its bound. - `concurrent_restriction_and_host_misses_share_the_in_flight_read`: waiters on both caches use the leader's entry; with an unreachable database, any waiter that read on its own would fail. - `postgres_tests::cache_misses_behind_a_held_pool_share_one_fresh_read`: with a one-connection pool held for longer than every max age, eight concurrent misses on each of the three caches cost one database read, and each entry still lands fresh. - `tenant::tests::an_overlong_host_fails_closed_before_the_resolver`: a host over the limit fails as unmapped with no gate entry and no lookup; a host at the limit, including one in multibyte characters, is still looked up. - Mutation-checked: each of the following turns these tests red: stamping any of the three entries before the pool wait, dropping either new waiter path, letting waiters give up after the cache's max age, dropping the host length check, or counting the host length in bytes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Kalvin C
|
3d26ed7dfd |
feat(relay): match verified names in profile search (#9019)
## Summary `POST /profiles/search` only matches kind:0 `name` and `display_name`. In communities that publish NIP-FI verified profiles, a person's surname or directory username often appears only in their relay-signed kind:30382, so typeahead cannot find them by it. - Where the community advertises verified profiles, the search also matches the `display_name`, `username` and `full_name` tags of each current kind:30382 head signed by the relay key. - Verified names need a trailing prefix of at least three characters, as in NIP-50, so a short prefix doesn't make the relay rank every verified head. kind:0 keeps one-character typeahead. Lookups from heads to `events` also match `created_at`, so Postgres prunes to one partition. - Results are paged by person. Each person returns their current kind:0, when they have one, then their current kind:30382, when they have one, whichever matched. A person is ranked by their best match across both sources, using the existing tiers. - NIP-11 `federated_identity.public_profile` gains `searchable: true`, so clients can rely on the response instead of a separate kind:30382 read. - Migration `0081` adds a GIN expression index over the head's name tags on `nip_fi_profile_heads` (built-in `jsonb_path_query_first` and `to_tsvector`, no new functions). The table holds one row per verified person. Like `0078`, the migration builds the index only while the table is empty; populated databases prebuild it with `CREATE INDEX CONCURRENTLY` (`docs/profile-search.md`), and the migration then checks it is valid and has the expected definition. Compatibility: the request schema is unchanged. Existing clients that keep only kind:0 events work as before and also find people who match only by verified name. Trust stays pinned to the relay `self` key; the event shape follows NIP-85 user assertions, but discovery is relay-advertised rather than kind:10040. ## Testing - `buzz-search` Postgres integration: new `profile_names_match_verified_names` covers forged-key heads, replaced heads, other communities, people without kind:0, per-person grouping, paging, and words split across sources; the existing profile-name tests pass unchanged. The test fails when the author filter is inverted. - `buzz-relay` end to end in enforce mode: `profile_search_matches_verified_names_where_advertised` publishes through the real writer and finds the person by surname only where verified profiles are advertised. - Desired-state and migration index definitions match. - `profile_names_use_verified_name_index` asserts the search scans the new index. Migration `0081` checked against an empty table, a rerun, a populated table without the index (raises), the documented concurrent prebuild, and an index with a different definition (raises). - Latency, 50k people each with a kind:0 and a verified head, release build, local Postgres, best of 5, 30 people per page: 11–22 ms with verified names versus 1–8 ms kind:0 only. `EXPLAIN` shows a bitmap scan on the new index. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kalvin Chau <kalvin@block.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
865b150371 | fix(db): use ScratchDb in the storage usage delivery test (#9032) | ||
|
Will Pfleger
|
0a06c0b3cb |
feat(relay): log reads, logins and connections on access target (#8376)
🤖 Adds an access log for relay reads, logins, connections and HTTP requests on the dedicated `buzz_relay::access` target. It writes to stdout, not `audit_log`, so reads never take the audit advisory lock. The log is shaped for volume: one line per HTTP request, one line per WebSocket read, and one line per WebSocket connection. ## Lines per operation | Operation | Lines | |---|---| | HTTP request on a matched route | 1 (`http`), except the skip list below | | HTTP request matching no route | 1 (`http`, `route="<unmatched>"`) at status 400 and above, otherwise none | | WebSocket connection | 1 (`close`, carrying the login) | | WebSocket `REQ` / `COUNT` / NIP-AR sync | 1 each (`req` / `count` / `sync`), plus the `buzz_ws_reads_total` counter | | WebSocket NIP-42 login | 1 (`auth`) per decided or cut-off attempt, capped after the decision as below | | Audio login | 1 (`auth`, at its final admission check) | | WebSocket `EVENT` | 1 (`Event ingested`, not on the access target) | ## HTTP requests One middleware, `http_access_log`, writes every HTTP line. It sits between `track_metrics` and `http_trace_layer`, so rebuilt errors, body-limit refusals and localhost refusals all reach it. It writes the line when the response is ready, which is not proof the body was delivered. A request dropped before then logs `request=aborted` at WARN with no `status`. - Every line carries `route` (the matched route template, never the raw path), `peer_addr` when the listener has one, `status`, `time_to_response_ms` and `http_method` (the HTTP verb). The verb stays on the line itself because the `http.request` span is absent under an access-only filter and when a pending request is dropped. - Handlers record what they learn on the request's `AccessSlot`: the login (`method`, `pubkey`, `pubkey_verified`, `community`), the decision, the filter summary, and the route's own fields. The decision is `outcome` (`success` / `denied` / `failed` / `challenged`) with a fixed `reason`, or `undecided` when the route recorded none. A refusal is final: a later admission doesn't replace it, so a replayed NIP-98 proof that verifies and is then refused logs `failed` / `nip98_replay`. - `outcome` describes the login. What the route did is in `status` and the route's fields: a rejected `/events` write logs `outcome=success` with `status=400`, `accepted=false` and its `reject_reason`. - Route fields: - `/query` and `/count`: the filter summary and `result_count`. An incremental `/query` response (`{changes:[...]}`) counts its changes, removals included. Artifact reads record their summary before the query runs, so a read dropped mid-query keeps it. - `/events`: `accepted`, `kind`, `event_id`, and on a refused write the parse `category` / `line` / `column` or a fixed `reject_reason` (such as `invalid` or `restricted`), never the ingest message. - Invite mint: `invite_id`, `expires_at` and `max_uses`. Invite claim: `joined`. - Admin feedback attachment reads: `feedback_id` and `attachment_sha256`. With `BUZZ_ADMIN_AUTH=disabled` they log `method=network_trusted` with no key, and Host or Origin refusals log their fixed reason. - Every other route logs only the login fields. - No field can hold an invite code, a token, or a Blossom read hash. Media routes log the `/media/{sha256_ext}` template. - Skip list: NIP-11 and other non-upgrade requests on `/`, `/info`, `/.well-known/nostr.json`, `/health`, `/_liveness`, `/_readiness`, and the SPA and static files the relay serves when enabled (web assets, invite landing pages, git web pages, the admin bundle) at any status. Successful WebSocket upgrades (101) log no `http` line. Refusals before the upgrade on `/` and the audio path log one, with the status actually sent. - Routes that aren't Nostr-key logins (the command-token disconnect endpoint, the workflow webhook, the git server's internal callback) log `outcome=undecided` with no key. - Dev-mode `X-Pubkey` requests log `method=x_pubkey`, `pubkey_verified=false`. - A successful `/query` or `/count` that returned nothing (`result_count=0`) logs at DEBUG: on staging, 93% of `/query` lines were empty polls. Denied, failed, aborted and error reads, reads with no `result_count`, writes and logins keep their levels. - Other successful lines and git's credential-less first request (`challenged`, with its 401) are INFO. Refusals, statuses of 400 and above, and aborted requests are WARN. In Enforce mode the line records the remapped status (for example 403). Denial bodies and headers are unchanged. ## WebSocket reads - `REQ` / `COUNT` / `sync` lines carry `pubkey`, `conn_id`, `sub_id`, the filter summary, the outcome (`served` / `denied` / `aborted` / `error` plus a fixed reason), and `events_queued` (REQ and sync) or `count` (a COUNT that counted). A `REQ` is `served` only once its EOSE is queued to the client. A read logs WARN only when it ends in `error`. `served`, `denied` and `aborted` are INFO, since `aborted` means the read ended before completing: the client disconnected or sent CLOSE, a same-ID `REQ` replaced it, or the session was cancelled. - A NIP-AR sync subscription writes its `sync` line when its initial set completes, not when live delivery ends. Its summary is the registered live filter (kinds and channels, never artifact predicates), and `events_queued` counts the initial set. - `REQ` lines stay at INFO even when empty at EOSE, because the subscription can still deliver live events later. - Filter summary, shared with HTTP `/query` and `/count`: `kinds`, channel ids, authors and `p` values each list the first 10 distinct values. The exact `kind_count` / `channel_count` / `author_count` / `p_count` (summed per filter) appears only when the list is `*` or shows fewer values than the filters held. A dimension that any filter leaves open logs `*`. `since` / `until` / `limit` take the widest bound and are absent when any filter leaves them open, so no field understates what the read covered. `ids` is a count only. NIP-50 search logs `search: true`, never the text. Only well-formed channel ids and pubkeys are listed. An artifact read's summary uses only the standard dimensions of each raw filter, so artifact predicates and client-defined tags (such as `#1`) are never logged. The summary is skipped when the access target is off. Under `=warn`, every WARN line, HTTP or WebSocket, still carries it. - Per connection, only the first 16 reads that are refused before they run (`auth_required`, quota, concurrency, overload) or denied are logged. The rest are counted as `reads_suppressed` on the close line. A refusal that arrives after the close line has been written, from a read still running when the client left, logs its own line, so every refusal is either logged or counted, never both. Served reads are never capped. ## WebSocket read counter `buzz_ws_reads_total{type="req"|"count"|"sync"}` counts every parsed `REQ`, `COUNT` and NIP-AR sync frame before admission, exported at zero from startup. It is a metric, so it counts at any `RUST_LOG`, including production's current `error`. ## WebSocket logins A NIP-42 login writes one `auth` line when it is decided. A successful login logs INFO `outcome=success` at its commit point, decided under the connection's close-reason lock, so a disconnect racing the login (session expiry, admin disconnect, revocation, ban or roster removal) either wins, and the login logs no success, or loses and closes the connection right after the success line. A refused or failed login writes one WARN `auth` line with `outcome`, a fixed `reason` from the shared `AuthReason` code set, `pubkey` and `pubkey_verified`. Root WebSocket `auth` lines carry `peer_addr`. AUTH frames after the login is decided are capped at 16 per connection, and the rest are counted as `auth_suppressed`. An attempt cut off by close or session expiry logs `aborted`. Audio logins log one `auth` line at their final admission check. ## Connections The close line is the connection's only line. It is owned by the upgraded socket, so it is written exactly once however the connection ends: a normal close, a refusal while checking the community is active (`refused=not_admitted`), the connection limit (`refused=connection_limit`), or the handler being dropped. It carries `conn_id`, `community`, `peer_addr`, `duration_ms`, the login, `events_queued` / `bytes_queued` (frames the writer accepted and flushed, not client delivery), and the two suppression counters. ## Error text - Database errors log as `DbError::kind()` (SQLSTATE or variant), including the command-executor event insert. Search errors, on both read paths and in the search-cancel task, log as `SearchError::kind()` / the same SQLx classifier. Workflow database and webhook errors log as their codes, since webhook URLs can be secrets. - Git NIP-98 failures log a fixed code. Git subprocess stderr moved to DEBUG because hook output can contain message text. ## Operator notes `.env.example` documents the target, its `event` values (`http`, `auth`, `req`, `count`, `sync`, `close`), the skip list, the levels, and how to turn it down: `buzz_relay::access=warn` keeps refusals, errors and aborted HTTP requests, and of WebSocket reads only those that end in `error`, and `=off` drops everything. It also lists the removed lines: `NIP-42 auth successful/failed`, `WebSocket connection established/closed`, `event=connect`, `HTTP bridge request`, `Event ingested via pipeline`, `relay invite minted`, `relay member added via invite`, `admin feedback attachment read`, and `addr` (now `peer_addr`). JSON lines no longer repeat the current span as `span`, since `spans` already ends with it. A line logged under an explicit `parent:` span loses that parent's fields. The default OTel export filter sets `buzz_relay::access=off`, so access lines aren't exported a second time as span events. `peer_addr` is the TCP peer. `BUZZ_CLIENT_IP_HEADER` optionally names a request header (for example `cf-connecting-ip` behind Cloudflare or `x-real-ip` behind nginx, matched case-insensitively). When set, `http`, `auth` (root and audio), `close` and connection-limit lines also log `client_ip`: the rightmost comma-separated entry of the header's last field, only if it parses as an IP address. It never replaces `peer_addr`, and it is only trustworthy when every request arrives through a proxy that overwrites the header. --------- Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
Shu Hui
|
fa7a3bfbda |
feat(relay): push per-community storage usage to kgoose (#8714)
### Why?
To bill hosted communities for storage in Orb, kgoose needs
per-community storage usage. The relay already computes it in a daily
snapshot, but nothing delivered it anywhere.
## Summary
- Adds opt-in `BUZZ_STORAGE_USAGE_URL`. When it is set, one
leader-elected relay pod checks every 5 minutes for a new storage
snapshot and POSTs it to kgoose's `POST /v1/buzz/storage-usage/record`
(squareup/cash-server#144285) in consecutive batches of up to 50
communities until every community is sent.
- Each batch is `{measured_at, communities: [{community_host,
storage_bytes, storage_objects}]}`. Active communities with no stored
bytes are reported as zero, and re-sends are safe because kgoose bills
each community once per UTC day.
- Retries on 408, 429, 5xx, and transport errors. No migration;
documented in `docs/storage-accounting.md`. kgoose authenticates the
relay as the `buzz` mesh service, so the relay needs mesh egress to
kgoose-builderbot.
### Related issue
None found. Complements #7899 (snapshot history in the DB). Receiving
endpoint: squareup/cash-server#144285.
### Testing
Unit tests cover batching, retry and reject handling, zero-filling, and
a real HTTP delivery that checks the exact kgoose request body. A
Postgres-gated DB test covers the new reads. End to end, a local relay
delivered to a fake endpoint once per new snapshot.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Shu Hui <shu@squareup.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
Salman Mohammed
|
6847882b14 |
fix(relay): stop logging unmapped-host NIP-FI denials as server failures (#8997)
## What this does In enforce mode, a request whose Host maps to no community gets NIP-FI's `authorization_unavailable` 503 (`docs/nips/NIP-FI.md`, Rejection and privacy table). tower-http's default classifier treats every 5xx as a failure, so each of these requests also logs a `tower_http::trace::on_failure` error. That line has no host or path, only `503` and `latency: 0 ms`. When a scanner probes a host this relay doesn't serve, it produces thousands of these lines in a minute, and they trip error-rate alerts even though nothing is down. This PR keeps the wire response the same and stops counting these denials as server failures: - `nip_fi_core::community_denial` builds the same `http_denial` response. When the Host is confirmed unmapped, it adds an internal `UnresolvedHost` response extension. Extensions never reach the wire, so the NIP-FI denial contract ([FI-TRACE-DENIAL-ORACLE]) still holds. - Each site that turns a community-resolution failure into a denial now uses it: - the router guard (`nip_fi_assertion_guard`) - the WebSocket upgrade (`check_nip_fi_at_upgrade`) - HTTP admission, for routes the guard exempts (`Stage::Community`) - The HTTP trace layer now uses `RelayResponseClassifier`. It treats marked responses as expected and keeps tower-http's 5xx-is-failure rule for everything else. Other 503s still log as failures. That includes an unavailable JWKS key source on a mapped host, `deny_protected`, a stale snapshot, and a failed database lookup. ## Why not return 4xx for unmapped hosts? NIP-FI.md maps "community/Host resolution unavailable" to 503, and `nip_fi_enforce_unmapped_host_is_503_before_verification` pins that ruling. Changing the status would be a spec change. This PR only changes how the relay logs these denials. ## Database community source With `BUZZ_NIP_FI_COMMUNITY_SOURCE=database`, a not-found host and a failed lookup (DB error, timeout, shed waiter) return the same denial class. The marker is set only when the host is confirmed unmapped: a missing Host, a non-bindable host, or a live negative-cache entry. Resolver outages stay unmarked and still log as failures. With the `env` source, every resolution failure means the host is unmapped, so it is always marked. ## Testing - `nip_fi_enforce_unmapped_host_is_503_before_verification` (guard, root upgrade, audio upgrade) and `enforce_unmapped_host_is_503_before_nip98_and_verifier` (HTTP admission) now also assert the marker. Mutation check: removing the guard marker turns the router test red. - New `nip_fi_mapped_host_dependency_503_is_not_marked_unresolved`: a mapped-host 503 from an unavailable key source is unmarked and still classified as a failure. - New `relay_response_classifier_skips_only_unresolved_host_denials`: covers marked 503, unmarked 503, 500, 200 and 401. - New `only_confirmed_unmapped_hosts_are_marked_as_expected_denials` (database source, all three sites): a negative-cached host and a non-bindable host are marked; a bound host whose lookup fails is not. Mutation check: marking every database failure turns it red. - `cargo clippy -p buzz-relay --all-targets -D warnings` is clean, and the pre-push `rust-tests` lane passed. - Local relay, before and after. Setup: enforce mode, `RUST_LOG=error`, one mapped host, JWKS unreachable on purpose. Both builds got the same requests: | Request | Status (both builds) | `on_failure` errors before | after | |---|---|---|---| | Unmapped host, 4 protected paths | 503 | 4 | 0 | | Unmapped host, WebSocket upgrade on `/` | 503 | 1 | 0 | | Mapped host, no assertion | 401 | 0 | 0 | | Mapped host, well-formed JWT, JWKS unreachable | 503 | 1 | 1 | | Unmapped host, `/assets/` path and `POST /api/invites/claim` | 404 | 0 | 0 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> Co-authored-by: opus <f038cc09b32a776e7d95b65c267e9d851777ee2b6cdbd46a48edba01b1d10ebd@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
152441ae0a |
feat(loadgen): model desktop clients and large-community shape (#9005)
## Summary
The load generator models every community as many small open channels,
and every human as a bare WebSocket client: four REQs, a little history,
then nothing until the run ends. Large deployments don't look like that.
They have thousands of people in a few communities, a handful of very
wide channels, and some people in hundreds of channels. Their desktop
clients also make most of their relay requests over HTTP and on timers.
This PR adds opt-in scenario knobs for both, so a run can reproduce that
shape and compare the desktop client before and after its recent
request-volume fixes.
Every new key defaults to the previous behavior. Existing scenarios
parse, assign members and run exactly as before.
### Shape
| Key | Default | Meaning |
|---|---|---|
| `shape.private_channels` | `false` | `seed` creates channels private
and adds every member with kind:9000 before posting history. Members are
the humans and agents assigned to each channel up to the largest ramp
step, plus the hot thread's root and authors. Events carry up to 100
members each and run `seed_concurrency` at a time. A rerun adds nobody
new, and any member the relay refuses fails the seed. |
| `shape.wide_channels` | `0` | The first K channels of each community
are wide channels. |
| `population.wide_member_fraction` | `0.0` | Share of humans that join
every wide channel on top of their own. |
| `population.wide_activity_share` | unset | Share of its typing,
messages and reactions a member of the wide channels sends there; the
rest goes to its regular channels. Unset keeps the uniform pick, which
sends half of a 3-wide/3-regular member's traffic to the wide channels
and makes fan-out, not reads, the bottleneck. |
| `population.tiers` | `[]` | `[{ fraction, channels }]`: that share of
humans joins `channels` channels instead of `channels_per_member`. |
Assignment is deterministic by global index, so `seed` and every
`--shard` agree without coordination. With no wide channels and no tiers
it is the old assignment, and a test pins that.
### Desktop clients (`[desktop]`)
| Key | Default | Meaning |
|---|---|---|
| `fraction` | `0.0` | Share of humans that run the desktop model
instead of the basic human. |
| `mode` | `"current"` | `legacy` is the client before its volume fixes;
`current` is after them. |
| `windows`, `threads` | `3`, `1` | Open channel windows and threads,
which are refreshed on connect. |
| `tick_secs` | `60` | Recovery tick period. |
| `backstop_secs` | `900` | `current` only: each tick item (subscription
replacement, roster, unread repair) waits this long since it last ran. |
| `renewal_secs` | `0` | Identity renewal period. `0` disables renewal,
and the first renewal is jittered. |
A desktop human keeps the basic human's presence, typing, messages and
reactions. What it adds:
- **Cold start.** It reads NIP-11 and opens its profile and membership
subscriptions. Its channel subscriptions are batched per mode:
- up to 10 channels: one replaying batch;
- `legacy` with more than 10: one per-channel batch per 10 channels;
- `current` with more than 10: a live-only subscription per 2048
channels, plus replays of 10 that are closed at EOSE.
It also makes the app's HTTP reads: roster, channel names, sidebar,
profiles, statuses, emoji, read state, unread repair, reminders, window
catch-up plus notices for each window, and root plus thread page for
each thread. A window catch-up follows the cursor in the relay's window
bounds (kind 39006) for up to five older pages, until the bounds say
history is exhausted.
- **Route establishment.** `legacy` makes every read as it connects.
`current` reads NIP-11, names and sidebar up front, and the rest as each
subscription reaches EOSE, by its replay outcome. A replay of 500
events, a missed deadline, or the live-only batch over 10 channels is a
cut, and so is a route the previous connection never caught up. Only an
EOSE catches a route up: one recovered after a missed deadline cannot be
resumed or carried by the next connection.
- A continuous profile route skips its read (unless one is owed).
- A continuous membership route only repairs unreads, without the
read-state snapshot, unless a snapshot read failed since; then it runs
the full repair.
- A cut global reads in full, and channel routes always recover windows
and threads.
- **Ticks and renewals.** These follow each mode:
- `legacy`: every tick replaces subscriptions and rereads roster and
unreads, and a renewal is a full reconnect;
- `current`: tick items wait for the backstop, measured from the last
replacement, the last successful roster read, and the last full unread
repair. A failed roster read, unread repair or read-state snapshot is
owed and retries on the next tick; snapshot debt is tracked apart from
repair debt, so a repair without the snapshot never settles it. A
renewal carries its routes: nothing is read while every replay keeps
continuity, and once one is cut every route establishes as on a
reconnect. With more than 10 channels the live-only batch is always cut,
so the channels recover and unreads are repaired, but globals are not
reread.
- **Reporting.** Each request type is its own op (`desktop_roster`,
`desktop_unread`, `desktop_replay_eose`, …).
- **Bounds.** At most 4 REQs set up at once, with a 10 s EOSE timeout. A
tick replacement that misses it is closed, the batch it would have
replaced stays, and the next tick retries. At most 6 HTTP reads in
flight, with a 30 s timeout. A read that comes due while the same read
is running or queued is not added again, and each read leaves the
running set as soon as it settles, so a failed read can retry while a
slow one is still under way. Connects and failed sessions back off. A
stop abandons pending EOSE waits without counting them as failures.
- **Not modeled.** The app also establishes channel routes when a tick
replacement reaches EOSE; the generator does not.
`perf/LOADGEN.md` documents the knobs and the model.
`perf/scenarios/local-desktop.toml` is a short local example with every
knob on.
## What got simpler
- `human()`'s schedule loop (presence, typing, messages, reactions,
session end) moved into one `Activity`. The basic and desktop humans
share it, so neither client duplicates the schedule. The basic human's
random draw order is unchanged.
- Membership stays in one place. `member_channels` remains the single
assignment function, and `channel_members` derives from it rather than
from a second formula, so seeding and the clients can't drift apart.
The desktop model is a new concept: about 1,420 lines of model in its
own module, plus about 1,300 lines of tests. Everything else goes
through existing seams: `Stats`/`Op`, `connect`, `Backoff`, `observe`,
`roots` and `Net`.
Identified debt, not fixed here: CI's `test-unit` runs `buzz-test-client
--lib` with the filter `loadgen::`, so the five crate-root `tests::` in
`lib.rs` (relay message parsing, auth and text event building,
`collect_until_eose`) never run in CI. That gap predates this PR, which
doesn't touch `lib.rs`.
## Testing
- `cargo fmt --check`, `cargo clippy -p buzz-test-client --all-targets
-- -D warnings`, and `cargo test -p buzz-test-client` (plain parallel):
84 lib tests pass.
- Wide activity: a seeded `Rng` over 100k picks gives each share within
1%, and an unset share makes the identical draws to before. A run
against a loopback relay checks that a human's typing lands only in its
regular channels at share 0, and only in its wide channels at share 1.
- Scenario: validation of every new key; deterministic wide-channel and
tier assignment that honors the fractions; `channel_members` inverts the
assignment up to the peak; the default assignment is unchanged; the
shipped scenarios parse.
- Seed: with private channels, a fake relay sees the following, in
order:
- every 9007 before any 9000, and every 9000 before any post;
- owner signatures;
- ≤100 members per event;
- exactly the expected members per channel;
- posts authored by channel members.
A refused member fails the seed, and open channels add nobody.
- Desktop: request planning, window bounds validation, route
establishment, and tick and owed-read rules are unit-tested. End-to-end
tests drive clients against loopback WS and HTTP fakes. The WS fake
replays a planned number of events per subscription and can delay an
EOSE or drop a connection. The HTTP fake answers window pages with
relay-signed bounds, an older aux event and equal-timestamp rows,
refuses unknown cursors, and can fail a read once. They check:
- exact cold-start counts per op for `legacy` and `current` at 5 and 25
channels;
- a window catch-up sends the head, then exactly the bounds' cursor
(`until` + `before_id` of the last row, not the older aux event), and
stops when the bounds say history is exhausted even though that page
came back full;
- after a `current` renewal, 499 replayed events everywhere read nothing
over HTTP; 500 on the channel batch adds channel recovery plus a covered
unread repair; 500 on membership adds roster, read state, unread and
reminders plus channel recovery; 500 on profiles adds profiles, statuses
and emoji plus channel recovery and a covered repair;
- after a dropped connection, complete global replays read only NIP-11,
channel recovery and a covered repair; truncated ones reread both
globals in full;
- with renewals shorter than the backstop, read-state snapshots still
recur at the backstop;
- a tick replacement past its EOSE deadline is closed, the old batch
stays, the next tick replaces it, and one batch wire stays open;
- a failed roster and unread read retry on the next tick, and successful
reads do not run again;
- a failed roster read retries on the next tick while an unrelated slow
read is still running;
- after a failed snapshot, a continuous membership establishment before
any tick still reads the snapshot;
- a route whose EOSE missed its deadline is recovered, and after a
renewal with a complete replay it is recovered in full again rather than
carried;
- `legacy` ticks replace batches (new REQ before the CLOSE of the old
one) and reread;
- `current` ticks do nothing until the backstop, then replace live-only;
- `legacy` renewals are full reconnects;
- `current` renewals carry small sessions without rereading globals or
channels, and recover channels in large ones.
Each of the nine review-fix tests was checked against a mutant that
restores the old behavior, and every mutant fails its test.
- Smoke: a local relay with Postgres and Redis. I ran `seed` twice, then
`run` of `local-desktop.toml` (60 s) once per mode.
- Both seeds were accepted (780 of 780 posts each), including private
channels and memberships for 340 people. Neither run aborted, no client
was cancelled, and no op recorded an error.
- Each run used 18 desktop clients:
| op | `current` | `legacy` |
|---|---|---|
| `desktop_live_eose` | 266 | 483 |
| `desktop_replay_eose` | 24 | 0 |
| `desktop_roster` | 72 | 289 |
| `desktop_read_state` / `desktop_unread` | 72 / 99 | 289 / 289 |
| `desktop_profiles` | 18 | 73 |
| `desktop_nip11` | 18 | 73 |
| `desktop_window` | 372 | 596 |
Window reads are below the first smoke (406 and 1086) because catch-ups
stop where the relay's bounds say history is exhausted, rather than
paging while pages came back full. They are above the previous revision
(179 and 470) because each read now leaves the running set as soon as it
settles, so a channel recovery is no longer dropped as a duplicate of a
read that already finished inside a still-running batch.
- The first smoke caught two relay contract mismatches that the fakes
had missed:
- top-level window pages need `before_id` with `until`;
- the read-state snapshot answers with an object.
Both are fixed, and the fakes now enforce both.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Luke Tornquist
|
88697f1c59 |
fix(relay): apply kind 5 / 9005 deletions atomically (#8944)
## Summary Kind 5 and 9005 deletions now store the event and soft-delete their targets in one transaction. Before this PR the deletion committed first. The target soft-delete ran afterwards as a best-effort side effect, and a failure there was only logged. Resending the deletion then returned `duplicate:` without retrying. A crash between the two commits had the same result. Either way the relay accepted a deletion while its target stayed live. The bridge `include_aux` closure still compensates for those rows. Its second hop returns deletions of the reactions and edits it found, so clients can cancel them. This PR keeps that hop, because rows already torn this way can't be repaired by a resend: - Ingest rejects events more than 900 seconds from server time. - Clients don't resend deletions that were accepted. - During a rolling deploy, old pods keep writing torn rows until they are replaced. Dropping the hop waits for a follow-up, once existing torn rows have had a verified repair and no old writers remain. ### Changes - **Atomic deletion.** Kind 9005, and kind 5 with `e` targets, go through `Db::insert_deletion_event`, driven by `side_effects::persist_deletion`. The write stores the event and its mentions, soft-deletes the targets and updates their thread counters. It follows the existing `persisted_atomically` path used for workflow deletion and kinds 9000/9002. - A failure stores nothing and rejects the event, so the client can resend. - A resend of a stored deletion inside the ingest window soft-deletes any target still live and is dispatched as inserted. It doesn't repair older rows. - Live 39005, reaction-table cleanup and the 9005 system message still run after the commit and stay best-effort. - Kind 5 deletions of `a` coordinates stay a side effect. - **Closure unchanged.** The channel window and legacy thread-reply paths keep both hops, and the docs (`docs/bridge-channel-window.md`, `docs/nips/NIP-CW.md`) are unchanged. ### What got simpler Deletion no longer has a window where the event is accepted but not applied. `handle_standard_deletion_event` is now only the `a`-coordinate path. Target planning and soft-delete for `e` targets live in one place, `persist_deletion`. ### Follow-ups (not in this PR) - Repair the existing torn rows: soft-delete the live targets of stored kind 5 / 9005 `e` deletions and fix their thread counters. Then drop the closure's deletion-of-aux hop. - Hop 1 still probes every partition once per row id. A partition bound or the indexed `reactions` table would cut that. ### Related issue N/A, found while investigating relay database CPU. ### Testing - `torn_deletions_still_cancel_live_targets_in_both_windows` (bridge Postgres) stores kind 5 and 9005 deletions without soft-deleting their reaction and edit targets. It queries the channel window and the legacy thread path with `include_aux`, without a resend, and asserts each live target comes back with its deletion. - `insert_deletion_event_target_failure_stores_nothing_and_resend_applies` (buzz-db Postgres) uses a trigger to make the target's soft-delete fail. It asserts nothing is stored and the target is unchanged, then that a resend applies both. - `insert_deletion_event_resend_repairs_a_live_target` (buzz-db Postgres): a resend repairs a deletion stored without its target's soft-delete, and a further resend changes nothing. - `failed_target_soft_delete_stores_nothing` and `resent_deletion_repairs_a_live_target` (relay Postgres) cover the same two cases through ingest, for kinds 5 and 9005. - `test_channel_window_rows_overlays_and_exact_multiple_exhaustion` (`e2e_nostr_interop`, Relay E2E lane) deletes its reaction and asserts the reaction leaves the closure. - Blox: fmt, clippy (`buzz-db`, `buzz-relay`, `buzz-test-client`), and the `buzz-db` and `buzz-relay` Postgres lanes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
b8b425e8b1 |
fix(db): drop a superseded head's mention rows with it (#8948)
## Summary
Replacing a replaceable or addressable event soft-deletes the previous
head, but its `event_mentions` rows stayed behind. Only the hard-delete
kinds (read-state and mesh status) removed theirs, in a separate
statement. No mention-index read (`#p` filters and feeds) returns a
soft-deleted event, so those rows are pure cost: every `#p` read walks
them and throws them away.
A kind:39002 roster republish leaves one dead row per channel member, so
a large channel piles them up with every join. Relay-emitted replaceable
lists with p-tags do the same.
The statement that retires the old head now deletes that head's mention
rows as well, through a data-modifying CTE over `RETURNING id`, on all
three supersede paths:
- the relay's kind:39002 roster writer
(`LockedMemberSnapshot::replace_member_event`)
- `replace_addressable_event` (kind 0/3/10000–19999, including the
relay-signed kind:13535 list)
- the NIP-33 parameterized path
Because the CTE covers every row the retire statement touches, a
coordinate that somehow had more than one live row also gets cleaned up.
The shared `superseded_with_mentions_sql!` macro sits at the crate root
so all three mention-indexing replacement paths use the same statement.
The kind:13534 membership-list and push-lease writers also retire heads,
with a bare `UPDATE`, but neither indexes p-tags, so they leave no
mention rows.
## What got simpler
The hard-delete special case (`if hard_delete_superseded { DELETE FROM
event_mentions … }`) is gone. Both retire statements go through one
macro, so a head and its mention rows leave together on all three paths.
## Not in this PR
- Mention rows already left behind by earlier supersedes.
`docs/database.md` rules out data backfills in migrations, so these get
cleaned up out of band, in batches.
- Other soft-delete paths (NIP-09 deletion, moderation, channel deletion
retiring its 39000–39002 events) also keep mention rows. They are
low-volume, and are noted here as identified debt.
## Tests
`superseded_heads_drop_their_mention_rows` (pg) has one table row per
supersede path: the member-snapshot roster writer (lock, replace,
release), the addressable path, the NIP-33 path, and the two NIP-33
kinds that are hard-deleted (NIP-RS read state and mesh status). In each
case the old head keeps zero mention rows and the new head is indexed.
Soft-delete rows check that the old head stays soft-deleted, and
hard-delete rows check that it is gone. Reverting the macro, any single
call site, or the hard-delete statement alone fails this test.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Luke Tornquist
|
298a555e34 |
perf(db): bind the channel scope as one array (#8946)
## Summary Event reads scoped to a set of channels pushed one bind parameter per channel: `channel_id IN ($5, $6, …)`. This applies to a viewer's access scope, which the relay adds to every REQ and `/query` without an `#h`, and to explicit multi-value `#h` filters. As a result, the SQL text depended on the viewer's channel count, and every count was a separate prepared statement. That caused two problems in production: - **Re-planning on every call.** A connection rarely reused a cached plan for these reads, so Postgres planned each one again. With no `since`, the plan covers every `events` partition. For `#p` mention reads (the `event_mentions` join), planning took far longer than executing the query. - **Hidden cost.** Monitoring that groups by statement split one query into one row per channel count, so its total cost never showed up in a single place. This PR binds the set as `channel_id = ANY($n)`, so one statement serves every channel count. On a production replica, the same query prepared with `ANY($5)` moved to Postgres's cached generic plan by its sixth run. Planning time dropped to near zero, and that plan was no slower than the custom one. ## What changed - `push_channel_scope` in `crates/buzz-db/src/store/event.rs` builds the channel-scope predicate as one array bind. - `build_query_events_sql` and `count_events_sql` each had their own copy of this block. Both now call the helper. - Semantics are unchanged: - Access scope still keeps channel-less global events. - Explicit `#h` still matches only the requested channels. - An empty set still means "no channels". ## What got simpler Two duplicated 20-line blocks became one helper. The statement count for channel-scoped reads no longer grows with channel count. ## Generic-plan exposure These statements were already persistent prepared statements, so a cached generic plan was always possible whenever the connection's statement cache held one. This change makes that reuse consistent; it doesn't introduce generic plans. Tag-value reads (`has_tag_values`) still run unnamed, as before. Plain (non-`#p`) access-scoped reads stay on custom plans. A production replica, with a viewer in hundreds of channels, ran the plain page read repeatedly under the default `plan_cache_mode`, and every run used the custom ordered index scan. Postgres estimated the generic plan as far more expensive, and when forced, it was orders of magnitude slower. Postgres switches only when the generic plan is no costlier than the average custom plan, so it never chose this one. The generic plan's cost comes from the `LIMIT` and `kind` values it can't see, not from the array length, and the per-arity `IN (…)` statements had the same blind spot. ## Tests - `channel_scope_sql_does_not_depend_on_channel_count` (unit): for the `#p` join and plain reads, for access scope and explicit `#h`, in both the page SELECT and the COUNT, the SQL for 1 channel equals the SQL for 40 channels. It also checks that the SQL uses `ANY(` and that global rows are retained only for access scope. - `channel_scoped_mention_reads_share_one_prepared_statement` (Postgres): on one connection, access-scoped `#p` reads with 1, 2 and 3 channels leave exactly one prepared `event_mentions` statement. With one mention per channel, one global mention, and one in an unscoped channel, every scope of 0–3 channels, with and without globals, returns and counts exactly its own rows. - The existing scope semantics tests still pass: - `access_scope_is_applied_before_historical_page_limit` - the explicit multi-channel `#h` page and count test - Mutation: reverting the helper to one bind per channel fails both new tests. ## Identified debt (not in this PR) - `get_last_message_at_bulk` still builds an `IN (…)` list per channel. It's a separate, lower-volume statement. - Superseded versions of replaceable roster events (kind 39002) keep their `event_mentions` rows. On large channels, a viewer's roster mention rows grow with every membership change. That's a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Will Pfleger
|
f6198f6997 |
feat(relay): record NIP-FI enforce denials at parity with shadow (#8974)
## What this does
In enforce mode, NIP-FI denials used to be returned without a reason
counter or a log line. This PR records enforce denials with the same
`WouldDeny` reasons, label sets and log fields as shadow. **Admission
behavior does not change**: statuses, close codes and frame order are
all the same.
- **`buzz_nip_fi_enforce_total{route,stage,outcome,community}`**: the
same shape as `buzz_nip_fi_shadow_total`, including an `admit`/`admit`
stage, so `1 - admit/total` is the enforce denial rate. Each socket or
HTTP request gets exactly one verdict.
-
**`buzz_nip_fi_enforce_session_end_total{route,reason=expired|revoked,community}`**:
once per admitted session.
- **`buzz_nip_fi_enforce_strict_proof_total{route,outcome,community}`**:
the outcome of the proof check that actually decided the request.
Nothing is verified twice.
- **Log lines** `nip-fi enforce deny` and `nip-fi enforce session end`
(info, `mode="enforce"`, from `buzz_relay::nip_fi_shadow`). Shadow's
`nip-fi shadow would-deny` gains `mode="shadow"`. Neither mode logs
admits.
### How each verdict is recorded once
- **Sockets (root, audio).** `EnforceSession` moves Pending →
Admitted(pubkey) → Ended under one std mutex, and records inside the
lock with no awaits.
- Root admits at its AUTH commit; audio admits at its join commit.
- A refusal for a non-NIP-FI reason retires the session without a
record. Audio retires it before rollback.
- A deadline the effect gate sees before the expiry worker is claimed at
that seam.
- **Ends.** Revocation and expiry are recorded only by the terminal
transition that wins (`publish_authorization_denied`), and only while
the socket isn't already cancelled. A community close or cancellation
that wins first records nothing.
- **HTTP.** The router guard scopes each enforced request with one
verdict. An agent token's verdict is deferred to the end of the request:
- the first owner-decided refusal is the verdict, otherwise the request
was admitted;
- membership rechecks, batches, Git success and early returns each count
once.
- **Owner provenance** comes from the decision itself
(`BanVerdict{outcome,source}` from one `restriction_state` read), with
no rereads.
- Reasons: `owner_banned`, `owner_ban_unavailable`, `not_stored_owner`,
`owner_link_unavailable`.
- When the agent's own ban read fails and there's a token owner, the
result is `owner/unavailable`, as shadow records it.
- A ban or failed restriction read that ingest catches on `POST /events`
after membership passed replaces the request's admit with `owner/denied`
when the owner's row decided the ban, or `owner/unavailable` on a failed
read. An agent's own ban is not recorded as its owner's.
## Shadow-to-enforce map
| Shadow signal | Enforce counterpart |
|---|---|
| `buzz_nip_fi_shadow_total` denials: upgrade, guard, HTTP admission,
AUTH pairing, deny set, stale binding | `buzz_nip_fi_enforce_total` with
the same stage/outcome, plus a `nip-fi enforce deny` line |
| `buzz_nip_fi_shadow_total` owner stage: ws, audio, http | the same,
from the original decision: root/audio ban seam and owner link; HTTP
`enforce_membership`, smart Git, Git settings, buzz/v1 ban reread |
| `buzz_nip_fi_shadow_total` `deadline/session_deadline` (pending) |
`EnforceSession::expire`, from the worker's winning terminal or the
effect gate that saw it first |
| `buzz_nip_fi_shadow_total` admit |
`buzz_nip_fi_enforce_total{stage="admit",outcome="admit"}`, one per
socket or request |
| `buzz_nip_fi_shadow_session_end_total{reason=expired\|revoked}` |
`buzz_nip_fi_enforce_session_end_total`, same labels |
| `buzz_nip_fi_shadow_strict_proof_total` |
`buzz_nip_fi_enforce_strict_proof_total`, from the deciding check (a
rejection is the route's 403, or 401 when the proof is missing) |
| `buzz_nip_fi_shadow_disconnect_total{admin,accepted}` |
`buzz_nip_fi_disconnect_total` (`api/nip_fi.rs:160`) |
| `buzz_nip_fi_shadow_disconnect_total{admin,propagation_failure}` |
`buzz_nip_fi_disconnect_propagation_failures_total`
(`api/nip_fi.rs:180–185`) |
| `buzz_nip_fi_shadow_disconnect_total{admin,capacity}` |
`buzz_nip_fi_disconnect_capacity_rejections_total`
(`api/nip_fi.rs:196–201`) |
| `buzz_nip_fi_shadow_disconnect_total{cross_pod,capacity}` |
`buzz_nip_fi_cross_pod_capacity_exceeded_total`
(`api/nip_fi.rs:433–434`) |
| `buzz_nip_fi_shadow_disconnect_total{cross_pod,poison}` |
`buzz_nip_fi_cross_pod_shard_poison_total` (`api/nip_fi.rs:442–443`) |
| `buzz_nip_fi_shadow_disconnect_total{cross_pod,lag}` |
`buzz_nip_fi_disconnect_lag_total` (`main.rs:1596–1602`) |
The enforce series was renamed from `buzz_nip_fi_enforce_denied_total`
to `buzz_nip_fi_enforce_total` when it gained the `admit` stage.
## Tests
Infra-free, in the `test-unit` lane:
- `state::enforce_end_is_recorded_only_by_the_winning_terminal`: root
scan, audio scan and deadline, each against a community close or
cancellation first and FI first.
- `nip_fi_http::enforce_strict_proof_records_the_deciding_check_once`:
pass and reject, and no record when cardinality refuses.
-
`nip_fi_shadow_session::audio_owner_link_refusal_shares_roots_seam_and_reasons`:
calls the shared owner-link helper with route `audio`, not a huddle
socket.
- The existing shadow, upgrade, router and HTTP enforce tests.
Postgres lane:
-
`handlers::auth::enforce_root_auth_records_owner_refusals_from_one_decision`:
clean admit then `expired` through the real expiry worker; banned token
owner, banned stored owner, other stored owner and a failed ban read;
authorization reads counted from the decision only.
-
`handlers::auth::enforce_root_deadline_seen_by_the_gate_first_is_recorded_once`:
both orderings.
- `audio::handler::enforce_join_commit_is_the_audio_admission`: drives
the join-commit seam directly, not a huddle socket or the expiry worker:
committed then expired; archived with an expiry held inside the
rollback; gate-first deadline.
-
`api::bridge::token_owner_reaches_membership_on_every_nip_fi_http_route`:
one verdict per request on every NIP-FI HTTP route, including sidebar,
read-state, Blossom and Git, for an owner ban, a stored-owner conflict
and a clear owner.
-
`api::bridge::token_owner_ban_reread_is_the_fixed_nip_fi_denial_on_http`:
own ban, owner ban and unavailable at the reread, on sidebar and Git
settings.
-
`api::bridge::token_owner_publish_ban_is_the_fixed_nip_fi_denial_on_http`:
a late ban or failed read on `POST /events` at the write, typing,
moderation and relay-admin checks records exactly one owner denial and
no admit when only the owner is banned, and no owner denial when the
agent alone or both are banned.
-
`audio::handler::tests::already_expired_audio_bootstrap_records_the_deadline`:
an already-expired bootstrap records `deadline/session_deadline`, not a
deny-set denial.
Each new test names the mutation that turns it red. These were run red:
- deleting root's `session.admit(pubkey)`
- deleting audio's commit `session.admit(pubkey)`
- deleting `effect_refused`
- deleting enforce from `retire_shadow`
- deleting the commit seam's `expire`
- deleting the cancellation guard
- deleting the own-read-failure provenance
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <wpfleger@block.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
|
||
|
Daniel Rouleau
|
a9aa1f73b0 |
Enable incremental artifact reads on every relay (#8975)
## Why Compatible clients need incremental artifact reads for local collections on every Buzz relay. ## What Expose incremental artifact reads and capability discovery on every Buzz relay. ## How Serve incremental queries through the existing authentication and authorization checks. Keep the existing work budgets and the separate WebSocket sync capability. The HTTP test verifies capability discovery and a successful incremental query. ## Risk The relay starts no additional background work. Clients that use incremental reads consume database capacity. The net database load is not yet measured. Serving writers must maintain the modification timestamps. ## Testing Read-only SQL checks on November 3, 2026 verify the prerequisites below. Each database records successful migrations 76–78. All five required columns have the expected types, defaults, and nullability. All five indexes have the expected definitions and are valid, ready, and live. | Deployment | Schema and indexes | Running timestamp writers | | --- | --- | --- | | Production `bb-block` | Pass | `f6f6d9a` includes PR 8523 | | Production `bb-public` | Pass | `f3a7bab` includes PR 8523 | | Staging `bb-block` | Pass | `b4509de` includes PR 8523 | The checked columns are `artifact_heads.modified_at`, `artifact_revisions.modified_at`, `artifact_revisions.source_channel_id`, `channel_members.sync_modified_at`, and `communities.artifact_policy_modified_at`. The checked indexes are `idx_artifact_revisions_artifact`, `artifact_heads_modified`, `artifact_heads_channel`, `channel_members_sync`, and `artifact_revisions_moves`. At commit `62b26ad`, all 21 NIP-11 tests and the HTTP capability test pass on Linux with PostgreSQL 17 and Redis. The HTTP test checks capability discovery and a successful incremental query. All required push hooks also pass. No manual testing. Human confirmation remains pending. ## Bigger picture [Client PR 1507](https://github.com/block/buzz-app/pull/1507) uses the advertised capability for local artifact sync. Existing deployments receive this behavior when they deploy the relay version that includes this PR. Generated with Codex --------- Signed-off-by: Daniel Rouleau <drouleau@squareup.com> |
||
|
Will Pfleger
|
ce47aefdc3 |
fix(relay): send membership notices on channel delete, moderation kick and moderation DM (#8977)
Related: [buzz-app#2215](https://github.com/block/buzz-app/pull/2215) ## Problem A connected client learns about membership changes from kind:44100/44101 notices and then re-reads its channel list. Three relay paths changed a member's channel list without sending a notice, so clients only caught up on their periodic full read: - **Channel delete (9008):** no member was told. - **Moderation kick:** the kicked member wasn't told, and the channel's 39000/39002 discovery wasn't refreshed, so it still listed them. - **Moderation DM:** a new, existing or hidden moderation DM didn't surface live. ## Change - **Channel delete:** `soft_delete_channel_capturing_members` (buzz-db) reads the channel type and active members under deletion's existing membership lock, in the same transaction. A join either commits first and is included, or waits and is then refused. After the commit, discovery cleanup and cache invalidation, the relay sends a type-tagged 44101 to each captured member in a spawned task, eight at a time, outside any lock. The 9008 OK doesn't wait for them, and the task is started before the `channel_deleted` system message is emitted, so a system-message failure can't drop them. An already-deleted channel sends nothing. `soft_delete_channel` keeps its signature and delegates. - **Moderation kick:** `apply_kick_live_side_effects` refreshes 39000/39002 after the removal fence commits, then sends 44101 to the kicked member. If the refresh fails, it returns the error before the notice, so the moderation driver's existing mutation-marker recovery retries. If the channel was deleted since the kick, there's no roster to refresh: it removes the channel's discovery events and treats that as converged, so the action still finalizes. The removal/re-add fence and the re-added early return are unchanged. - **Moderation DM:** `send_moderation_notice` drops both participants' cached channel sets on every pod (`invalidate_membership`), as opening a DM does in `command_executor::handle_dm_open`, so the read the notice prompts includes the DM. After clearing `hidden_at`, it republishes the recipient's NIP-DV kind:30622 visibility snapshot (clients filter the sidebar on it), then emits discovery, then sends 44100. It sends the notice on every delivery, tagged `moderation_source` with the delivery's source id, so two deliveries in the same second each get their own 44100. The content JSON is unchanged. A failed snapshot or discovery returns an error, so the outbox retries. - **Ordering guarantee:** each notice is sent after its change is committed and readable on the relay instance that sends it. Ordering across relay instances, and other instances' cached channel sets, are best-effort, the same as for existing membership notices. - A failed notice delivery is logged and never undoes the change, like the existing 9001 path. There's no new retry system. ## Tests - `membership_notice_postgres_tests.rs`: one journey per path. A `#[cfg(test)]` barrier at the point a notice is signed parks the producer, keyed by recipient. While it's parked, the test reads the channel list (and, for the DM, the 30622 snapshot) as a client would on receipt. The barrier can also pin a notice's `created_at`. - The DM journey pins both deliveries to the same second, hides the DM through the real kind:41012 path between them, and checks what a client sees: a live 44100 for each delivery, with distinct ids, then a channel-list read that lists the DM and a visibility read that no longer hides it. - `moderation_dm_notice_follows_a_cached_channel_list` warms the recipient's channel set with a read before the DM exists, then checks the read on receipt of the notice lists the DM. - `soft_delete_captures_members_at_its_serialization_point` (buzz-db): a join and then a delete queue behind a held membership lock. The delete must return the joiner, and a later join is refused. - `crash_recovery_finalizes_kick_after_channel_deleted` (admin): commits a kick, deletes the channel, then runs the recovery worker. The action finalizes, and the member's `removed_at` is unchanged, so the removal doesn't run again. Each ordering test fails when its prerequisite is moved or removed: the kick notice ahead of the roster refresh, the delete notices ahead of discovery cleanup, the DM without the snapshot republish or the cache drop, the DM notice without its source tag (the second notice coalesces), and the member capture outside the lock. The recovery test fails if a deleted channel's `ChannelNotFound` propagates. No spec files change. This reads a channel delete as removing every member under the existing `NOSTR.md` notice promise. --------- Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
Salman Mohammed
|
d3a4909cca |
perf(search): rank only the newest prefix matches and plan with the real tsquery (#8976)
## Summary Message search is one of the slowest relay queries. Short typeahead prefixes make it worst: `a:*` matches most of a community. **Why it was slow:** the tsquery was built inside a `CROSS JOIN LATERAL` subquery, so the planner saw an InitPlan and couldn't read the `search_tsv` statistics. It fell back to the default 0.5% selectivity for `@@`: for `a:*` on 1M rows it estimated 4,497 rows when 517,065 matched. Every search therefore took every GIN match, ran `ts_rank_cd` on each one, and top-N sorted them before `LIMIT 20`. This PR: 1. **Binds the real tsquery.** `push_tsquery` is unchanged. A separate `SELECT (…)::text` round trip resolves the tsquery, and the search binds it as `$q::tsquery`. An empty tsquery returns no hits without running the search. 2. **Plans each search for its own tsquery.** The search runs with `.persistent(false)`. A cached prepared statement can switch to a generic plan after five runs, and a generic plan can't see the tsquery. Forcing a generic plan lost the gain in testing. 3. **Ranks only the newest 1,000 prefix matches.** Prefix-mode relevance search ranks the newest `PREFIX_RANK_WINDOW` (1,000) matches. The inner `ORDER BY created_at DESC, id DESC LIMIT 1000` matches each partition's primary key read backwards, so for common prefixes Postgres streams the newest rows and stops at 1,000. For rarer terms it keeps the GIN bitmap plan, now with an accurate estimate. Full-text search and profile (kind:0) typeahead still rank every match. ### Behavior change: every human message search (product decision needed) Prefix mode isn't limited to keystroke typeahead. buzz-app (`useSearchMessages.ts`) and legacy desktop (`desktop/src-tauri/src/commands/messages.rs:184`) send `search_mode: "prefix"` for **all** message search. Full-text search runs only on the WS REQ path, which agents and the CLI use. So for people: - Results are the best-ranked of the newest 1,000 matches in the search's scope, not of all matches. If more than 1,000 newer messages also match, an older, more relevant message drops out of ⌘K results. The scope includes `#channel`, `from:` and date bounds, so a narrower search has the same limit within it. People have no full-text fallback. - The bridge honors `page`, and buzz-app loads later pages at 20 per page. Every page ranks the same window, so paging stays consistent, but results end after 1,000 matches (page 50). Full-text search (agents/CLI) and profile (kind:0) typeahead still rank every match. ### Perf risk: matches the backward scan has to walk far to find The planner picks the backward scan when it expects 1,000 matches near the top of the community, and it estimates that by treating each filter as independent. Two known shapes break that assumption. **Terms common in old messages but absent from recent ones.** The planner assumes matches are spread evenly over time. For a term that was common long ago but doesn't appear recently (an old project name, a former teammate), it can pick the backward primary-key scan and walk every newer row before reaching the first match. Reviewers measured this on 1.2M synthetic rows: about 2.05–2.1 s, against 0.7–1.8 s for the old query shape and 1.2 s with the GIN plan forced. The 20 s statement timeout and the #8954 cancel still bound it. The worst case is close to the old `a:*` worst case, but it now scales with community size, not match count. The planned EXPLAIN on production-shaped data will include such a term. **Terms common in channels the searcher can't read.** Human search always adds `channel_id = ANY(<readable channels>) OR channel_id IS NULL` (`build_search_channel_scope_filter`). A term that is common in private channels or other people's DMs can get a high match estimate, take the backward scan, and walk the community's rows to find a few readable matches, where the old shape used the GIN bitmap. This hasn't been reproduced yet; the production-shaped EXPLAIN will include such a term next to the old-only one. The same timeout and cancel bound it. ### Cost - One extra round trip per search to resolve the tsquery (about 0.03 ms of execution). It runs on the same pool checkout as the search (`cancel::fetch_all` takes the caller's connection), so a busy pool queues each search once. An IMMUTABLE SQL function could remove the round trip, but that needs a migration, so it's not done here. ### Related issue - Follows #8954 (cancel abandoned searches, bound the search pool) and #8964 (closed in favor of #8954). - Builds on #8947 (3-character server-side prefix minimum), merged from main. The resolve step passes its minimum. When the trailing prefix is too short, the resolved tsquery is NULL, and the search returns no hits without running the second query. - Client-side pairing: block/buzz-app#2210 (3-character last-word minimum for ⌘K message search). ## Testing **EXPLAIN ANALYZE on a synthetic 1M-row `events` table** (monthly partitions, GIN on `search_tsv`, kinds 9/40002/40008, custom plans, warm cache): | Prefix | Matches | Before | After | Plan after | |---|---|---|---|---| | `a` | 517k | 322 ms | **0.7 ms** | backward primary-key scan per partition, stops at 1,000 | | `th` | 137k | 127 ms | **1.2 ms** | backward primary-key scan, stops at 1,000 | | `a` (community with 10% of rows) | — | ~100 ms | **1–2 ms** | backward primary-key scan | | `deploy` | 8.2k | 15 ms | 5–10 ms | GIN bitmap, accurate estimate | | `d278` | 500 | 1.1 ms | 0.5 ms | GIN bitmap | | rare word | 0 | 0.07 ms | 0.03 ms | GIN bitmap | **Postgres integration tests:** - New test `prefix_relevance_ranks_only_the_newest_matches`. An old, highly relevant message drops out of prefix results behind 1,000 newer matches but still ranks first in full-text search. Prefix pages end at page 50. An old, best-ranked profile still tops profile typeahead behind 1,000 newer profile matches. Removing the window fails the test, and so does applying it to profiles. - `abandoned_search_cancels_its_statement`: the `pg_stat_activity` matcher now looks for `ts_rank_cd(`, because the old `search_query.query` text no longer exists. - Locally, each test ran against a fresh database because `cargo-nextest` isn't installed, so the hosted lane hasn't run yet. 26 of 27 pass. `artifact_search_uses_only_live_current_head` also fails on unmodified main in that local setup, so the failure isn't from this change. **Checks:** `cargo fmt --check` and `cargo clippy -p buzz-search --all-targets -D warnings` are clean. **Not yet done:** - hosted CI; - a manual check against a local relay with ⌘K typeahead; - EXPLAIN against production-shaped data, including a term common only in old messages and a term common only in channels the searcher can't read. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
d44031f418 |
perf(relay): cut writer-pool round trips past the 48k relay ceiling (#8846)
## Summary Five changes that take load off the writer pool and the relay's connection bookkeeping. On a single-relay load rig, the clean ceiling moves from 40k to about 48k WebSocket clients. In the A/B, connect p99 at 48k fell from 4.1 s (abort) to 1.65 s. 1. **Ban/mute restriction cache with a 1 s max age.** HTTP API requests, publishes and ephemeral writes read `(community, pubkey)` restriction state through one cache. An entry is served for 1 s from when its read began, matching the open-channel cache's bound. Socket admission, moderation-command actor checks and the `/buzz/v1` ban recheck still read fresh. An admitted socket outlives the 1 s window, so a banned client that reconnects inside it would otherwise keep a socket open indefinitely. A ban drops the target's entry, and each owned agent's entry, on every pod as part of the live-disconnect fan-out, best-effort. Timeouts, untimeouts and unbans drop it on the pod that handles them. 2. **Fewer round trips per event read.** Replica reads skip the per-read `statement_timeout` tighten, because the reader pool installs the deadline at connect. They keep the savepoint, so a failed or cancelled read leaves the session usable and releases its locks. That takes them from 4 round trips to 3. Writer reads send `BEGIN` and the tighten as one message, which also takes them from 4 to 3. 3. **Concurrent open-channel cache misses share one read.** The first miss in a community takes a per-community gate and reads; the others wait for it and use its result. A waiter whose leader failed reads once on its own and does not queue again. A waiter also stops waiting after the cache's 1 s max age and reads on its own, because the leader's set would be stale by then anyway. So a leader whose read hangs stalls only its own request. 4. **`has_admitted_connection` is one map lookup.** A `(community, pubkey) → admitted count` index is maintained under each socket's pubkey lock in `mark_admitted`, `set_authenticated_identity` and `deregister`. It replaces a scan of every connection on each disconnect. 5. **Host → community cache with a 1 s max age.** Only successful lookups are cached; unmapped hosts and errors go to the database every time. ## What got simpler - The ban handler's separate pod-local cache drop is gone; the disconnect every ban already runs covers it on every pod. - One write-restriction gate instead of two. `enforce_cached_write_restriction`, `enforce_relay_membership_ephemeral` and the `cached_ban` parameter chain through `enforce_membership` / `admit_member` / `deny_banned` are deleted. - The ephemeral path's separate 30 s ban cache is gone, so the relay has one staleness rule for restriction state instead of two. ## Notes - Bans reach the restriction cache on every pod. Every ban path (signed command, report action, relay admin) closes the target's and its agents' sockets clusterwide, and each pod drops those entries in the same step. A dropped Redis publish, or a restriction read already in flight when the drop lands (it can refill the old row), falls back to the 1 s bound. Timeouts on other pods also rely on the 1 s bound, and so do unbans and untimeouts, which fail closed. - On the load rig the restriction-cache hit rate was 28%, because HTTP agents poll every 5 s. A longer TTL with Redis invalidation would raise it, but that is a separate change. ## Load test Single relay pinned to 8 cores, native Postgres 16, a seeded fixture with 2,000 open channels, and a 6-shard generator. Steady ramp with 120 s holds; a step aborts at p99 > 3 s. Each step adds 10% HTTP agents. | clients | base connect p99 | this PR connect p99 | base relay cores | this PR relay cores | |---|---|---|---|---| | 32k | 34 ms | 31 ms | 1.71 | 1.66 | | 40k | 45 ms | 26 ms | 2.33 | 2.15 | | 48k | 4,107 ms (abort) | 1,652 ms | 4.7 | 3.0 | | 56k | n/a | 4,535 ms (abort) | n/a | 4.0 | Open-channel re-reads over the run fell from 72k to 1.6k. The tenant cache hit rate was 94%. 48k is this branch's knee. A later run of the same build aborted at 48k: connect p99 2.2 s, but 29% subscribe errors from the concurrent-handler cap. With a 10 s restriction max age, it passed 48k, at connect p99 1.56 s, 60% hit rate and 10% fewer relay cores, and both arms stopped at 56k. ## Testing - `cargo fmt --check`, `cargo clippy -p buzz-relay -p buzz-db --all-targets -D warnings` - `cargo test -p buzz-db --lib` and `-p buzz-relay --lib`, run in parallel. The only failures were 8 DB-seeding tests that need a migrated default database; the Postgres lane covers them. Commits after that run changed the replica savepoint, the `/buzz/v1` recheck revert and some test edits. - Postgres lane at head: `scripts/postgres-test-run.sh -p buzz-relay -p buzz-db --lib --bins --tests`. All 1,183 tests passed. - Mutants. Each one must turn its new test red: - waiters skipping the gate - a no-op index removal - the tenant cache ignoring age - the restriction cache ignoring age - a waiter waiting on the gate with no limit - a disconnect that keeps the cached restriction row - replica reads dropping the savepoint buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
d005c7bb9c |
fix(search): skip message prefix search below three characters (#8947)
## Summary
Typeahead surfaces (the buzz-app and desktop topbars) send a NIP-50
search with `search_mode: "prefix"` whenever typing pauses. A one- or
two-character trailing `:*` prefix matches a large share of every
indexed message. The GIN index on `search_tsv` can't stop early or
return rows in rank order, so Postgres fetches, rechecks and ranks every
match before it applies `LIMIT`. In a large community, each of these
queries costs seconds of database CPU.
With this change, prefix-mode `search()` returns no hits when the
trailing whitespace token yields a lexeme shorter than three characters
(`PREFIX_MIN_CHARS`). The check runs inside the prefix tsquery SQL, on
the output of the same `to_tsvector('simple', …)` call that decides
which lexemes get `:*`. Punctuation therefore can't pad the count:
`pr.`, `@pr`, `#pr`, `c++`, `don't` and `pre-p` are all held back, and
`project ?` still runs, because `?` yields no `:*` lexeme. When the rule
applies, the tsquery is NULL, so it matches nothing and the GIN index
lookup returns no rows. Completed words in front of the trailing token
don't matter: `"the p"` runs the same broad `p:*` lookup as `"p"`.
Kind:0-only queries (profile typeahead) are exempt because there are few
profiles, and the short-name exact-lexeme ordering relies on one- and
two-character queries. `search_profile_names` is unchanged. Full-text
mode is unchanged.
Only the HTTP bridge (`POST /query` with `search_mode: "prefix"`)
reaches prefix mode. WebSocket REQ search always uses full-text mode, so
this change doesn't affect it.
## What got simpler
The kind:0-only predicate is now computed once (`profile_only`) and
shared by the new minimum and the existing exact-lexeme ordering. The
minimum lives in the one SQL fragment that already tokenizes the query,
so there is no second tokenizer in Rust to keep in sync with Postgres.
No new concepts beyond one constant.
## Known limits
- This bounds the worst case only. A common three-character prefix can
still match enough rows to be expensive. Bounding that needs a product
decision, such as a recency window for typeahead or a different index.
- Two-character CJK words (for example `東京`) get no message typeahead
until a third character. A trailing contraction or hyphenated word with
a short part (`don't`, `e-m`) gets none until the next word is started.
Full-text mode is unaffected.
- Clients should not show "no results" for one- and two-character input.
A client-side minimum is a companion change. That includes the legacy
desktop topbar in this repo (`desktop/src/features/search/hooks.ts`,
which searches at 2 characters globally and 1 in a channel); it is left
out here to keep this PR relay-only.
## Testing
- Extended
`prefix_mode_matches_final_token_prefix_without_changing_full_text` with
a table of prefix queries and the exact hit sets they must return. It
pins no hits for a trailing token that yields a lexeme below three
characters: alone, after a completed word (`project pr`), padded with
punctuation (`pr.`, `@pr`, `#pr`, `...p`), as a contraction (`don't`),
and hyphenated (`pre-p`). It covers `kinds = [9]`, no kinds, mixed `[0,
9]`, and `Newest` sort. Each of those rows has data it would match
without the minimum. The same table has controls that must still match:
`pro.`, `project ?` (punctuation-only trailing token), `pre-pro`, `t
pan` (a short completed word stays exact and isn't limited), and a
one-character kind:0 `p`. A full-text `don't` control shows that
full-text mode is unaffected.
`short_kind0_prefix_prioritizes_exact_lexeme_on_a_noisy_page` pins the
kind:0 exemption.
- Mutation-checked on Blox. Each of these fails a test: removing the
minimum, removing the kind:0 exemption, and applying the minimum to
every lexeme instead of only the trailing ones.
- Synthetic plan check on Blox Postgres (300k rows, GIN on a generated
`simple` tsvector, same `CROSS JOIN LATERAL` query shape). `'pr':*` took
3.6 s and read 10,411 buffers. The NULL tsquery took 0.01 ms: the bitmap
scan returned 0 rows and read no buffers.
- Two existing prefix tests used a two-character trailing token
(`project pl`, `alpha be`). They now use three or more characters
(`project plan`, `alpha bet`) and assert the same behavior.
- Blox: `cargo fmt --check`, `cargo clippy -p buzz-search --all-targets
-D warnings`, `buzz-search` unit tests, and the `buzz-search` Postgres
lane (`scripts/postgres-test-run.sh`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Luke Tornquist
|
83bd8b1cae | fix(loadgen): sign public URLs when dialling connect_addr in plaintext (#8926) | ||
|
Luke Tornquist
|
46da35810d | db: recycle pool connections after 5 minutes instead of 30 (#8951) | ||
|
Luke Tornquist
|
02925ceb6e |
fix(relay): cap bridge searches in flight per user (#8956)
## Summary
Typeahead starts a bridge `/query` search on almost every keystroke. The
client aborts the previous request. But if the abort never reaches the
relay, for example because an intermediary keeps the upstream request
open, every one of those searches keeps running.
Each `(community, pubkey)` may now have **4** bridge searches in flight.
The next search supersedes the oldest one. The superseded request
returns **409** `{"error":"search superseded by a newer search"}`. It is
deliberately not a 429: desktop arms its relay-wide admission gate on
any 429, which would hold every relay request for 10 s, and the CLI
retries a `rate-limited:` 429. Its future is dropped, so #8954's
cancel-on-drop cancels its statement on the server.
- **Newest wins.** The newcomer is never rejected, because the newest
keystroke is the one the user is waiting for.
- **The cap is above 1.** Sessions that share a pubkey keep their
parallel searches: several agent sessions, or people search and message
search running side by side.
- **Bookkeeping lives in process.** `SearchSlots` holds a
`Mutex<HashMap<key, VecDeque<(id, CancellationToken)>>>`. A slot removes
itself by id when its search finishes, fails, is dropped or is
superseded. So a superseded search can never remove the newer search
that replaced it, and a user's entry is removed when their last search
leaves.
- **A search that finishes as it is superseded still returns its
result.** The `select!` is biased toward the search.
The limit is per relay process, not global. That is enough to bound how
much one user can pile onto one process's search pool.
## What got simpler
Nothing was removed. The change adds one small, self-contained module
and one call site. It also wires the new tests into the unit lane
(`Justfile test-unit` and `scripts/run-tests.sh`). The relay's unit lane
is an allowlist, so without that wiring the tests would run nowhere in
CI.
## Tests
`api::bridge::search_slots::tests`:
- `the_search_past_the_cap_supersedes_the_oldest`
- `a_finished_search_frees_its_slot`
- `a_superseded_search_leaves_its_successor_in_place`
- `no_entry_outlives_its_searches`
- `a_superseded_search_is_a_conflict_not_back_pressure`
Each of these mutants made at least one test fail:
- Removing the cap.
- Replacing the by-id removal with `pop_front`.
- Leaving empty user entries in the map.
- Answering supersession with 429.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Luke Tornquist
|
949f61dfab | fix(search): cancel abandoned searches and bound the search pool (#8954) | ||
|
Phil Azar
|
8f83c63209 |
fix(db): preserve partition reader grants (#8485)
## Summary PostgreSQL does not copy parent table grants to new partitions. A reader can query the `events` parent but fail direct partition reads after the relay creates another month or replaces its catch-all. Copy each managed parent's explicit table-level `SELECT` grants to its new children inside the existing maintenance transaction. The parent ACL supplies the policy; there is no new setting or deployment-specific role name. Writes, column-only grants, owner entries, and grant options are excluded. A grant failure rolls back the table's partition changes. Existing children and manually created or attached partitions still need grants from database provisioning. Apply the [companion initializer fix](https://github.com/squareup/bb-utility/pull/2253) for existing partitions before relying on automatic grants for future ones. ### Related issue No duplicate found. The closest related PR is #8464, which enables catch-all advancement by default; this change independently preserves reader access during that maintenance. ### Testing No manual testing. Production rollout and the Fivetran setup check remain pending. Generated with Codex Signed-off-by: coder2 <4acc49cb93576448ba82e631688c51a40bf03d6a889d78fddf57917d198de10f@buzz.block.builderlab.xyz> Co-authored-by: coder2 <4acc49cb93576448ba82e631688c51a40bf03d6a889d78fddf57917d198de10f@buzz.block.builderlab.xyz> |
