4 Commits

Author	SHA1	Message	Date
Luke Tornquist	ff15c347c2	fix(db): auto-migrate prebuild-guarded indexes on small databases (#8802) ... ## Problem Migrations `0076`, `0078` and `0081` raise an exception on any table with at least one row unless its index was prebuilt with `CREATE INDEX CONCURRENTLY`. A relay that runs migrations at startup (`BUZZ_AUTO_MIGRATE`, on by default in the Helm chart) and has data in those tables therefore crash-loops on upgrade: ``` migration error: while executing migration 81: error returned from database: prebuild idx_nip_fi_profile_heads_names concurrently; see docs/profile-search.md ``` `0081` (#9019) hits every auto-migrating relay that has published NIP-FI verified profiles. `0076`/`0078` (#8523) hit any relay with artifact data or channel members. Each of these tables is small enough on such relays that a plain index build takes milliseconds. The root cause is `docs/database.md` Rule 6. It told authors that an index on a populated table MUST be built concurrently by an operator ahead of time, so migrations were written to refuse populated tables. With auto-migration that refusal is a crash loop, not a safe stop. ## Fix The migrations are already applied on some databases, so they can't be edited: sqlx would reject the changed checksums. Instead, `run_migrations_locked` builds their missing indexes first. For each entry in `INLINE_INDEX_BUILDS` (`0076`, `0078`, `0081`), it runs migrations through the version before it, then builds any missing index. Each build is a plain `CREATE INDEX` in its own transaction, under the same 1s `lock_timeout` and 5s `statement_timeout` the migrations use. That is the bounded inline build `0049` already does: - **Small populated databases** upgrade the same way empty ones do. - **Large databases** still fail closed. Only a lock or statement timeout (`55P03` / `57014`) gets the "prebuild concurrently" hint, pointing at that migration's own doc. Other errors pass through unchanged. The blocked migration is not recorded. - **Prebuilt indexes** are found with `to_regclass` and skipped before `CREATE INDEX`, so deployments that prebuilt concurrently never request the writer-conflicting lock. - **Invalid prebuilt indexes** are left for the migration to reject, as before. `CONCURRENTLY` is not used in this path. The runner holds the migration advisory lock, and another replica blocked on that lock holds a snapshot that a concurrent build would wait on. ### Rules - `docs/database.md` Rule 6: with `BUZZ_AUTO_MIGRATE` on, every migration must apply its own change on a populated database and must never refuse one because it has rows. A new index is built inline under the migration's budgets. Only deployments that apply migrations out of band (auto-migration off) prebuild concurrently. - Root `AGENTS.md` (step 4) and `migrations/AGENTS.md` repeat the never-refuse rule. - `docs/artifact-sync-deployment.md` and `docs/profile-search.md` describe the startup behavior first and the concurrent prebuild as the out-of-band path. ## Tests - `populated_database_auto_migrates_through_every_migration` (Postgres) migrates to `0075`, seeds artifact rows, and checks four phases: 1. **Lock held:** while a writer lock is held, the `0076` build fails closed. 2. **Index prebuilt:** with the `0076` index prebuilt, the build is skipped. 3. **Partial `0078`:** a blocked `0078` build keeps the builds before it committed. 4. **Verified profile seeded:** with a verified profile head inserted before `0081`, a final startup succeeds. It asserts that the applied versions equal every embedded migration and that all six indexes are valid, ready and live. - `frozen_prebuild_guards_have_inline_builds` (no database) scans `migrations/`. It fails if any migration refuses a populated table (`RAISE EXCEPTION 'prebuild …`) without an inline build, and if a new guard is added. - `migration_execution_cannot_bypass_schema_destruction_lock` allows exactly one production `run_to`, inside `run_migrations_locked`. ## What got simpler - Auto-migrating relays no longer need a manual, operator-only prebuild step for small deployments. That restores the chart README's contract that `helm upgrade` is the whole upgrade procedure. - Rule 6 no longer steers authors toward refusing populated tables, and a test enforces that. - The change adds one table-driven step for three frozen migrations and no new configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 13:52:06 -05:00
Kalvin C	3d26ed7dfd	feat(relay): match verified names in profile search (#9019) ... ## Summary `POST /profiles/search` only matches kind:0 `name` and `display_name`. In communities that publish NIP-FI verified profiles, a person's surname or directory username often appears only in their relay-signed kind:30382, so typeahead cannot find them by it. - Where the community advertises verified profiles, the search also matches the `display_name`, `username` and `full_name` tags of each current kind:30382 head signed by the relay key. - Verified names need a trailing prefix of at least three characters, as in NIP-50, so a short prefix doesn't make the relay rank every verified head. kind:0 keeps one-character typeahead. Lookups from heads to `events` also match `created_at`, so Postgres prunes to one partition. - Results are paged by person. Each person returns their current kind:0, when they have one, then their current kind:30382, when they have one, whichever matched. A person is ranked by their best match across both sources, using the existing tiers. - NIP-11 `federated_identity.public_profile` gains `searchable: true`, so clients can rely on the response instead of a separate kind:30382 read. - Migration `0081` adds a GIN expression index over the head's name tags on `nip_fi_profile_heads` (built-in `jsonb_path_query_first` and `to_tsvector`, no new functions). The table holds one row per verified person. Like `0078`, the migration builds the index only while the table is empty; populated databases prebuild it with `CREATE INDEX CONCURRENTLY` (`docs/profile-search.md`), and the migration then checks it is valid and has the expected definition. Compatibility: the request schema is unchanged. Existing clients that keep only kind:0 events work as before and also find people who match only by verified name. Trust stays pinned to the relay `self` key; the event shape follows NIP-85 user assertions, but discovery is relay-advertised rather than kind:10040. ## Testing - `buzz-search` Postgres integration: new `profile_names_match_verified_names` covers forged-key heads, replaced heads, other communities, people without kind:0, per-person grouping, paging, and words split across sources; the existing profile-name tests pass unchanged. The test fails when the author filter is inverted. - `buzz-relay` end to end in enforce mode: `profile_search_matches_verified_names_where_advertised` publishes through the real writer and finds the person by surname only where verified profiles are advertised. - Desired-state and migration index definitions match. - `profile_names_use_verified_name_index` asserts the search scans the new index. Migration `0081` checked against an empty table, a rerun, a populated table without the index (raises), the documented concurrent prebuild, and an index with a different definition (raises). - Latency, 50k people each with a kind:0 and a verified head, release build, local Postgres, best of 5, 30 people per page: 11–22 ms with verified names versus 1–8 ms kind:0 only. `EXPLAIN` shows a bitmap scan on the new index. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kalvin Chau <kalvin@block.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 21:25:15 -05:00
Logan Johnson	7852b1670c	fix(search): normalize profile name tiers in one place (#8334) ... 🤖 ## Summary Profile name search (`POST /profiles/search`, added in #8301) sorts results into tiers: an exact name first, then a name that starts with the query, then a name with a later word that starts with it. This PR stacks on #8301 and fixes two review items. **Tier normalization in one place.** Before this change, the query was lower-cased in Rust (`to_lowercase`), and the names were lower-cased in SQL (`lower()`). These can disagree. Under a `C` collation, PostgreSQL's `lower()` changes only ASCII letters, while Rust also lower-cases letters like `É`. So a search for `Émile` did not put the profile named `Émile` in the exact tier. Now the query and both name fields go through the same SQL expression: lower-case, collapse whitespace runs to one space, then trim. The query is bound once and both sides compare equal by construction. The trim also moved after the collapse, so surrounding whitespace that the regex turns into a space is now trimmed too. **Ranking wording.** The doc comment and `docs/profile-search.md` said "ties keep the NIP-50 order". That was wrong. Ties are ordered by `ts_rank_cd` over the two name fields only (not the whole profile, as in NIP-50), then newest first, then event id. Out of scope, not changed here: matching names that contain escaped Unicode or newlines, and any new name index. ## Related issue Review on #8301: https://github.com/block/buzz/pull/8301#pullrequestreview-5524491230 ## Testing - New case in `profile_names_match_name_words_only`: profiles `Émile` and a newer `Émilee`, query `Émile`. The exact match must come first. - Ran the Postgres integration tests against a `postgres:17-alpine` database created with `--locale=C`: - this branch: `profile_names_match_name_words_only` and `profile_names_survive_unparsable_content` pass; - #8301 code with the new case: `profile_names_match_name_words_only` fails (`Émilee` sorts before `Émile`). - Nine other tests in that file fail locally on both this branch and #8301 because they reuse one database (duplicate community hosts). CI gives each test its own database. - `cargo fmt --check`, `cargo clippy -p buzz-search --all-targets -D warnings`, and `cargo test -p buzz-search`: clean. - The pre-push hook's `rust-tests` and `desktop-tauri-checks` passed. Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-15 18:02:10 -04:00
Logan Johnson	f148c0f22c	feat(relay): add profile search over name and display_name only (#8301) ... 🤖 > **Review as a pair with block/buzz-app#1084** (the app change that uses this endpoint). Either can merge first: the app falls back to the old search on relays without this endpoint. ## Summary Searching for a person by name over the relay's general search returns profiles that only *mention* the text somewhere else: in an avatar image embedded as text, in the bio, or in the verification address. A search for "wes" could fill every result page with profiles whose avatar data happens to contain "Wes", and never return the person actually named Wes. This adds a relay endpoint, `POST /profiles/search`, that looks only at a profile's **name** and **display name**. Every other profile field is ignored. **Matching and paging are the same as the relay's existing full-text profile search** (NIP-50 search over kind:0 profiles with `search_mode: "prefix"`). A client can switch to this endpoint without a visible change in which people a query finds: - Each query word must match a whole word of the name; the last word can be a prefix. Words can be in any order, so "johnson wes" finds "Wes Johnson". Case is ignored. - Pages work the same way: up to 100 results per page, up to page 1000. - The results are exactly the old results minus profiles that match only outside their names. The order is better for people search: exact name first, then names that start with the query, then names with a later word that starts with it, then other matches. Ties are ordered by `ts_rank_cd` over the two name fields only (not the whole profile, as in NIP-50), then newest first, then event id. The response is a JSON array of the original signed profile events (Nostr kind:0), so clients can verify them as they do today. The buzz-app client change is block/buzz-app#1084. ## Details - Request: `{"query": "wes", "limit": 20, "page": 1}`. `query` is 1–100 characters after trimming; `limit` is 1–100 (default 20); `page` is 1–1000 (default 1); unknown fields are rejected with 400. - The search reuses the full-text search's own query builder (`buzz-search`), so the word rule cannot drift from it. The existing full-text index on events stays the access path; the name check runs only on rows the index already matched. There is no new index. One small migration (`0061`) adds the function `profile_name_fields()`; it reads no table data, so it runs instantly. - Only current profiles count: a replaced profile stops matching. Profiles that are not a JSON object, or whose name fields are not strings, never match. Content that PostgreSQL cannot read as `jsonb` (for example a `\^@` escape or very deep nesting) also never matches, and it cannot make the search fail for everyone else. - The handler loads the matched events through the normal routed event read and applies the same reader visibility check as other reads, in search order. - Access uses the same checks as the relay's GIF search endpoint: a signed HTTP request (NIP-98), identity admission (NIP-FI), rate limit, replay protection, and relay membership. That code moved out of `gifs.rs` into one shared helper in `bridge.rs`; GIF behavior is unchanged. - The route is added to the existing protected-route and NIP-FI shadow/enforce route tables. Docs: `docs/profile-search.md`. - Two test-only lines get `sadscan:disable` comments, as other fixtures already do: an existing public ES256 test key in `router.rs`, and the local Postgres test URL in the search integration test. ### Related issue None found. ### Testing - Postgres test for the search query (`profile_names_match_name_words_only`): only name fields match (avatar data URL, `about`, `nip05`, `banner`, non-string names and non-object content are excluded); words in any order and across both name fields; whole words only for completed words; ranking order; pages split without overlap or gaps; replaced profile; other community; `%` literal. - Postgres end-to-end test through the router: an admitted member gets only the name-matching signed event, and the returned event still verifies. - Request validation unit tests; existing GIF admission tests and router route-table tests pass. - Local end-to-end run with buzz-app#1084 against a relay built from this head, with about 195 seeded profiles. For every query tried, the endpoint's results were a subset of the full-text search's results; the only profiles missing were avatar-only matches. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-15 17:35:33 -04:00