mirror of
https://github.com/block/buzz.git
synced 2026-11-06 17:31:19 +00:00
## Problem Migrations `0076`, `0078` and `0081` raise an exception on any table with at least one row unless its index was prebuilt with `CREATE INDEX CONCURRENTLY`. A relay that runs migrations at startup (`BUZZ_AUTO_MIGRATE`, on by default in the Helm chart) and has data in those tables therefore crash-loops on upgrade: ``` migration error: while executing migration 81: error returned from database: prebuild idx_nip_fi_profile_heads_names concurrently; see docs/profile-search.md ``` `0081` (#9019) hits every auto-migrating relay that has published NIP-FI verified profiles. `0076`/`0078` (#8523) hit any relay with artifact data or channel members. Each of these tables is small enough on such relays that a plain index build takes milliseconds. The root cause is `docs/database.md` Rule 6. It told authors that an index on a populated table MUST be built concurrently by an operator ahead of time, so migrations were written to refuse populated tables. With auto-migration that refusal is a crash loop, not a safe stop. ## Fix The migrations are already applied on some databases, so they can't be edited: sqlx would reject the changed checksums. Instead, `run_migrations_locked` builds their missing indexes first. For each entry in `INLINE_INDEX_BUILDS` (`0076`, `0078`, `0081`), it runs migrations through the version before it, then builds any missing index. Each build is a plain `CREATE INDEX` in its own transaction, under the same 1s `lock_timeout` and 5s `statement_timeout` the migrations use. That is the bounded inline build `0049` already does: - **Small populated databases** upgrade the same way empty ones do. - **Large databases** still fail closed. Only a lock or statement timeout (`55P03` / `57014`) gets the "prebuild concurrently" hint, pointing at that migration's own doc. Other errors pass through unchanged. The blocked migration is not recorded. - **Prebuilt indexes** are found with `to_regclass` and skipped before `CREATE INDEX`, so deployments that prebuilt concurrently never request the writer-conflicting lock. - **Invalid prebuilt indexes** are left for the migration to reject, as before. `CONCURRENTLY` is not used in this path. The runner holds the migration advisory lock, and another replica blocked on that lock holds a snapshot that a concurrent build would wait on. ### Rules - `docs/database.md` Rule 6: with `BUZZ_AUTO_MIGRATE` on, every migration must apply its own change on a populated database and must never refuse one because it has rows. A new index is built inline under the migration's budgets. Only deployments that apply migrations out of band (auto-migration off) prebuild concurrently. - Root `AGENTS.md` (step 4) and `migrations/AGENTS.md` repeat the never-refuse rule. - `docs/artifact-sync-deployment.md` and `docs/profile-search.md` describe the startup behavior first and the concurrent prebuild as the out-of-band path. ## Tests - `populated_database_auto_migrates_through_every_migration` (Postgres) migrates to `0075`, seeds artifact rows, and checks four phases: 1. **Lock held:** while a writer lock is held, the `0076` build fails closed. 2. **Index prebuilt:** with the `0076` index prebuilt, the build is skipped. 3. **Partial `0078`:** a blocked `0078` build keeps the builds before it committed. 4. **Verified profile seeded:** with a verified profile head inserted before `0081`, a final startup succeeds. It asserts that the applied versions equal every embedded migration and that all six indexes are valid, ready and live. - `frozen_prebuild_guards_have_inline_builds` (no database) scans `migrations/`. It fails if any migration refuses a populated table (`RAISE EXCEPTION 'prebuild …`) without an inline build, and if a new guard is added. - `migration_execution_cannot_bypass_schema_destruction_lock` allows exactly one production `run_to`, inside `run_migrations_locked`. ## What got simpler - Auto-migrating relays no longer need a manual, operator-only prebuild step for small deployments. That restores the chart README's contract that `helm upgrade` is the whole upgrade procedure. - Rule 6 no longer steers authors toward refusing populated tables, and a test enforces that. - The change adds one table-driven step for three frozen migrations and no new configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
87 lines
4.3 KiB
Markdown
87 lines
4.3 KiB
Markdown
# Profile name search
|
|
|
|
`POST /profiles/search` finds community members by profile name. It matches
|
|
only the `name` and `display_name` string fields of each current global kind:0
|
|
profile and, where the community advertises NIP-FI verified profiles
|
|
(`federated_identity.public_profile.searchable` in NIP-11), the
|
|
`display_name`, `username` and `full_name` tags of each current kind:30382
|
|
verified profile signed by the relay key. It does not search `picture`
|
|
(including data URLs), `about`, `nip05`, `banner`, or any other profile field. NIP-50 search over kind:0 matches the
|
|
whole profile JSON, so it can return profiles whose avatar data or bio happens
|
|
to contain the query text.
|
|
|
|
Apart from the fields it reads, the endpoint matches and pages like a NIP-50
|
|
kind:0 search with `search_mode: "prefix"`. A client can switch from one to the
|
|
other without a visible change in which people a query finds.
|
|
|
|
The endpoint requires NIP-98 authentication over the request body and relay
|
|
membership, like other relay-owned member endpoints. A relay without the
|
|
endpoint answers `404`; clients fall back to NIP-50 search for that request.
|
|
|
|
Request:
|
|
|
|
```json
|
|
{ "query": "wes", "limit": 20, "page": 1 }
|
|
```
|
|
|
|
- `query`: 1 through 100 characters after trimming. Required.
|
|
- `limit`: results per page, 1 through 100. Defaults to 20.
|
|
- `page`: 1-based page of `limit` results, 1 through 1000. Defaults to 1.
|
|
- Unknown fields are rejected with `400`.
|
|
|
|
Response: a JSON array of signed events for one page of people, best match
|
|
first. Each person contributes their current kind:0, when they have one, then
|
|
their current relay-signed kind:30382, when verified profiles are advertised
|
|
and they have one, whichever of the two matched. A page with fewer than
|
|
`limit` people is the last page. Clients that read only kind:0 keep working;
|
|
they also find people who match only by verified name.
|
|
|
|
Matching is the NIP-50 prefix rule, case-insensitive. The query is split on
|
|
whitespace. Every word except the last must equal a word of `name` or
|
|
`display_name`; the last word may be a word prefix. Words may appear in any
|
|
order and may be split across the two fields, so "johnson wes" finds
|
|
"Wes Johnson". A verified profile matches the same way over its three name
|
|
tags, except that its last word must be at least three characters, as in
|
|
NIP-50. The two sources match separately: a query whose words are split
|
|
between a person's kind:0 and their verified profile does not match.
|
|
Accents are not folded.
|
|
|
|
People are ordered by their best match across both sources:
|
|
|
|
1. a name equal to the query;
|
|
2. a name that starts with the query;
|
|
3. a name with a later word that starts with the query;
|
|
4. any other match.
|
|
|
|
For these tiers the query and the names are lower-cased, with whitespace
|
|
runs collapsed to one space and trimmed, by the same database expression.
|
|
Ties go by relevance of the query to the names alone (not the whole
|
|
profile, as in NIP-50), then newest first, then pubkey, so pages do not
|
|
overlap. Profile content that is not a JSON object, or a name field that
|
|
is not a string, never matches. Content that PostgreSQL cannot read as `jsonb`
|
|
(for example a `\u0000` escape or very deep nesting) has no names, so it
|
|
never matches and never makes the search fail
|
|
(`schema/functions/profile_name_fields.sql`).
|
|
|
|
## Deployment
|
|
|
|
Verified-name matching reads `idx_nip_fi_profile_heads_names`. Relay startup
|
|
migration and `buzz-admin migrate` build it before migration `0081` under a
|
|
one-second lock and five-second statement budget. Applied directly as SQL,
|
|
`0081` builds it only while `nip_fi_profile_heads` is empty. A deployment that
|
|
applies migrations out of band (auto-migration off) builds it first, outside a
|
|
transaction:
|
|
|
|
```sql
|
|
CREATE INDEX CONCURRENTLY idx_nip_fi_profile_heads_names
|
|
ON public.nip_fi_profile_heads USING GIN (
|
|
to_tsvector('simple',
|
|
COALESCE(jsonb_path_query_first(event, 'strict $.tags[*] ? (@[0] == "display_name")[1]', '{}', true) #>> '{}', '')
|
|
|| ' ' || COALESCE(jsonb_path_query_first(event, 'strict $.tags[*] ? (@[0] == "username")[1]', '{}', true) #>> '{}', '')
|
|
|| ' ' || COALESCE(jsonb_path_query_first(event, 'strict $.tags[*] ? (@[0] == "full_name")[1]', '{}', true) #>> '{}', ''))
|
|
);
|
|
```
|
|
|
|
If the build fails, drop the invalid index and build it again. The migration
|
|
then checks that the index is valid and has this definition.
|