Files
buzz/scripts/maintenance/profile_search_text_fields.sql
Matt Toohey 7a2cc00001 feat(search): index kind-0 profile text fields instead of avatar bytes
`to_tsvector('simple', content)` over a kind-0 event tokenizes the whole
profile JSON, including an inline base64 avatar, so 111 agent profiles
sharing one `picture` matched `wes:*` and pushed the real Wes to row 121.
Add `profile_search_tsv(content)`, an IMMUTABLE SQL function that indexes
`name`/`display_name`/`displayName` at weight A, `nip05`/`lud16`/`website`
at B, and `about` at D, skipping `picture`, `banner`, `image`, and unknown
keys. `pg_input_is_valid` (PostgreSQL 16+) guards the `::jsonb` cast so
rows that predate ingest validation fall back to today's raw-text vector,
and `CASE` keeps the cast from ever being evaluated for invalid input; the
IMMUTABLE declaration is truthful because `jsonb_in` is immutable. The
`search_tsv` generated column gains a `kind = 0` arm ahead of the privacy
blocklist in `schema/schema.sql`, which stays intact.

Migration 0056 creates the function unconditionally and rewrites the
column only when `events` is empty, in 0008's shape, because replacing a
generated column copies every partition and rebuilds the partitioned GIN
under ACCESS EXCLUSIVE with no lock_timeout at relay startup. Populated
databases take the new out-of-band
`scripts/maintenance/profile_search_text_fields.sql`, which wraps the live
expression read through `pg_get_expr` so it is correct for both the
fresh-install allowlist and the brownfield blocklist, refuses to run before
0056 has created the function, is a no-op once applied, and recreates the
GIN index under a 5s lock_timeout. The older
`nip_rs_search_allowlist.sql` rewrite now emits the same kind-0 arm so
running it afterwards cannot undo this one. Docs in `buzz-search`,
CONTRIBUTING.md, and ARCHITECTURE.md describe the new expression.

Tests, each verified to fail when the seam it binds is removed:
`kind0_avatar_data_url_token_is_not_searchable` (desired-state lane; two
hits with the `kind = 0` arm removed from schema.sql) and, in the
migration lane, `fresh_install_profile_search_uses_text_fields`,
`maintenance_profile_search_rewrite_drops_avatar_tokens_on_populated_database`
(avatar row keeps matching with the wrap removed from the script; also
checks the GIN index and the idempotent rerun), and
`maintenance_profile_search_rewrite_wraps_fresh_install_allowlist` for the
post-0008 expression shape. `populated_upgrade_preserves_search_policy_
except_for_private_kinds` gains a kind-0 avatar row and asserts 0056
leaves a populated column untouched. The embedded migrator count moves to
56; the fresh-install allowlist assertion drops 0 from the array because
kind 0 now has its own arm. The two kind-0 tests from the earlier commits
and the display-name and short-prefix tests pass unchanged.

Verified against local PostgreSQL 17.9 with
`scripts/postgres-test-run.sh -p buzz-search --tests` (23 passed) and
`scripts/postgres-test-run.sh -p buzz-db --lib -E 'test(migration::postgres_tests)'`
(12 passed), `cargo test -p buzz-db --lib runtime::migration`,
`scripts/test-postgres-test-discovery.sh`, `cargo fmt`, and
`cargo clippy -p buzz-db -p buzz-search --all-targets -D warnings`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matt Toohey <contact@matttoohey.com>
2026-10-02 14:29:13 +10:00

74 lines
3.3 KiB
PL/PgSQL

-- OUT-OF-BAND MAINTENANCE: do not run from relay startup migrations.
--
-- Route kind-0 (profile) rows of a populated database through
-- profile_search_tsv(content) so profile search indexes names, not inline
-- avatar bytes. Migration 0056 creates the function at relay startup and
-- rewrites the column only when `events` is empty; this script finishes the
-- job for every other database. Run it after the relay has started at least
-- once on 0056 (the guard below refuses otherwise):
--
-- psql -v ON_ERROR_STOP=1 -f scripts/maintenance/profile_search_text_fields.sql
--
-- It wraps whatever expression the database has today, so it is correct for
-- both shapes in the field: the fresh-install allowlist from 0008 (kind 0 is
-- one of ARRAY[0, 9, 40002, 45001, 45003]) and the brownfield blocklist from
-- 0001/0005 (kind 0 falls through to to_tsvector). Every other kind keeps the
-- policy the database already had. Re-running is a no-op once applied.
--
-- Cost: PostgreSQL cannot alter a generated expression in place. DROP COLUMN is
-- metadata-only, but ADD COLUMN ... GENERATED ... STORED copies every row of
-- every events partition evaluating the expression, then CREATE INDEX rebuilds
-- the partitioned GIN. All of it runs under ACCESS EXCLUSIVE on `events` in
-- one transaction, so relay reads and writes of events queue behind it.
-- Transient disk is roughly a second copy of the events heap and TOAST plus
-- the new GIN plus WAL. Time is dominated by tokenizing every indexed chat
-- row (about 20 MB of content per second single-threaded). Size it first:
--
-- SELECT kind, count(*), pg_size_pretty(sum(octet_length(content)))
-- FROM events GROUP BY kind ORDER BY 2 DESC;
-- SELECT pg_size_pretty(pg_total_relation_size('events'));
--
-- Rehearse on a staging clone, time it, then schedule production. The
-- lock_timeout makes a busy relay fail the script quickly rather than queue
-- traffic behind the lock wait; rerun when the table is quieter. The index is
-- recreated from the stock definition; non-stock indexes or storage
-- parameters on search_tsv are not captured or replayed.
BEGIN;
SET LOCAL lock_timeout = '5s';
DO $$
DECLARE
existing_expression TEXT;
BEGIN
IF to_regprocedure('profile_search_tsv(text)') IS NULL THEN
RAISE EXCEPTION 'profile_search_tsv(text) is missing; start the relay once so migration 0056 creates it, then rerun';
END IF;
SELECT pg_get_expr(d.adbin, d.adrelid)
INTO existing_expression
FROM pg_attrdef d
JOIN pg_attribute a
ON a.attrelid = d.adrelid
AND a.attnum = d.adnum
WHERE d.adrelid = 'events'::regclass
AND a.attname = 'search_tsv';
IF existing_expression IS NULL THEN
RAISE EXCEPTION 'events.search_tsv generated expression not found';
END IF;
IF existing_expression LIKE '%profile_search_tsv(content)%' THEN
RAISE NOTICE 'events.search_tsv already routes kind 0 through profile_search_tsv; nothing to do';
RETURN;
END IF;
ALTER TABLE events DROP COLUMN search_tsv;
EXECUTE format(
'ALTER TABLE events ADD COLUMN search_tsv TSVECTOR GENERATED ALWAYS AS (CASE WHEN kind = 0 THEN profile_search_tsv(content) ELSE (%s) END) STORED',
existing_expression
);
CREATE INDEX idx_events_search_tsv ON events USING GIN (search_tsv);
END $$;
COMMIT;