5c91e0d6fe
## Summary
This PR adds kind 30180, `KIND_EMOJI_ENTRY`. One event holds one custom
emoji.
The relay accepts, validates, stores, and replaces these events.
The SDK gets `build_emoji_entry`.
`docs/nips/NIP-CE.md` documents the kind.
This is the relay part of the custom-emoji scale-up (Option C).
It does not change kind 30030 sets, message `emoji` tags, or reaction
`emoji` tags.
It adds no migration and no schema change.
**Read this before you deploy.** Prefix search for kind 30180 works only
on a database whose `events.search_tsv` indexes that kind. See "Search
index" below.
## Design
- `crates/buzz-core/src/kind.rs`: `KIND_EMOJI_ENTRY = 30180`. It is in
`ALL_KINDS`. A compile-time check proves it is parameterized
replaceable. 30180 was free on `origin/main`.
- `crates/buzz-relay/src/handlers/ingest.rs`:
- The kind needs `UsersWrite`.
- The kind is global-only. It is never channel-scoped.
- `validate_emoji_entry` runs before storage. It requires exactly one
`d` tag and exactly one `emoji` tag.
- It rejects any `h` tag.
- It requires `d` to equal the shortcode in the `emoji` tag, and both to
equal the normalized shortcode.
- It requires an `http://` or `https://` URL of at most 2048 bytes. It
uses the SDK check `check_custom_emoji_url`, which is now public.
- It caps `content` at 256 bytes (`MAX_EMOJI_ENTRY_CONTENT_LEN`). A
shortcode is at most 64 bytes, so the cap leaves room.
- Every rejection starts with `invalid:`.
- `crates/buzz-sdk/src/builders.rs`: `build_emoji_entry(shortcode, url)`
and `emoji_entry_search_text`. The content is the shortcode with `_` and
`-` replaced by spaces.
- Replace semantics use the existing parameterized-replaceable path. A
second publish with the same `(pubkey, d)` leaves one live event. Two
authors with the same `d` keep two events. The NIP tells clients to
prefer the newest event and report the conflict.
### Decisions
- **WebSocket `search_mode`: not changed.** Prefix search works through
`POST /query`, which already reads `search_mode`. This is the smallest
correct change. The `REQ` search path still uses full-word matching. The
NIP says so. A client that needs typeahead uses `POST /query`.
- **Search text.** Postgres `simple` already splits `party_parrot` at
the underscore. It keeps `sad-parrot` as one token and also indexes
`sad` and `parrot`. So the space replacement is not needed for matching
today. It is kept because it is harmless and does not depend on parser
details.
- **Content cap of 256 bytes.** The cap is documented in the NIP and the
code doc comment.
### Search index (important)
The design doc says kind 30030 is not excluded from the full-text index.
That is true for `schema/` (the desired state): it indexes every kind
except a deny list. It is not true for a database that was created by
`migrations/`:
- Migration 0008 gives a fresh install a positive allowlist: kinds 0, 9,
40002, 45001, 45003.
- `scripts/maintenance/nip_rs_search_allowlist.sql` moves a populated
database to the same allowlist.
On an allowlist database, kind 30180 is **stored and resolvable by `#d`,
but not searchable**.
I did not change this. A change to `search_tsv` rewrites the partitioned
`events` table. That needs the reviewed out-of-band procedure in
`docs/database.md` (rules 6 and 5), and this task forbids migrations and
schema files.
Before clients depend on search, a human must check the deployed
expression:
```sql
SELECT pg_get_expr(adbin, adrelid) FROM pg_attrdef d JOIN pg_attribute a
ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attname = 'search_tsv' AND d.adrelid = 'events'::regclass;
```
If the result lists `kind IN (0, 9, ...)`, add 30180 to the allowlist
through the out-of-band procedure first.
## Spike
I ran this on a throwaway local Postgres 17 (Docker), desired-state
schema from `pgschema`.
The loader inserted 100,000 kind 30180 events from 200 authors through
`Db::replace_parameterized_event`. It used 16 parallel tasks, and it did
not use the network.
The loader was a scratch example. It is not in this PR.
Rows landed in `events_p_future` because the local partitions end at
2026-06. Real deployments have monthly partitions, so scans split across
partitions.
**Insert rate.** 100,000 inserts took 33.9 s (about 2,900 per second) at
the DB layer. The relay adds rate limits on top (see the design doc,
section 2.5).
**`#d` batch of 50 names** (actual relay SQL shape, warm cache, `LIMIT
100`):
| Case | Plan | Execution time |
|------|------|----------------|
| No `authors` | Index scan on `idx_events_parameterized`. It uses
`community_id` and `kind` only. `d_tag` is not a leading key, so it
scans the kind range. | 12 ms warm, 28 ms cold |
| 50 names, all their authors (50 distinct) | Index scan with `pubkey`
and `d_tag` in the index condition | 1.5 ms |
| 50 names, one author | Same index, tight condition | 0.13 ms |
Result: with `authors`, the lookup is fast. Without `authors`, the cost
grows with the number of entries of the kind. At 100,000 rows it is
still tens of milliseconds. No index change is needed now. Re-measure at
1,000,000 rows.
**Prefix search**, `kinds=[30180]`, relevance order (the default),
`LIMIT 50`:
| Query | Matching rows | Execution time (warm) |
|-------|---------------|-----------------------|
| `parr` | 8,169 | 115 ms (124 ms cold) |
| `p` | 15,990 | 219 ms |
| `party par` | 8,169 | 266 ms (cold) |
| `zap blob 999` (rare) | 1 | 0.4 ms |
| `parr` with `search_sort: "newest"` | 8,169 | 0.8 ms |
Result: the GIN index finds matches fast. Relevance order then scores
every matching row (`ts_rank_cd`) and sorts them. The cost grows with
the number of matches, not with `limit`. My synthetic words are few
(24), so each prefix matches many rows. Real names are more varied, but
a one- or two-letter query will still match many rows. The `search_sort:
"newest"` option avoids the scoring and was 150 times faster here.
Clients should use `search_sort: "newest"` for typeahead, or require at
least 3 characters. This is a recommendation for the client work. It
needs no relay change.
**Side effects of bulk insert** (from reading `ingest.rs` and
`handlers/event.rs`):
- Audit log: each accepted event queues one `EventCreated` audit entry.
The audit worker writes them one at a time under an advisory lock. The
queue holds 1000 entries and uses `send().await`, so a bulk run slows
the writer down instead of dropping entries. 96,000 events add 96,000
audit rows.
- Live fan-out: each event is published to the global topic and matched
against open subscriptions in `sub_registry`. A client with an open
filter for kind 30180 receives every event.
- Workflow triggers: each event spawns a task that calls
`workflow_engine.on_event`. That is one trigger lookup per event.
- Push: the DB push enqueue only matches kinds 9, 40002, 45001, and
45003 (`PUSH_MATCH_KINDS`). Kind 30180 does not wake push subscribers.
- No `channel_id`, so the `events_created_at_floor_guard` trigger does
not apply.
## Testing
What ran (all on the final tree):
- `cargo test -p buzz-core -p buzz-sdk -p buzz-relay -p buzz-search`:
pass.
- `cargo clippy -p buzz-core -p buzz-sdk -p buzz-relay -p buzz-search
--all-targets -- -D warnings`: pass.
- `cargo fmt --all --check`: pass.
- `just file-size-check`: pass.
- Postgres lane, desired-state schema (`BUZZ_TEST_SCHEMA_MODE=desired`)
with a local Redis: `cargo test -p buzz-relay --lib handlers::ingest --
--include-ignored`: 205 passed, including the four new tests.
New tests:
- `emoji_entry_validation_table` (no infra): one row for each rule.
- `emoji_entry_is_a_global_users_write_kind` (no infra).
- `emoji_entry_ingest_validates_and_replaces` (Postgres): goes through
`ingest_event`. It rejects a bad shortcode, `d` not equal to shortcode,
an `h` tag, and a non-http URL. It checks that nothing is stored. It
then publishes twice with the same `d` and checks that one live event
remains and that it is the newer one.
- `emoji_entry_prefix_search_finds_by_word_prefix` (Postgres): `parr`
finds `party parrot` and `sad parrot`. `party par` finds one. `joy`
finds `bufo joy`. It fails with a clear message on an allowlist
database.
- SDK: `emoji_entry_happy_path` and `emoji_entry_rejects_bad_input`.
Not run:
- `just test` was not run in full. I ran the relevant Postgres tests
directly.
- `cargo test -p buzz-search --test postgres_fts_integration` fails 23
of 23 on my shared local database. Those fixtures use fixed hostnames
and need the fresh-database lane (`scripts/postgres-test-run.sh`). I did
not change that crate. This is an environment limit, not a result of
this change.
- The new search test fails on a database built by `buzz-admin migrate`
(allowlist). That is the finding above. It passes on the desired-state
schema that the CI lane uses (`postgres-test-run.sh`).
- I ran no live relay and no end-to-end client.
Interop evidence: none for this item. The desktop and mobile constants
are out of scope here. `/home/bloxer/ref/interop.mjs` was not needed
because no TypeScript changed.
Mutation checks (break the guard, confirm a test fails, restore):
| Mutation | Result |
|----------|--------|
| Remove the `validate_emoji_entry` call |
`emoji_entry_ingest_validates_and_replaces` fails |
| Remove the `h` tag check | table test and ingest test fail |
| Remove the `d == shortcode` check | table test and ingest test fail |
| Remove the URL check | table test and ingest test fail |
| Remove the content cap | table test fails |
| Allow more than one `d` tag | table test fails |
| Allow more than one `emoji` tag | table test fails |
| Remove the kind from `required_scope_for_kind` | scope test, ingest
test, and search test fail |
| Remove the kind from `is_global_only_kind` | global test and ingest
test fail |
| SDK: skip shortcode normalization | both SDK tests fail |
| SDK: skip the URL check | `emoji_entry_rejects_bad_input` fails |
| SDK: skip the space replacement | `emoji_entry_happy_path` fails |
One guard has no relay-level test: the space replacement does not change
search results, because Postgres already splits `_`. The SDK test covers
it.
## Manual test steps
You need Docker and a local checkout. Use a local relay only.
1. Start the services and apply the desired-state schema, so that search
indexes kind 30180:
```bash
cd ~/buzz && . ./bin/activate-hermit
docker run -d --name buzz-pg -e POSTGRES_USER=buzz -e
POSTGRES_PASSWORD=buzz_dev -e POSTGRES_DB=buzz -p 127.0.0.1:5432:5432
postgres:17-alpine
redis-server --daemonize yes --save ""
PGHOST=localhost PGPORT=5432 PGUSER=buzz PGPASSWORD=buzz_dev
PGDATABASE=buzz ./bin/pgschema apply --file schema/schema.sql
--auto-approve
docker exec -i buzz-pg psql -U buzz -d buzz -v ON_ERROR_STOP=1 <
scripts/reconcile-schema-after-pgschema.sql
```
Success: `Changes applied successfully!`.
2. Run the new tests:
```bash
export
BUZZ_TEST_DATABASE_URL=postgres://buzz:buzz_dev@localhost:5432/buzz
BUZZ_TEST_SCHEMA_MODE=desired
cargo test -p buzz-relay --lib emoji_entry -- --include-ignored
cargo test -p buzz-sdk --lib emoji_entry
```
Success: 4 relay tests and 2 SDK tests pass.
3. Optional, with a local relay (`just relay`) and a local key. Send
these through `POST /events` with the SDK or a small script that calls
`build_emoji_entry("party_parrot", "https://example.com/p.gif")`.
Success: the relay returns `accepted: true`.
- Publish a second entry with the same shortcode. Query `POST /query`
with `{"kinds":[30180],"#d":["party_parrot"]}`. Success: one event, the
newer one.
- Query
`{"kinds":[30180],"search":"parr","search_mode":"prefix","limit":50}`.
Success: the entry is in the result.
- Publish an entry that has an `h` tag or an `ftp://` URL. Success: the
relay returns `invalid:`.
4. Check your target database before you rely on search (the SQL in
"Search index"). Success: the expression does not list an allowlist such
as `kind IN (0, 9, 40002, 45001, 45003)`. If it does, search does not
find kind 30180 yet.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: bmcanally <bmcanally@squareup.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
5.2 KiB
NIP-CE
Custom Emoji Entries
draft optional relay
This NIP defines kind:30180, an addressable event that holds one custom emoji. A community can hold many thousands of emoji without any client loading the whole catalog.
NIP-30 and NIP-51 put many emoji in one kind:30030 set. A set is one event, so every client must read and parse the whole set. This NIP uses one event for each emoji. Clients ask the relay for the names they need.
Event
kind:30180 is an addressable event keyed by (pubkey, kind, d) as defined in NIP-01.
{
"kind": 30180,
"content": "party parrot",
"tags": [
["d", "party_parrot"],
["emoji", "party_parrot", "https://example.com/media/9f2c.gif"]
]
}
Tags
An entry MUST have exactly one d tag and exactly one emoji tag.
The emoji tag has the form ["emoji", <shortcode>, <url>], as in NIP-30.
The shortcode MUST be 1 to 64 bytes of ASCII letters, digits, hyphens (-), and underscores (_). It MUST be lowercase. The d value MUST equal the shortcode in the emoji tag.
The URL MUST start with http:// or https://. It MUST NOT be longer than 2048 bytes.
An entry MUST NOT have an h tag. Entries are never scoped to a channel.
Content
content is the search text. It is the shortcode with each _ and - replaced by a space. For party_parrot-2 the content is party parrot 2.
Full-text search splits on spaces. The replacement lets a search for parrot find party_parrot.
content MUST equal this search text exactly. A relay MUST reject an entry with any other content, including empty content. Because the content comes from the shortcode, it is never longer than 64 bytes.
The rule keeps the full-text index in step with the shortcode. An author cannot add keywords that the shortcode does not contain.
Replace semantics
A new entry with the same (pubkey, d) replaces the old entry. The relay keeps the entry with the newest created_at. When two entries have the same created_at, the relay keeps the entry with the lowest event ID. This is the standard NIP-01 rule.
A member can only replace their own entries. Another member who publishes the same d creates a second, separate entry. The relay stores both.
When clients see several entries for one d, they MUST pick the entry with the newest created_at across all authors. Clients SHOULD report the conflict to the user. Clients MUST apply the same tie-break as the relay.
To remove an entry, the author sends a NIP-09 deletion request.
Reading
Exact lookup
To resolve names, send a filter with kinds and #d:
{ "kinds": [30180], "#d": ["party_parrot", "sad_parrot"] }
A filter SHOULD NOT carry more than 50 names. A client that knows the author SHOULD add authors. Without authors, a relay can scan every entry of the kind to answer.
Prefix search
To search, use NIP-50 search over HTTP POST /query with prefix mode:
{ "kinds": [30180], "search": "parr", "search_mode": "prefix", "limit": 50 }
Prefix mode treats each word except the last as a whole word. It treats the last word as a prefix. A search for party par finds party parrot. A search for joy finds bufo joy.
The relay returns at most 500 hits for each request. Clients SHOULD ask for 50 or fewer.
The relay reads search_mode on POST /query only. A WebSocket REQ with search uses full-word matching.
Search needs the relay to index content of kind:30180 in its full-text index. A relay that indexes only an allowlist of kinds MUST add 30180 to the allowlist before clients rely on search. Exact lookup does not depend on the full-text index.
In Buzz, a database that migration 0008_fresh_install_search_allowlist.sql built, or that an operator converted with scripts/maintenance/nip_rs_search_allowlist.sql, indexes only an allowlist of kinds. Such a database does not index kind:30180 until an operator runs the updated maintenance script. The operator MUST run it before the emoji migration rollout. Without it, entries are stored and exact lookup works, but prefix search returns no results.
Relay behavior
A relay that supports this NIP:
- MUST accept
kind:30180from any community member with the same permission askind:30030. - MUST reject an invalid entry with an
invalid:message. - MUST store entries as global events, not channel events.
Relationship to kind 30030
This NIP does not change kind:30030. Sets with d equal to buzz:custom-emoji keep their meaning. A client MAY read both. A client that reads both SHOULD prefer the newest event when a shortcode appears in both.
The emoji tags on messages and reactions do not change. A message carries the emoji tags for the emoji it uses, so history renders without any lookup.
Clients that do not know kind:30180 ignore these events.
Limits summary
| Item | Limit |
|---|---|
| Shortcode | 1 to 64 bytes, [A-Za-z0-9_-], lowercase in d and emoji |
| URL | 2048 bytes, http:// or https:// |
| Content | 256 bytes |
d tags |
exactly 1 |
emoji tags |
exactly 1 |
h tags |
0 |
Names per #d filter |
50 (SHOULD) |
| Search hits per request | 500 (relay maximum) |