mirror of
https://github.com/block/buzz.git
synced 2026-11-04 16:31:38 +00:00
6440aaec7c824d2ed7feef2c789c90afa1ed30d4
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Luke Tornquist
|
152441ae0a |
feat(loadgen): model desktop clients and large-community shape (#9005)
## Summary
The load generator models every community as many small open channels,
and every human as a bare WebSocket client: four REQs, a little history,
then nothing until the run ends. Large deployments don't look like that.
They have thousands of people in a few communities, a handful of very
wide channels, and some people in hundreds of channels. Their desktop
clients also make most of their relay requests over HTTP and on timers.
This PR adds opt-in scenario knobs for both, so a run can reproduce that
shape and compare the desktop client before and after its recent
request-volume fixes.
Every new key defaults to the previous behavior. Existing scenarios
parse, assign members and run exactly as before.
### Shape
| Key | Default | Meaning |
|---|---|---|
| `shape.private_channels` | `false` | `seed` creates channels private
and adds every member with kind:9000 before posting history. Members are
the humans and agents assigned to each channel up to the largest ramp
step, plus the hot thread's root and authors. Events carry up to 100
members each and run `seed_concurrency` at a time. A rerun adds nobody
new, and any member the relay refuses fails the seed. |
| `shape.wide_channels` | `0` | The first K channels of each community
are wide channels. |
| `population.wide_member_fraction` | `0.0` | Share of humans that join
every wide channel on top of their own. |
| `population.wide_activity_share` | unset | Share of its typing,
messages and reactions a member of the wide channels sends there; the
rest goes to its regular channels. Unset keeps the uniform pick, which
sends half of a 3-wide/3-regular member's traffic to the wide channels
and makes fan-out, not reads, the bottleneck. |
| `population.tiers` | `[]` | `[{ fraction, channels }]`: that share of
humans joins `channels` channels instead of `channels_per_member`. |
Assignment is deterministic by global index, so `seed` and every
`--shard` agree without coordination. With no wide channels and no tiers
it is the old assignment, and a test pins that.
### Desktop clients (`[desktop]`)
| Key | Default | Meaning |
|---|---|---|
| `fraction` | `0.0` | Share of humans that run the desktop model
instead of the basic human. |
| `mode` | `"current"` | `legacy` is the client before its volume fixes;
`current` is after them. |
| `windows`, `threads` | `3`, `1` | Open channel windows and threads,
which are refreshed on connect. |
| `tick_secs` | `60` | Recovery tick period. |
| `backstop_secs` | `900` | `current` only: each tick item (subscription
replacement, roster, unread repair) waits this long since it last ran. |
| `renewal_secs` | `0` | Identity renewal period. `0` disables renewal,
and the first renewal is jittered. |
A desktop human keeps the basic human's presence, typing, messages and
reactions. What it adds:
- **Cold start.** It reads NIP-11 and opens its profile and membership
subscriptions. Its channel subscriptions are batched per mode:
- up to 10 channels: one replaying batch;
- `legacy` with more than 10: one per-channel batch per 10 channels;
- `current` with more than 10: a live-only subscription per 2048
channels, plus replays of 10 that are closed at EOSE.
It also makes the app's HTTP reads: roster, channel names, sidebar,
profiles, statuses, emoji, read state, unread repair, reminders, window
catch-up plus notices for each window, and root plus thread page for
each thread. A window catch-up follows the cursor in the relay's window
bounds (kind 39006) for up to five older pages, until the bounds say
history is exhausted.
- **Route establishment.** `legacy` makes every read as it connects.
`current` reads NIP-11, names and sidebar up front, and the rest as each
subscription reaches EOSE, by its replay outcome. A replay of 500
events, a missed deadline, or the live-only batch over 10 channels is a
cut, and so is a route the previous connection never caught up. Only an
EOSE catches a route up: one recovered after a missed deadline cannot be
resumed or carried by the next connection.
- A continuous profile route skips its read (unless one is owed).
- A continuous membership route only repairs unreads, without the
read-state snapshot, unless a snapshot read failed since; then it runs
the full repair.
- A cut global reads in full, and channel routes always recover windows
and threads.
- **Ticks and renewals.** These follow each mode:
- `legacy`: every tick replaces subscriptions and rereads roster and
unreads, and a renewal is a full reconnect;
- `current`: tick items wait for the backstop, measured from the last
replacement, the last successful roster read, and the last full unread
repair. A failed roster read, unread repair or read-state snapshot is
owed and retries on the next tick; snapshot debt is tracked apart from
repair debt, so a repair without the snapshot never settles it. A
renewal carries its routes: nothing is read while every replay keeps
continuity, and once one is cut every route establishes as on a
reconnect. With more than 10 channels the live-only batch is always cut,
so the channels recover and unreads are repaired, but globals are not
reread.
- **Reporting.** Each request type is its own op (`desktop_roster`,
`desktop_unread`, `desktop_replay_eose`, …).
- **Bounds.** At most 4 REQs set up at once, with a 10 s EOSE timeout. A
tick replacement that misses it is closed, the batch it would have
replaced stays, and the next tick retries. At most 6 HTTP reads in
flight, with a 30 s timeout. A read that comes due while the same read
is running or queued is not added again, and each read leaves the
running set as soon as it settles, so a failed read can retry while a
slow one is still under way. Connects and failed sessions back off. A
stop abandons pending EOSE waits without counting them as failures.
- **Not modeled.** The app also establishes channel routes when a tick
replacement reaches EOSE; the generator does not.
`perf/LOADGEN.md` documents the knobs and the model.
`perf/scenarios/local-desktop.toml` is a short local example with every
knob on.
## What got simpler
- `human()`'s schedule loop (presence, typing, messages, reactions,
session end) moved into one `Activity`. The basic and desktop humans
share it, so neither client duplicates the schedule. The basic human's
random draw order is unchanged.
- Membership stays in one place. `member_channels` remains the single
assignment function, and `channel_members` derives from it rather than
from a second formula, so seeding and the clients can't drift apart.
The desktop model is a new concept: about 1,420 lines of model in its
own module, plus about 1,300 lines of tests. Everything else goes
through existing seams: `Stats`/`Op`, `connect`, `Backoff`, `observe`,
`roots` and `Net`.
Identified debt, not fixed here: CI's `test-unit` runs `buzz-test-client
--lib` with the filter `loadgen::`, so the five crate-root `tests::` in
`lib.rs` (relay message parsing, auth and text event building,
`collect_until_eose`) never run in CI. That gap predates this PR, which
doesn't touch `lib.rs`.
## Testing
- `cargo fmt --check`, `cargo clippy -p buzz-test-client --all-targets
-- -D warnings`, and `cargo test -p buzz-test-client` (plain parallel):
84 lib tests pass.
- Wide activity: a seeded `Rng` over 100k picks gives each share within
1%, and an unset share makes the identical draws to before. A run
against a loopback relay checks that a human's typing lands only in its
regular channels at share 0, and only in its wide channels at share 1.
- Scenario: validation of every new key; deterministic wide-channel and
tier assignment that honors the fractions; `channel_members` inverts the
assignment up to the peak; the default assignment is unchanged; the
shipped scenarios parse.
- Seed: with private channels, a fake relay sees the following, in
order:
- every 9007 before any 9000, and every 9000 before any post;
- owner signatures;
- ≤100 members per event;
- exactly the expected members per channel;
- posts authored by channel members.
A refused member fails the seed, and open channels add nobody.
- Desktop: request planning, window bounds validation, route
establishment, and tick and owed-read rules are unit-tested. End-to-end
tests drive clients against loopback WS and HTTP fakes. The WS fake
replays a planned number of events per subscription and can delay an
EOSE or drop a connection. The HTTP fake answers window pages with
relay-signed bounds, an older aux event and equal-timestamp rows,
refuses unknown cursors, and can fail a read once. They check:
- exact cold-start counts per op for `legacy` and `current` at 5 and 25
channels;
- a window catch-up sends the head, then exactly the bounds' cursor
(`until` + `before_id` of the last row, not the older aux event), and
stops when the bounds say history is exhausted even though that page
came back full;
- after a `current` renewal, 499 replayed events everywhere read nothing
over HTTP; 500 on the channel batch adds channel recovery plus a covered
unread repair; 500 on membership adds roster, read state, unread and
reminders plus channel recovery; 500 on profiles adds profiles, statuses
and emoji plus channel recovery and a covered repair;
- after a dropped connection, complete global replays read only NIP-11,
channel recovery and a covered repair; truncated ones reread both
globals in full;
- with renewals shorter than the backstop, read-state snapshots still
recur at the backstop;
- a tick replacement past its EOSE deadline is closed, the old batch
stays, the next tick replaces it, and one batch wire stays open;
- a failed roster and unread read retry on the next tick, and successful
reads do not run again;
- a failed roster read retries on the next tick while an unrelated slow
read is still running;
- after a failed snapshot, a continuous membership establishment before
any tick still reads the snapshot;
- a route whose EOSE missed its deadline is recovered, and after a
renewal with a complete replay it is recovered in full again rather than
carried;
- `legacy` ticks replace batches (new REQ before the CLOSE of the old
one) and reread;
- `current` ticks do nothing until the backstop, then replace live-only;
- `legacy` renewals are full reconnects;
- `current` renewals carry small sessions without rereading globals or
channels, and recover channels in large ones.
Each of the nine review-fix tests was checked against a mutant that
restores the old behavior, and every mutant fails its test.
- Smoke: a local relay with Postgres and Redis. I ran `seed` twice, then
`run` of `local-desktop.toml` (60 s) once per mode.
- Both seeds were accepted (780 of 780 posts each), including private
channels and memberships for 340 people. Neither run aborted, no client
was cancelled, and no op recorded an error.
- Each run used 18 desktop clients:
| op | `current` | `legacy` |
|---|---|---|
| `desktop_live_eose` | 266 | 483 |
| `desktop_replay_eose` | 24 | 0 |
| `desktop_roster` | 72 | 289 |
| `desktop_read_state` / `desktop_unread` | 72 / 99 | 289 / 289 |
| `desktop_profiles` | 18 | 73 |
| `desktop_nip11` | 18 | 73 |
| `desktop_window` | 372 | 596 |
Window reads are below the first smoke (406 and 1086) because catch-ups
stop where the relay's bounds say history is exhausted, rather than
paging while pages came back full. They are above the previous revision
(179 and 470) because each read now leaves the running set as soon as it
settles, so a channel recovery is no longer dropped as a duplicate of a
read that already finished inside a still-running batch.
- The first smoke caught two relay contract mismatches that the fakes
had missed:
- top-level window pages need `before_id` with `until`;
- the read-state snapshot answers with an object.
Both are fixed, and the fakes now enforce both.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Luke Tornquist
|
83bd8b1cae | fix(loadgen): sign public URLs when dialling connect_addr in plaintext (#8926) | ||
|
Luke Tornquist
|
4cf2a3bd66 |
feat(loadgen): add a hot-thread stressor and teardown (#8746)
## Summary `buzz-loadgen` spreads load evenly: every channel gets the same history and the same short thread chains. That never puts heavy load on a single thread. A busy conversation, such as agents talking to each other, can give one root a huge number of replies, and thread-summary work grows with it (see #8635). This PR adds an optional `[hot_thread]` scenario section so a load test can measure that case. - **Seed:** `seed` posts a hot root in community 0, channel 0, then `seed_replies` replies directly under it. The root comes from a dedicated identity that posts nothing else, so `run` finds it however many other roots the channel has. The replies go out concurrently, like the rest of the seed, so the run starts with the thread already large. - **Run:** `authors` hot-thread clients are scaled by each ramp step's `users`, the same way humans and agents are. Each one replies to the hot root over HTTP `/events` and reads the thread's newest 100 replies over HTTP `/query`, at `replies_per_hour` and `fetches_per_hour`. - **Metrics:** two new report ops, `hot_reply` and `thread_fetch`. The replies are timed, so humans subscribed to that channel also record delivery latency for them. - **No root, failed run:** an author that can't find the seeded root counts a `thread_fetch` error on every attempt, so a run without the hot thread fails instead of quietly measuring nothing. - **Off by default:** the section is disabled while `seed_replies` and `authors` are both 0, so existing scenarios are unchanged. `perf/scenarios/local-smoke.toml` gets a small hot thread, and `perf/LOADGEN.md` documents the knobs. ### Teardown The PR also adds `buzz-loadgen teardown <scenario>`, so a staging fixture can be removed after the load test. - For each community from `target.community_host_template`, it first calls `POST /operator/communities/archive` as the fixture owner, using the operator key `seed` uses. It then calls `POST /operator/communities/delete`. - The deletion `request_id` is derived from `run_id` and the host, so a re-run replays the same requests. That retries failures and reports progress without a `deletion_request_conflict`. - The relay's deletion drain does the actual deletion asynchronously. - If the fixture owner does not own a host, the host is reported `absent` and left untouched. - A scenario without a host template is refused. - Deleted hosts stay as tombstones, so `LOADGEN.md` says each fixture needs a fresh `run_id`. The signing has to happen as each derived owner, which is why this lives in the generator rather than in deployment config. ## What got simpler Nothing got simpler; this adds a scale vector. To keep it small, the PR reuses what's already there: - the seeder's post plan, plus `find_roots` with a content prefix - the existing HTTP helper, whose auth tag is now optional - humans' delivery measurement A separate chain-depth knob and dedicated viewer clients were left out. `shape.thread_depth` already seeds chains up to the relay's depth cap, and humans in the hot channel already receive the replies. ## Testing - `loadgen::seed::tests::the_hot_thread_seeds_direct_replies_under_its_own_root` checks that the hot root and its replies are seeded, that every reply is direct (depth 1), and that a rejected root fails the seed. - `loadgen::run::tests::hot_thread_authors_reply_to_the_hot_root_and_fail_without_it` checks several things: - authors post replies tagged to the hot root; - root lookups query only the hot-root identity, and a newer reply-shaped decoy is never taken as the root; - scheduled reads all target the hot root's channel and id; - the run fails when the root isn't seeded. - `loadgen::run::tests::hot_thread_authors_start_nothing_once_the_run_stops` runs 8 hot authors directly. It holds every in-flight `/events` reply at a gate until the stop is signalled, so each author's next reply is already overdue when the gate opens. It then checks that exactly 8 replies were posted, none started after the stop. Removing the post-`select!` stop check fails the test, and the gate is released before any assertion, so a failure can't hang the job. - The run now signals the stop before the last step's snapshot instead of after building that step's report. Under a loaded parallel test run, work started in that gap leaked into `shutdown`. - `loadgen::teardown::tests::teardown_deletes_only_the_fixture_owners_communities_and_replays_on_rerun` runs teardown twice against a fake operator that checks the owner, requires archive before delete, and conflicts on a different request id. It checks that: - the owned host is archived and deletion-requested; - another owner's host and an unseeded host are reported absent and left untouched; - a 503 on archive is reported as a failure; - the second pass converges to the same result; - a deletion already past membership purge, where archive returns 404, is replayed and reported at its stage under `requested`; - the same run under another host template gets its own request ids: another owner's host is reported absent rather than conflicting with a request id already used. If the request id is replaced with a fresh one per run, the test fails. - `loadgen::teardown::tests::teardown_refuses_a_scenario_without_templated_communities` - All of these run in the existing `loadgen::` unit lane. - Teardown was run end to end at `28192004` against a local relay. The relay had operator keys configured, local S3 behind it, and `buzz-admin deletions drain`. The run covered: - seeding three templated communities; - teardown, with every fixture archived and a deletion request at its derived id; - a re-run that converged; - the drain, which tombstoned the fixtures; - teardown again after the purge, which reported each fixture at `retention_pending` and exited 0; - a re-seed of the torn-down run, which failed at provisioning. A community outside the fixture stayed active, with no deletion request. - At `28192004`, `cargo fmt --check`, `cargo clippy -p buzz-test-client --all-targets -D warnings` and six parallel runs of `cargo test -p buzz-test-client` pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Luke Tornquist
|
ef0fce8c51 | ci(loadgen): publish buzz-loadgen as ghcr.io/block/buzz-loadgen (#8717) | ||
|
Luke Tornquist
|
aed4be8592 |
Add a scenario-driven relay load generator (#8636)
## Summary Buzz has no way to load-test the relay as a whole. `wamp_bench` sends kind:9 messages from one identity to one channel, and `perf/relay_bus_scaling.py` only measures Redis fan-out. Neither can show what happens to a relay with many more users, channels, or deep threads. This PR adds `buzz-loadgen`, a load generator in `crates/buzz-test-client`: - `buzz-loadgen seed <scenario.toml>` creates communities (through `POST /operator/communities`), channels, channel history, and deep threads. - `buzz-loadgen run <scenario.toml>` drives simulated humans and agents through ramp steps (1x, 2x, ...) and prints p50/p90/p99 latency, errors, and rate-limit rejections per operation for each step. - Humans hold a WebSocket with live subscriptions, fetch history on connect, and send presence, typing, messages, thread replies, and reactions. Agents authenticate with a NIP-OA tag, poll HTTP `/query`, post through HTTP `/events`, and send encrypted observer frames to their owner. - Per-client rates come from the scenario. Set them to model a deployment, and a step of `users = 10` means 10x that deployment's population. - Every identity, host, and channel id comes from the scenario's `run_id`, so `--shard i/n` splits a population across machines with no coordination. - Each step measures from ramp start to hold end, so connect failures during the ramp count. A step aborts the run when an operation's error rate or p99 (delivery included) crosses the scenario's `[abort]` limits, when a relay `CLOSED`s a live subscription often enough, or when a client exits unexpectedly. The report names the limit that tripped. - Each connect plus authentication has a deadline (`target.connect_timeout_secs`), and shutdown is bounded, so a relay that stalls connections still produces a report and a non-zero exit. - `seed` exits non-zero unless the relay accepted every planned post. - `NostrWsConnection::connect_to_addr` opens the TCP connection to a given address while the URL still names the host. This lets a client inside the relay's cluster reach a community host through the relay's service address. - `wamp_bench` is deleted. Its paced single-channel load is a subset of a scenario. What got simpler: one load tool instead of two partial ones. The change touches only test tooling, the unit-test lane, and one additive method in `buzz-ws-client`. Relay code is unchanged. ### Related issue None found. ### Testing Smoke run on a Linux workstation against a local relay built from this branch, with its own Postgres, Redis, and MinIO containers: ```bash cargo run --release -p buzz-test-client --bin buzz-loadgen -- seed perf/scenarios/local-smoke.toml cargo run --release -p buzz-test-client --bin buzz-loadgen -- run perf/scenarios/local-smoke.toml ``` - `seed`: 710 of 710 posts accepted (10 channels, 200 history messages, 10 thread roots, 500 thread replies). - `run`: both steps (1x = 50 humans + 5 agents, then 2x) completed with 0 errors and 0 rate-limit rejections. Every operation recorded traffic: 55/55 connect/auth, 205/205 live subscriptions, 150 history fetches, presence/typing/observer frames, messages and reactions, HTTP `/query`, HTTP `/events`, and publish→delivery. Delivery p99 was 45 ms at 1x and 42 ms at 2x. - Failure paths are covered by unit tests: a listener that accepts TCP but never answers (connect deadline, then `run` aborts and returns), a fake `/events` server that rejects posts (`seed` fails), `CLOSED` subscriptions, and invalid scenarios. `just test-unit` (and its `scripts/run-tests.sh` fallback) now runs the new `loadgen::` unit tests. Before this change, no CI lane ran `buzz-test-client` lib tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Tyler
|
14fba21e57 |
Multi-tenant Buzz relay: community_id as a server-resolved key (comprehensive rewrite) (#1321)
Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com> Signed-off-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co> Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Quinn <96f056ad5f2305c8ddf637dc65d048aa4c12d7daeb8867690e34fca46b0ef64c@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Dawn <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Co-authored-by: Sami <sami@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co> |