jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-11-01 15:01:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,118 Commits 1,288 Branches 247 Tags
6e6f4e8a0de2ee85c4a6366ecdb15e1fad7fe429
Commit Graph

4 Commits

Author SHA1 Message Date
Luke Tornquist
4cf2a3bd66 feat(loadgen): add a hot-thread stressor and teardown (#8746) 
## Summary

`buzz-loadgen` spreads load evenly: every channel gets the same history
and the same short thread chains. That never puts heavy load on a single
thread. A busy conversation, such as agents talking to each other, can
give one root a huge number of replies, and thread-summary work grows
with it (see #8635). This PR adds an optional `[hot_thread]` scenario
section so a load test can measure that case.

- **Seed:** `seed` posts a hot root in community 0, channel 0, then
`seed_replies` replies directly under it. The root comes from a
dedicated identity that posts nothing else, so `run` finds it however
many other roots the channel has. The replies go out concurrently, like
the rest of the seed, so the run starts with the thread already large.
- **Run:** `authors` hot-thread clients are scaled by each ramp step's
`users`, the same way humans and agents are. Each one replies to the hot
root over HTTP `/events` and reads the thread's newest 100 replies over
HTTP `/query`, at `replies_per_hour` and `fetches_per_hour`.
- **Metrics:** two new report ops, `hot_reply` and `thread_fetch`. The
replies are timed, so humans subscribed to that channel also record
delivery latency for them.
- **No root, failed run:** an author that can't find the seeded root
counts a `thread_fetch` error on every attempt, so a run without the hot
thread fails instead of quietly measuring nothing.
- **Off by default:** the section is disabled while `seed_replies` and
`authors` are both 0, so existing scenarios are unchanged.

`perf/scenarios/local-smoke.toml` gets a small hot thread, and
`perf/LOADGEN.md` documents the knobs.

### Teardown

The PR also adds `buzz-loadgen teardown <scenario>`, so a staging
fixture can be removed after the load test.

- For each community from `target.community_host_template`, it first
calls `POST /operator/communities/archive` as the fixture owner, using
the operator key `seed` uses. It then calls `POST
/operator/communities/delete`.
- The deletion `request_id` is derived from `run_id` and the host, so a
re-run replays the same requests. That retries failures and reports
progress without a `deletion_request_conflict`.
- The relay's deletion drain does the actual deletion asynchronously.
- If the fixture owner does not own a host, the host is reported
`absent` and left untouched.
- A scenario without a host template is refused.
- Deleted hosts stay as tombstones, so `LOADGEN.md` says each fixture
needs a fresh `run_id`.

The signing has to happen as each derived owner, which is why this lives
in the generator rather than in deployment config.

## What got simpler

Nothing got simpler; this adds a scale vector. To keep it small, the PR
reuses what's already there:
- the seeder's post plan, plus `find_roots` with a content prefix
- the existing HTTP helper, whose auth tag is now optional
- humans' delivery measurement

A separate chain-depth knob and dedicated viewer clients were left out.
`shape.thread_depth` already seeds chains up to the relay's depth cap,
and humans in the hot channel already receive the replies.

## Testing

-
`loadgen::seed::tests::the_hot_thread_seeds_direct_replies_under_its_own_root`
checks that the hot root and its replies are seeded, that every reply is
direct (depth 1), and that a rejected root fails the seed.
-
`loadgen::run::tests::hot_thread_authors_reply_to_the_hot_root_and_fail_without_it`
checks several things:
  - authors post replies tagged to the hot root;
- root lookups query only the hot-root identity, and a newer
reply-shaped decoy is never taken as the root;
  - scheduled reads all target the hot root's channel and id;
  - the run fails when the root isn't seeded.
-
`loadgen::run::tests::hot_thread_authors_start_nothing_once_the_run_stops`
runs 8 hot authors directly. It holds every in-flight `/events` reply at
a gate until the stop is signalled, so each author's next reply is
already overdue when the gate opens. It then checks that exactly 8
replies were posted, none started after the stop. Removing the
post-`select!` stop check fails the test, and the gate is released
before any assertion, so a failure can't hang the job.
- The run now signals the stop before the last step's snapshot instead
of after building that step's report. Under a loaded parallel test run,
work started in that gap leaked into `shutdown`.
-
`loadgen::teardown::tests::teardown_deletes_only_the_fixture_owners_communities_and_replays_on_rerun`
runs teardown twice against a fake operator that checks the owner,
requires archive before delete, and conflicts on a different request id.
It checks that:
  - the owned host is archived and deletion-requested;
- another owner's host and an unseeded host are reported absent and left
untouched;
  - a 503 on archive is reported as a failure;
  - the second pass converges to the same result;
- a deletion already past membership purge, where archive returns 404,
is replayed and reported at its stage under `requested`;
- the same run under another host template gets its own request ids:
another owner's host is reported absent rather than conflicting with a
request id already used.

If the request id is replaced with a fresh one per run, the test fails.
-
`loadgen::teardown::tests::teardown_refuses_a_scenario_without_templated_communities`
- All of these run in the existing `loadgen::` unit lane.
- Teardown was run end to end at `28192004` against a local relay. The
relay had operator keys configured, local S3 behind it, and `buzz-admin
deletions drain`. The run covered:
  - seeding three templated communities;
- teardown, with every fixture archived and a deletion request at its
derived id;
  - a re-run that converged;
  - the drain, which tombstoned the fixtures;
- teardown again after the purge, which reported each fixture at
`retention_pending` and exited 0;
  - a re-seed of the torn-down run, which failed at provisioning.

A community outside the fixture stayed active, with no deletion request.
- At `28192004`, `cargo fmt --check`, `cargo clippy -p buzz-test-client
--all-targets -D warnings` and six parallel runs of `cargo test -p
buzz-test-client` pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-30 13:52:30 -04:00
Luke Tornquist
ef0fce8c51 ci(loadgen): publish buzz-loadgen as ghcr.io/block/buzz-loadgen (#8717) 2026-10-29 20:33:10 -04:00
Luke Tornquist
aed4be8592 Add a scenario-driven relay load generator (#8636) 
## Summary

Buzz has no way to load-test the relay as a whole. `wamp_bench` sends
kind:9 messages from one identity to one channel, and
`perf/relay_bus_scaling.py` only measures Redis fan-out. Neither can
show what happens to a relay with many more users, channels, or deep
threads.

This PR adds `buzz-loadgen`, a load generator in
`crates/buzz-test-client`:

- `buzz-loadgen seed <scenario.toml>` creates communities (through `POST
/operator/communities`), channels, channel history, and deep threads.
- `buzz-loadgen run <scenario.toml>` drives simulated humans and agents
through ramp steps (1x, 2x, ...) and prints p50/p90/p99 latency, errors,
and rate-limit rejections per operation for each step.
- Humans hold a WebSocket with live subscriptions, fetch history on
connect, and send presence, typing, messages, thread replies, and
reactions. Agents authenticate with a NIP-OA tag, poll HTTP `/query`,
post through HTTP `/events`, and send encrypted observer frames to their
owner.
- Per-client rates come from the scenario. Set them to model a
deployment, and a step of `users = 10` means 10x that deployment's
population.
- Every identity, host, and channel id comes from the scenario's
`run_id`, so `--shard i/n` splits a population across machines with no
coordination.
- Each step measures from ramp start to hold end, so connect failures
during the ramp count. A step aborts the run when an operation's error
rate or p99 (delivery included) crosses the scenario's `[abort]` limits,
when a relay `CLOSED`s a live subscription often enough, or when a
client exits unexpectedly. The report names the limit that tripped.
- Each connect plus authentication has a deadline
(`target.connect_timeout_secs`), and shutdown is bounded, so a relay
that stalls connections still produces a report and a non-zero exit.
- `seed` exits non-zero unless the relay accepted every planned post.
- `NostrWsConnection::connect_to_addr` opens the TCP connection to a
given address while the URL still names the host. This lets a client
inside the relay's cluster reach a community host through the relay's
service address.
- `wamp_bench` is deleted. Its paced single-channel load is a subset of
a scenario.

What got simpler: one load tool instead of two partial ones. The change
touches only test tooling, the unit-test lane, and one additive method
in `buzz-ws-client`. Relay code is unchanged.

### Related issue

None found.

### Testing

Smoke run on a Linux workstation against a local relay built from this
branch, with its own Postgres, Redis, and MinIO containers:

```bash
cargo run --release -p buzz-test-client --bin buzz-loadgen -- seed perf/scenarios/local-smoke.toml
cargo run --release -p buzz-test-client --bin buzz-loadgen -- run perf/scenarios/local-smoke.toml
```

- `seed`: 710 of 710 posts accepted (10 channels, 200 history messages,
10 thread roots, 500 thread replies).
- `run`: both steps (1x = 50 humans + 5 agents, then 2x) completed with
0 errors and 0 rate-limit rejections. Every operation recorded traffic:
55/55 connect/auth, 205/205 live subscriptions, 150 history fetches,
presence/typing/observer frames, messages and reactions, HTTP `/query`,
HTTP `/events`, and publish→delivery. Delivery p99 was 45 ms at 1x and
42 ms at 2x.
- Failure paths are covered by unit tests: a listener that accepts TCP
but never answers (connect deadline, then `run` aborts and returns), a
fake `/events` server that rejects posts (`seed` fails), `CLOSED`
subscriptions, and invalid scenarios.

`just test-unit` (and its `scripts/run-tests.sh` fallback) now runs the
new `loadgen::` unit tests. Before this change, no CI lane ran
`buzz-test-client` lib tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-29 13:49:37 -04:00
Tyler
14fba21e57 Multi-tenant Buzz relay: community_id as a server-resolved key (comprehensive rewrite) (#1321) 
Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com>
Signed-off-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Quinn <96f056ad5f2305c8ddf637dc65d048aa4c12d7daeb8867690e34fca46b0ef64c@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Dawn <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: Sami <sami@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
 2026-06-29 12:39:02 -04:00