jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-11-01 23:11:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
cfcaef4e9fb562ff0acd4ed749464bdd733fdfc8
buzz/perf/LOADGEN.md
Luke Tornquist 4cf2a3bd66 feat(loadgen): add a hot-thread stressor and teardown (#8746) 
## Summary

`buzz-loadgen` spreads load evenly: every channel gets the same history
and the same short thread chains. That never puts heavy load on a single
thread. A busy conversation, such as agents talking to each other, can
give one root a huge number of replies, and thread-summary work grows
with it (see #8635). This PR adds an optional `[hot_thread]` scenario
section so a load test can measure that case.

- **Seed:** `seed` posts a hot root in community 0, channel 0, then
`seed_replies` replies directly under it. The root comes from a
dedicated identity that posts nothing else, so `run` finds it however
many other roots the channel has. The replies go out concurrently, like
the rest of the seed, so the run starts with the thread already large.
- **Run:** `authors` hot-thread clients are scaled by each ramp step's
`users`, the same way humans and agents are. Each one replies to the hot
root over HTTP `/events` and reads the thread's newest 100 replies over
HTTP `/query`, at `replies_per_hour` and `fetches_per_hour`.
- **Metrics:** two new report ops, `hot_reply` and `thread_fetch`. The
replies are timed, so humans subscribed to that channel also record
delivery latency for them.
- **No root, failed run:** an author that can't find the seeded root
counts a `thread_fetch` error on every attempt, so a run without the hot
thread fails instead of quietly measuring nothing.
- **Off by default:** the section is disabled while `seed_replies` and
`authors` are both 0, so existing scenarios are unchanged.

`perf/scenarios/local-smoke.toml` gets a small hot thread, and
`perf/LOADGEN.md` documents the knobs.

### Teardown

The PR also adds `buzz-loadgen teardown <scenario>`, so a staging
fixture can be removed after the load test.

- For each community from `target.community_host_template`, it first
calls `POST /operator/communities/archive` as the fixture owner, using
the operator key `seed` uses. It then calls `POST
/operator/communities/delete`.
- The deletion `request_id` is derived from `run_id` and the host, so a
re-run replays the same requests. That retries failures and reports
progress without a `deletion_request_conflict`.
- The relay's deletion drain does the actual deletion asynchronously.
- If the fixture owner does not own a host, the host is reported
`absent` and left untouched.
- A scenario without a host template is refused.
- Deleted hosts stay as tombstones, so `LOADGEN.md` says each fixture
needs a fresh `run_id`.

The signing has to happen as each derived owner, which is why this lives
in the generator rather than in deployment config.

## What got simpler

Nothing got simpler; this adds a scale vector. To keep it small, the PR
reuses what's already there:
- the seeder's post plan, plus `find_roots` with a content prefix
- the existing HTTP helper, whose auth tag is now optional
- humans' delivery measurement

A separate chain-depth knob and dedicated viewer clients were left out.
`shape.thread_depth` already seeds chains up to the relay's depth cap,
and humans in the hot channel already receive the replies.

## Testing

-
`loadgen::seed::tests::the_hot_thread_seeds_direct_replies_under_its_own_root`
checks that the hot root and its replies are seeded, that every reply is
direct (depth 1), and that a rejected root fails the seed.
-
`loadgen::run::tests::hot_thread_authors_reply_to_the_hot_root_and_fail_without_it`
checks several things:
  - authors post replies tagged to the hot root;
- root lookups query only the hot-root identity, and a newer
reply-shaped decoy is never taken as the root;
  - scheduled reads all target the hot root's channel and id;
  - the run fails when the root isn't seeded.
-
`loadgen::run::tests::hot_thread_authors_start_nothing_once_the_run_stops`
runs 8 hot authors directly. It holds every in-flight `/events` reply at
a gate until the stop is signalled, so each author's next reply is
already overdue when the gate opens. It then checks that exactly 8
replies were posted, none started after the stop. Removing the
post-`select!` stop check fails the test, and the gate is released
before any assertion, so a failure can't hang the job.
- The run now signals the stop before the last step's snapshot instead
of after building that step's report. Under a loaded parallel test run,
work started in that gap leaked into `shutdown`.
-
`loadgen::teardown::tests::teardown_deletes_only_the_fixture_owners_communities_and_replays_on_rerun`
runs teardown twice against a fake operator that checks the owner,
requires archive before delete, and conflicts on a different request id.
It checks that:
  - the owned host is archived and deletion-requested;
- another owner's host and an unseeded host are reported absent and left
untouched;
  - a 503 on archive is reported as a failure;
  - the second pass converges to the same result;
- a deletion already past membership purge, where archive returns 404,
is replayed and reported at its stage under `requested`;
- the same run under another host template gets its own request ids:
another owner's host is reported absent rather than conflicting with a
request id already used.

If the request id is replaced with a fresh one per run, the test fails.
-
`loadgen::teardown::tests::teardown_refuses_a_scenario_without_templated_communities`
- All of these run in the existing `loadgen::` unit lane.
- Teardown was run end to end at `28192004` against a local relay. The
relay had operator keys configured, local S3 behind it, and `buzz-admin
deletions drain`. The run covered:
  - seeding three templated communities;
- teardown, with every fixture archived and a deletion request at its
derived id;
  - a re-run that converged;
  - the drain, which tombstoned the fixtures;
- teardown again after the purge, which reported each fixture at
`retention_pending` and exited 0;
  - a re-seed of the torn-down run, which failed at provisioning.

A community outside the fixture stayed active, with no deletion request.
- At `28192004`, `cargo fmt --check`, `cargo clippy -p buzz-test-client
--all-targets -D warnings` and six parallel runs of `cargo test -p
buzz-test-client` pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 13:52:30 -04:00

167 lines
8.8 KiB
Markdown
Raw Blame History

# Relay load generator
`buzz-loadgen` (in `crates/buzz-test-client`) seeds a data shape and then
drives a population of simulated humans and agents through a ramp of user
multipliers, reporting latency percentiles, errors and rate-limit rejections
per operation for each step.
```bash
cargo run --release -p buzz-test-client --bin buzz-loadgen -- seed perf/scenarios/local-smoke.toml
cargo run --release -p buzz-test-client --bin buzz-loadgen -- run perf/scenarios/local-smoke.toml
cargo run --release -p buzz-test-client --bin buzz-loadgen -- teardown perf/scenarios/<templated>.toml
```
## Scale vectors
| Vector | Scenario knob |
|---|---|
| Communities | `shape.communities` (each provisioned through `POST /operator/communities`) |
| Channels | `shape.channels_per_community`, `population.channels_per_member` |
| History depth | `shape.history_messages_per_channel`, `human.history_limit` |
| Deep threads | `shape.thread_roots_per_channel`, `shape.thread_depth`, `human.thread_reply_fraction` |
| One hot thread | `[hot_thread]`: `seed_replies`, `authors`, `replies_per_hour`, `fetches_per_hour` |
| Humans / agents | `population.humans`, `population.agents`, multiplied by each `[[ramp]]` step's `users` |
| Throughput per client | `[human]` and `[agent]` rates |
| Reconnect churn | `human.mean_session_secs`, `population.connect_rate_per_sec` |
`shape.thread_depth` seeds each thread as a chain: reply `k` replies to reply
`k-1`, so the thread's deepest reply sits at depth `thread_depth`. The relay
rejects depths above 100, so scenarios do too. Threads are seeded in parallel
with each other.
A scenario that cannot be scheduled is rejected when it is parsed: for
example `seed_concurrency` of 0, `human.presence_interval_secs` below 1, or a
`community_host_template` with a port while `target.connect_addr` is set (the
override maps hostnames, so HTTP would ignore it).
`[hot_thread]` models one thread that keeps growing, such as a conversation
between busy agents. `seed` posts its root in community 0, channel 0 and
then `seed_replies` replies directly under it, concurrently. During `run`,
`authors` clients (multiplied by each ramp step's `users`, like humans and
agents) reply to that root and read its newest 100 replies over HTTP. Their
replies are timed, so humans subscribed to that channel record delivery.
Leave `seed_replies` and `authors` at 0 to disable it.
## What each client does
- **Human**: authenticates over WebSocket and subscribes to its channels'
messages, its mentions, typing, and ten peers' presence. On connect it
fetches `history_limit` messages per channel. Then it sends presence
heartbeats, typing indicators, messages (some as replies into seeded deep
threads) and reactions on Poisson schedules.
- **Agent**: authenticates with a NIP-OA tag from its owner, polls HTTP
`/query` (channel window plus mentions), posts messages through HTTP
`/events`, and sends encrypted kind:24200 observer frames to its owner.
- **Hot-thread author**: finds the seeded hot root, then posts replies to it
through HTTP `/events` (`hot_reply`) and reads the thread through HTTP
`/query` (`thread_fetch`). Until the root is found every attempt counts as
a `thread_fetch` error, so a run without the seeded thread fails.
Messages carry their send time, so every subscriber records publish→delivery
latency (`deliver`). Across machines this depends on clock sync.
## Modeling a deployment
The `[human]` and `[agent]` defaults are round placeholders. To model a known
deployment, divide its hourly event counts by its client counts and set the
results as the scenario's rates. Then a ramp step's `users` reads as a
multiple of that deployment's population. Keep such scenarios with that
deployment's own configuration.
## Targeting and membership
`target.connect_addr` (`host:port`) applies only together with
`target.community_host_template`: it then replaces host resolution for the
community hosts, over both WebSocket and HTTP, while TLS names, `Host` headers
and the NIP-42 relay tag still name the community. Without a host
template, the generator connects to `ws_url` and `http_url` as written and
`connect_addr` has no effect.
Seeded identities do not join communities or channels. A relay that requires
membership rejects both `seed` and `run`.
## Sharding
`run --shard i/n` runs the clients whose global index `g` satisfies
`g % n == i`. Identities, community hosts and channel ids derive from
`run_id`, so shards need no coordination: run `seed` once, then start `n`
processes, such as a Kubernetes Indexed Job, with the same scenario. Each
shard reports its own clients; add counts and take the worst percentiles
across shards.
`ghcr.io/block/buzz-loadgen:main` (and `:sha-<7>`) packages the binary as the
image entrypoint, published by `.github/workflows/loadgen-image.yml` from
`Dockerfile.loadgen`. Mount the scenario and pass the subcommand as the
container arguments, for example `run /etc/loadgen/scenario.toml --shard
$(JOB_COMPLETION_INDEX)/30`. Pin `:sha-<7>` or a digest for runs you intend
to compare; `:main` moves. The container runs as UID 10001 with `/` as its
working directory and no home, so point `--report` at a writable mount.
## Abort
Each step reports every operation from the start of its ramp to the end of
its hold, so connection failures during the ramp count. At the end of the
hold, the run stops if any operation's p99 (publish→delivery included)
exceeds `abort.max_p99_ms`, if any operation other than delivery has an
error rate above `abort.max_error_rate`, or if a client exited unexpectedly.
An operation still in flight when the hold ends is reported as `pending`
but not judged: it has not failed yet. A relay `CLOSED` on a live
subscription counts as a `subscribe_eose` error and ends that client's
session. Each connect plus authentication must finish within
`target.connect_timeout_secs` (default 10). Quota rejections, meaning an HTTP 429 or the relay's
`rate-limited: quota exceeded` text, are reported as `throttled`, not errors.
Every other rejection, including `rate-limited: shared admission unavailable`
(the relay's admission store is down), counts as an error.
Ramp steps must increase `users`, since clients are only ever added.
When the run stops, clients get 15 seconds to settle what they have in flight
and disconnect. An operation already under way runs to its own deadline
instead of being abandoned: a connect attempt finishes within
`target.connect_timeout_secs`, and a publish waits at most 30 seconds for its
`OK`. A stalled connect therefore ends as a `connect timeout` error and a
healthy one as a success. What settles after the last step is reported in the
`shutdown` object (`ops`, `failures`, `abort_reason`) and judged by the same
limits as a step, so a connect that never completes fails the run there. A client
still busy when the grace period ends is cancelled; its in-flight operations
are recorded as `unfinished at stop` errors, and the cancellation fails the run
whatever the limits. Keep `target.connect_timeout_secs` below 15, or a stalled
connect is cancelled instead of timing out. Publishes, history fetches and
HTTP requests can outlast the grace period (30 s, 30 s and 60 s deadlines), in
which case the client is cancelled and the run fails.
The process exits non-zero when a step aborts, a client panics, a client had to
be cancelled, or `shutdown.abort_reason` is set.
`seed` exits non-zero unless the relay accepted every planned post.
`seed` provisions each community with `create_only`, so it never rotates the
owner of an existing host. A host that already exists is reused only when the
fixture owner derived from `run_id` owns it (re-running `seed` for the same
scenario); otherwise `seed` fails. Use a fresh `run_id` or a host template
with `{run}` to avoid colliding with another community.
## Teardown
`teardown <scenario>` removes the communities `seed` provisioned from
`target.community_host_template`, using the operator key `seed` used. For each
host it archives the community as its fixture owner, then requests owner
deletion with a request id derived from `run_id` and the host. The
relay's deletion drain does the deletion asynchronously; `teardown` reports
the requests by the stage the relay returned (`requested`), hosts this run's
fixture owner does not own (`absent`), and failures. It never touches a host
owned by anyone else, and refuses a scenario without a host template.
Re-running `teardown` replays the same request ids, so it retries failures
and reports progress without conflicting. It exits non-zero while any
community failed.
Deleted hosts stay reserved as tombstones, so a torn-down `run_id` cannot be
seeded again: `seed` fails on the first host. Give each fixture a new
`run_id`.
## Safety
The generator creates real communities, channels and events. Point it only at
environments built for load tests, never at a production relay. Use a fresh
`run_id` per fixture.
Reference in New Issue View Git Blame Copy Permalink