4cf2a3bd66
## Summary `buzz-loadgen` spreads load evenly: every channel gets the same history and the same short thread chains. That never puts heavy load on a single thread. A busy conversation, such as agents talking to each other, can give one root a huge number of replies, and thread-summary work grows with it (see #8635). This PR adds an optional `[hot_thread]` scenario section so a load test can measure that case. - **Seed:** `seed` posts a hot root in community 0, channel 0, then `seed_replies` replies directly under it. The root comes from a dedicated identity that posts nothing else, so `run` finds it however many other roots the channel has. The replies go out concurrently, like the rest of the seed, so the run starts with the thread already large. - **Run:** `authors` hot-thread clients are scaled by each ramp step's `users`, the same way humans and agents are. Each one replies to the hot root over HTTP `/events` and reads the thread's newest 100 replies over HTTP `/query`, at `replies_per_hour` and `fetches_per_hour`. - **Metrics:** two new report ops, `hot_reply` and `thread_fetch`. The replies are timed, so humans subscribed to that channel also record delivery latency for them. - **No root, failed run:** an author that can't find the seeded root counts a `thread_fetch` error on every attempt, so a run without the hot thread fails instead of quietly measuring nothing. - **Off by default:** the section is disabled while `seed_replies` and `authors` are both 0, so existing scenarios are unchanged. `perf/scenarios/local-smoke.toml` gets a small hot thread, and `perf/LOADGEN.md` documents the knobs. ### Teardown The PR also adds `buzz-loadgen teardown <scenario>`, so a staging fixture can be removed after the load test. - For each community from `target.community_host_template`, it first calls `POST /operator/communities/archive` as the fixture owner, using the operator key `seed` uses. It then calls `POST /operator/communities/delete`. - The deletion `request_id` is derived from `run_id` and the host, so a re-run replays the same requests. That retries failures and reports progress without a `deletion_request_conflict`. - The relay's deletion drain does the actual deletion asynchronously. - If the fixture owner does not own a host, the host is reported `absent` and left untouched. - A scenario without a host template is refused. - Deleted hosts stay as tombstones, so `LOADGEN.md` says each fixture needs a fresh `run_id`. The signing has to happen as each derived owner, which is why this lives in the generator rather than in deployment config. ## What got simpler Nothing got simpler; this adds a scale vector. To keep it small, the PR reuses what's already there: - the seeder's post plan, plus `find_roots` with a content prefix - the existing HTTP helper, whose auth tag is now optional - humans' delivery measurement A separate chain-depth knob and dedicated viewer clients were left out. `shape.thread_depth` already seeds chains up to the relay's depth cap, and humans in the hot channel already receive the replies. ## Testing - `loadgen::seed::tests::the_hot_thread_seeds_direct_replies_under_its_own_root` checks that the hot root and its replies are seeded, that every reply is direct (depth 1), and that a rejected root fails the seed. - `loadgen::run::tests::hot_thread_authors_reply_to_the_hot_root_and_fail_without_it` checks several things: - authors post replies tagged to the hot root; - root lookups query only the hot-root identity, and a newer reply-shaped decoy is never taken as the root; - scheduled reads all target the hot root's channel and id; - the run fails when the root isn't seeded. - `loadgen::run::tests::hot_thread_authors_start_nothing_once_the_run_stops` runs 8 hot authors directly. It holds every in-flight `/events` reply at a gate until the stop is signalled, so each author's next reply is already overdue when the gate opens. It then checks that exactly 8 replies were posted, none started after the stop. Removing the post-`select!` stop check fails the test, and the gate is released before any assertion, so a failure can't hang the job. - The run now signals the stop before the last step's snapshot instead of after building that step's report. Under a loaded parallel test run, work started in that gap leaked into `shutdown`. - `loadgen::teardown::tests::teardown_deletes_only_the_fixture_owners_communities_and_replays_on_rerun` runs teardown twice against a fake operator that checks the owner, requires archive before delete, and conflicts on a different request id. It checks that: - the owned host is archived and deletion-requested; - another owner's host and an unseeded host are reported absent and left untouched; - a 503 on archive is reported as a failure; - the second pass converges to the same result; - a deletion already past membership purge, where archive returns 404, is replayed and reported at its stage under `requested`; - the same run under another host template gets its own request ids: another owner's host is reported absent rather than conflicting with a request id already used. If the request id is replaced with a fresh one per run, the test fails. - `loadgen::teardown::tests::teardown_refuses_a_scenario_without_templated_communities` - All of these run in the existing `loadgen::` unit lane. - Teardown was run end to end at `28192004` against a local relay. The relay had operator keys configured, local S3 behind it, and `buzz-admin deletions drain`. The run covered: - seeding three templated communities; - teardown, with every fixture archived and a deletion request at its derived id; - a re-run that converged; - the drain, which tombstoned the fixtures; - teardown again after the purge, which reported each fixture at `retention_pending` and exited 0; - a re-seed of the torn-down run, which failed at provisioning. A community outside the fixture stayed active, with no deletion request. - At `28192004`, `cargo fmt --check`, `cargo clippy -p buzz-test-client --all-targets -D warnings` and six parallel runs of `cargo test -p buzz-test-client` pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
8.8 KiB
Relay load generator
buzz-loadgen (in crates/buzz-test-client) seeds a data shape and then
drives a population of simulated humans and agents through a ramp of user
multipliers, reporting latency percentiles, errors and rate-limit rejections
per operation for each step.
cargo run --release -p buzz-test-client --bin buzz-loadgen -- seed perf/scenarios/local-smoke.toml
cargo run --release -p buzz-test-client --bin buzz-loadgen -- run perf/scenarios/local-smoke.toml
cargo run --release -p buzz-test-client --bin buzz-loadgen -- teardown perf/scenarios/<templated>.toml
Scale vectors
| Vector | Scenario knob |
|---|---|
| Communities | shape.communities (each provisioned through POST /operator/communities) |
| Channels | shape.channels_per_community, population.channels_per_member |
| History depth | shape.history_messages_per_channel, human.history_limit |
| Deep threads | shape.thread_roots_per_channel, shape.thread_depth, human.thread_reply_fraction |
| One hot thread | [hot_thread]: seed_replies, authors, replies_per_hour, fetches_per_hour |
| Humans / agents | population.humans, population.agents, multiplied by each [[ramp]] step's users |
| Throughput per client | [human] and [agent] rates |
| Reconnect churn | human.mean_session_secs, population.connect_rate_per_sec |
shape.thread_depth seeds each thread as a chain: reply k replies to reply
k-1, so the thread's deepest reply sits at depth thread_depth. The relay
rejects depths above 100, so scenarios do too. Threads are seeded in parallel
with each other.
A scenario that cannot be scheduled is rejected when it is parsed: for
example seed_concurrency of 0, human.presence_interval_secs below 1, or a
community_host_template with a port while target.connect_addr is set (the
override maps hostnames, so HTTP would ignore it).
[hot_thread] models one thread that keeps growing, such as a conversation
between busy agents. seed posts its root in community 0, channel 0 and
then seed_replies replies directly under it, concurrently. During run,
authors clients (multiplied by each ramp step's users, like humans and
agents) reply to that root and read its newest 100 replies over HTTP. Their
replies are timed, so humans subscribed to that channel record delivery.
Leave seed_replies and authors at 0 to disable it.
What each client does
- Human: authenticates over WebSocket and subscribes to its channels'
messages, its mentions, typing, and ten peers' presence. On connect it
fetches
history_limitmessages per channel. Then it sends presence heartbeats, typing indicators, messages (some as replies into seeded deep threads) and reactions on Poisson schedules. - Agent: authenticates with a NIP-OA tag from its owner, polls HTTP
/query(channel window plus mentions), posts messages through HTTP/events, and sends encrypted kind:24200 observer frames to its owner. - Hot-thread author: finds the seeded hot root, then posts replies to it
through HTTP
/events(hot_reply) and reads the thread through HTTP/query(thread_fetch). Until the root is found every attempt counts as athread_fetcherror, so a run without the seeded thread fails.
Messages carry their send time, so every subscriber records publish→delivery
latency (deliver). Across machines this depends on clock sync.
Modeling a deployment
The [human] and [agent] defaults are round placeholders. To model a known
deployment, divide its hourly event counts by its client counts and set the
results as the scenario's rates. Then a ramp step's users reads as a
multiple of that deployment's population. Keep such scenarios with that
deployment's own configuration.
Targeting and membership
target.connect_addr (host:port) applies only together with
target.community_host_template: it then replaces host resolution for the
community hosts, over both WebSocket and HTTP, while TLS names, Host headers
and the NIP-42 relay tag still name the community. Without a host
template, the generator connects to ws_url and http_url as written and
connect_addr has no effect.
Seeded identities do not join communities or channels. A relay that requires
membership rejects both seed and run.
Sharding
run --shard i/n runs the clients whose global index g satisfies
g % n == i. Identities, community hosts and channel ids derive from
run_id, so shards need no coordination: run seed once, then start n
processes, such as a Kubernetes Indexed Job, with the same scenario. Each
shard reports its own clients; add counts and take the worst percentiles
across shards.
ghcr.io/block/buzz-loadgen:main (and :sha-<7>) packages the binary as the
image entrypoint, published by .github/workflows/loadgen-image.yml from
Dockerfile.loadgen. Mount the scenario and pass the subcommand as the
container arguments, for example run /etc/loadgen/scenario.toml --shard $(JOB_COMPLETION_INDEX)/30. Pin :sha-<7> or a digest for runs you intend
to compare; :main moves. The container runs as UID 10001 with / as its
working directory and no home, so point --report at a writable mount.
Abort
Each step reports every operation from the start of its ramp to the end of
its hold, so connection failures during the ramp count. At the end of the
hold, the run stops if any operation's p99 (publish→delivery included)
exceeds abort.max_p99_ms, if any operation other than delivery has an
error rate above abort.max_error_rate, or if a client exited unexpectedly.
An operation still in flight when the hold ends is reported as pending
but not judged: it has not failed yet. A relay CLOSED on a live
subscription counts as a subscribe_eose error and ends that client's
session. Each connect plus authentication must finish within
target.connect_timeout_secs (default 10). Quota rejections, meaning an HTTP 429 or the relay's
rate-limited: quota exceeded text, are reported as throttled, not errors.
Every other rejection, including rate-limited: shared admission unavailable
(the relay's admission store is down), counts as an error.
Ramp steps must increase users, since clients are only ever added.
When the run stops, clients get 15 seconds to settle what they have in flight
and disconnect. An operation already under way runs to its own deadline
instead of being abandoned: a connect attempt finishes within
target.connect_timeout_secs, and a publish waits at most 30 seconds for its
OK. A stalled connect therefore ends as a connect timeout error and a
healthy one as a success. What settles after the last step is reported in the
shutdown object (ops, failures, abort_reason) and judged by the same
limits as a step, so a connect that never completes fails the run there. A client
still busy when the grace period ends is cancelled; its in-flight operations
are recorded as unfinished at stop errors, and the cancellation fails the run
whatever the limits. Keep target.connect_timeout_secs below 15, or a stalled
connect is cancelled instead of timing out. Publishes, history fetches and
HTTP requests can outlast the grace period (30 s, 30 s and 60 s deadlines), in
which case the client is cancelled and the run fails.
The process exits non-zero when a step aborts, a client panics, a client had to
be cancelled, or shutdown.abort_reason is set.
seed exits non-zero unless the relay accepted every planned post.
seed provisions each community with create_only, so it never rotates the
owner of an existing host. A host that already exists is reused only when the
fixture owner derived from run_id owns it (re-running seed for the same
scenario); otherwise seed fails. Use a fresh run_id or a host template
with {run} to avoid colliding with another community.
Teardown
teardown <scenario> removes the communities seed provisioned from
target.community_host_template, using the operator key seed used. For each
host it archives the community as its fixture owner, then requests owner
deletion with a request id derived from run_id and the host. The
relay's deletion drain does the deletion asynchronously; teardown reports
the requests by the stage the relay returned (requested), hosts this run's
fixture owner does not own (absent), and failures. It never touches a host
owned by anyone else, and refuses a scenario without a host template.
Re-running teardown replays the same request ids, so it retries failures
and reports progress without conflicting. It exits non-zero while any
community failed.
Deleted hosts stay reserved as tombstones, so a torn-down run_id cannot be
seeded again: seed fails on the first host. Give each fixture a new
run_id.
Safety
The generator creates real communities, channels and events. Point it only at
environments built for load tests, never at a production relay. Use a fresh
run_id per fixture.