mirror of
https://github.com/block/buzz.git
synced 2026-10-31 14:31:02 +00:00
4cf2a3bd66
## Summary `buzz-loadgen` spreads load evenly: every channel gets the same history and the same short thread chains. That never puts heavy load on a single thread. A busy conversation, such as agents talking to each other, can give one root a huge number of replies, and thread-summary work grows with it (see #8635). This PR adds an optional `[hot_thread]` scenario section so a load test can measure that case. - **Seed:** `seed` posts a hot root in community 0, channel 0, then `seed_replies` replies directly under it. The root comes from a dedicated identity that posts nothing else, so `run` finds it however many other roots the channel has. The replies go out concurrently, like the rest of the seed, so the run starts with the thread already large. - **Run:** `authors` hot-thread clients are scaled by each ramp step's `users`, the same way humans and agents are. Each one replies to the hot root over HTTP `/events` and reads the thread's newest 100 replies over HTTP `/query`, at `replies_per_hour` and `fetches_per_hour`. - **Metrics:** two new report ops, `hot_reply` and `thread_fetch`. The replies are timed, so humans subscribed to that channel also record delivery latency for them. - **No root, failed run:** an author that can't find the seeded root counts a `thread_fetch` error on every attempt, so a run without the hot thread fails instead of quietly measuring nothing. - **Off by default:** the section is disabled while `seed_replies` and `authors` are both 0, so existing scenarios are unchanged. `perf/scenarios/local-smoke.toml` gets a small hot thread, and `perf/LOADGEN.md` documents the knobs. ### Teardown The PR also adds `buzz-loadgen teardown <scenario>`, so a staging fixture can be removed after the load test. - For each community from `target.community_host_template`, it first calls `POST /operator/communities/archive` as the fixture owner, using the operator key `seed` uses. It then calls `POST /operator/communities/delete`. - The deletion `request_id` is derived from `run_id` and the host, so a re-run replays the same requests. That retries failures and reports progress without a `deletion_request_conflict`. - The relay's deletion drain does the actual deletion asynchronously. - If the fixture owner does not own a host, the host is reported `absent` and left untouched. - A scenario without a host template is refused. - Deleted hosts stay as tombstones, so `LOADGEN.md` says each fixture needs a fresh `run_id`. The signing has to happen as each derived owner, which is why this lives in the generator rather than in deployment config. ## What got simpler Nothing got simpler; this adds a scale vector. To keep it small, the PR reuses what's already there: - the seeder's post plan, plus `find_roots` with a content prefix - the existing HTTP helper, whose auth tag is now optional - humans' delivery measurement A separate chain-depth knob and dedicated viewer clients were left out. `shape.thread_depth` already seeds chains up to the relay's depth cap, and humans in the hot channel already receive the replies. ## Testing - `loadgen::seed::tests::the_hot_thread_seeds_direct_replies_under_its_own_root` checks that the hot root and its replies are seeded, that every reply is direct (depth 1), and that a rejected root fails the seed. - `loadgen::run::tests::hot_thread_authors_reply_to_the_hot_root_and_fail_without_it` checks several things: - authors post replies tagged to the hot root; - root lookups query only the hot-root identity, and a newer reply-shaped decoy is never taken as the root; - scheduled reads all target the hot root's channel and id; - the run fails when the root isn't seeded. - `loadgen::run::tests::hot_thread_authors_start_nothing_once_the_run_stops` runs 8 hot authors directly. It holds every in-flight `/events` reply at a gate until the stop is signalled, so each author's next reply is already overdue when the gate opens. It then checks that exactly 8 replies were posted, none started after the stop. Removing the post-`select!` stop check fails the test, and the gate is released before any assertion, so a failure can't hang the job. - The run now signals the stop before the last step's snapshot instead of after building that step's report. Under a loaded parallel test run, work started in that gap leaked into `shutdown`. - `loadgen::teardown::tests::teardown_deletes_only_the_fixture_owners_communities_and_replays_on_rerun` runs teardown twice against a fake operator that checks the owner, requires archive before delete, and conflicts on a different request id. It checks that: - the owned host is archived and deletion-requested; - another owner's host and an unseeded host are reported absent and left untouched; - a 503 on archive is reported as a failure; - the second pass converges to the same result; - a deletion already past membership purge, where archive returns 404, is replayed and reported at its stage under `requested`; - the same run under another host template gets its own request ids: another owner's host is reported absent rather than conflicting with a request id already used. If the request id is replaced with a fresh one per run, the test fails. - `loadgen::teardown::tests::teardown_refuses_a_scenario_without_templated_communities` - All of these run in the existing `loadgen::` unit lane. - Teardown was run end to end at `28192004` against a local relay. The relay had operator keys configured, local S3 behind it, and `buzz-admin deletions drain`. The run covered: - seeding three templated communities; - teardown, with every fixture archived and a deletion request at its derived id; - a re-run that converged; - the drain, which tombstoned the fixtures; - teardown again after the purge, which reported each fixture at `retention_pending` and exited 0; - a re-seed of the torn-down run, which failed at provisioning. A community outside the fixture stayed active, with no deletion request. - At `28192004`, `cargo fmt --check`, `cargo clippy -p buzz-test-client --all-targets -D warnings` and six parallel runs of `cargo test -p buzz-test-client` pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>