buzz/perf/LOADGEN.md

Relay load generator

buzz-loadgen (in crates/buzz-test-client) seeds a data shape and then drives a population of simulated humans and agents through a ramp of user multipliers, reporting latency percentiles, errors and rate-limit rejections per operation for each step.

cargo run --release -p buzz-test-client --bin buzz-loadgen -- seed perf/scenarios/local-smoke.toml
cargo run --release -p buzz-test-client --bin buzz-loadgen -- run perf/scenarios/local-smoke.toml
cargo run --release -p buzz-test-client --bin buzz-loadgen -- teardown perf/scenarios/<templated>.toml

Scale vectors

Vector	Scenario knob
Communities	shape.communities (each provisioned through POST /operator/communities)
Channels	shape.channels_per_community, population.channels_per_member
History depth	shape.history_messages_per_channel, human.history_limit
Deep threads	shape.thread_roots_per_channel, shape.thread_depth, human.thread_reply_fraction
One hot thread	[hot_thread]: seed_replies, authors, replies_per_hour, fetches_per_hour
Humans / agents	population.humans, population.agents, multiplied by each [[ramp]] step's users
Throughput per client	[human] and [agent] rates
Reconnect churn	human.mean_session_secs, population.connect_rate_per_sec

shape.thread_depth seeds each thread as a chain: reply k replies to reply k-1, so the thread's deepest reply sits at depth thread_depth. The relay rejects depths above 100, so scenarios do too. Threads are seeded in parallel with each other.

A scenario that cannot be scheduled is rejected when it is parsed: for example seed_concurrency of 0, human.presence_interval_secs below 1, or a community_host_template with a port while target.connect_addr is set (the override maps hostnames, so HTTP would ignore it).

[hot_thread] models one thread that keeps growing, such as a conversation between busy agents. seed posts its root in community 0, channel 0 and then seed_replies replies directly under it, concurrently. During run, authors clients (multiplied by each ramp step's users, like humans and agents) reply to that root and read its newest 100 replies over HTTP. Their replies are timed, so humans subscribed to that channel record delivery. Leave seed_replies and authors at 0 to disable it.

What each client does

• Human: authenticates over WebSocket and subscribes to its channels' messages, its mentions, typing, and ten peers' presence. On connect it fetches history_limit messages per channel. Then it sends presence heartbeats, typing indicators, messages (some as replies into seeded deep threads) and reactions on Poisson schedules.
• Agent: authenticates with a NIP-OA tag from its owner, polls HTTP /query (channel window plus mentions), posts messages through HTTP /events, and sends encrypted kind:24200 observer frames to its owner.
• Hot-thread author: finds the seeded hot root, then posts replies to it through HTTP /events (hot_reply) and reads the thread through HTTP /query (thread_fetch). Until the root is found every attempt counts as a thread_fetch error, so a run without the seeded thread fails.

Messages carry their send time, so every subscriber records publish→delivery latency (deliver). Across machines this depends on clock sync.

Modeling a deployment

The [human] and [agent] defaults are round placeholders. To model a known deployment, divide its hourly event counts by its client counts and set the results as the scenario's rates. Then a ramp step's users reads as a multiple of that deployment's population. Keep such scenarios with that deployment's own configuration.

Targeting and membership

target.connect_addr (host:port) applies only together with target.community_host_template: the generator then dials the community hosts there in plaintext, over both WebSocket and HTTP, such as a relay's in-cluster service. Set ws_url and http_url to the community's public scheme anyway (wss:// and https:// for a relay behind TLS): Host headers, NIP-98 u tags and the NIP-42 relay tag keep that scheme, which the relay checks against its own relay URL. Without a host template, the generator connects to ws_url and http_url as written and connect_addr has no effect.

Seeded identities do not join communities or channels. A relay that requires membership rejects both seed and run.

Sharding

run --shard i/n runs the clients whose global index g satisfies g % n == i. Identities, community hosts and channel ids derive from run_id, so shards need no coordination: run seed once, then start n processes, such as a Kubernetes Indexed Job, with the same scenario. Each shard reports its own clients; add counts and take the worst percentiles across shards.

ghcr.io/block/buzz-loadgen:main (and :sha-<7>) packages the binary as the image entrypoint, published by .github/workflows/loadgen-image.yml from Dockerfile.loadgen. Mount the scenario and pass the subcommand as the container arguments, for example run /etc/loadgen/scenario.toml --shard $(JOB_COMPLETION_INDEX)/30. Pin :sha-<7> or a digest for runs you intend to compare; :main moves. The container runs as UID 10001 with / as its working directory and no home, so point --report at a writable mount.

Abort

Each step reports every operation from the start of its ramp to the end of its hold, so connection failures during the ramp count. At the end of the hold, the run stops if any operation's p99 (publish→delivery included) exceeds abort.max_p99_ms, if any operation other than delivery has an error rate above abort.max_error_rate, or if a client exited unexpectedly. An operation still in flight when the hold ends is reported as pending but not judged: it has not failed yet. A relay CLOSED on a live subscription counts as a subscribe_eose error and ends that client's session. Each connect plus authentication must finish within target.connect_timeout_secs (default 10). Quota rejections, meaning an HTTP 429 or the relay's rate-limited: quota exceeded text, are reported as throttled, not errors. Every other rejection, including rate-limited: shared admission unavailable (the relay's admission store is down), counts as an error. Ramp steps must increase users, since clients are only ever added.

When the run stops, clients get 15 seconds to settle what they have in flight and disconnect. An operation already under way runs to its own deadline instead of being abandoned: a connect attempt finishes within target.connect_timeout_secs, and a publish waits at most 30 seconds for its OK. A stalled connect therefore ends as a connect timeout error and a healthy one as a success. What settles after the last step is reported in the shutdown object (ops, failures, abort_reason) and judged by the same limits as a step, so a connect that never completes fails the run there. A client still busy when the grace period ends is cancelled; its in-flight operations are recorded as unfinished at stop errors, and the cancellation fails the run whatever the limits. Keep target.connect_timeout_secs below 15, or a stalled connect is cancelled instead of timing out. Publishes, history fetches and HTTP requests can outlast the grace period (30 s, 30 s and 60 s deadlines), in which case the client is cancelled and the run fails.

The process exits non-zero when a step aborts, a client panics, a client had to be cancelled, or shutdown.abort_reason is set.

seed exits non-zero unless the relay accepted every planned post.

seed provisions each community with create_only, so it never rotates the owner of an existing host. A host that already exists is reused only when the fixture owner derived from run_id owns it (re-running seed for the same scenario); otherwise seed fails. Use a fresh run_id or a host template with {run} to avoid colliding with another community.

Teardown

teardown <scenario> removes the communities seed provisioned from target.community_host_template, using the operator key seed used. For each host it archives the community as its fixture owner, then requests owner deletion with a request id derived from run_id and the host. The relay's deletion drain does the deletion asynchronously; teardown reports the requests by the stage the relay returned (requested), hosts this run's fixture owner does not own (absent), and failures. It never touches a host owned by anyone else, and refuses a scenario without a host template.

Re-running teardown replays the same request ids, so it retries failures and reports progress without conflicting. It exits non-zero while any community failed.

Deleted hosts stay reserved as tombstones, so a torn-down run_id cannot be seeded again: seed fails on the first host. Give each fixture a new run_id.

Safety

The generator creates real communities, channels and events. Point it only at environments built for load tests, never at a production relay. Use a fresh run_id per fixture.