jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-11-08 02:11:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(loadgen): model agent turns and sustained holds (#9182)

Browse Source 
## Why

The loadgen agent polled HTTP `/query` every few seconds. A real agent
harness (`buzz-acp`) never polls when idle. It works in bursts around
each prompt, and those bursts are the load a fleet of active agents puts
on the relay. To answer "how well does the relay handle N active users
plus their agents over a sustained period", the loadgen needs to model
those bursts and hold them for an hour or more.

## What

- **`[agent.turn]`**: an idle agent starts a turn on a Poisson schedule
and runs one turn at a time. A turn lasts 0.5–1.5 × `mean_secs`. The
steps follow `buzz-acp`:
- **Start**: a `limit:1` `/query` finds the message to answer
(`http_query`), which stands in for the harness's subscription. Then the
harness's own reads (`turn_context` op): for a threaded message, its
thread context in one `/query` (root by id, newest 13 replies, the
agent's own newest reply) plus a `/count` when the window is full. Then
kind:39000 metadata, project home (kind:30621, kind:30617) and the
author's profile. Then two kind:7 reactions on the message.
- **Work**: typing every `typing_secs` and observer frames of
`frame_bytes` every `frame_secs` over the socket, plus `reads` tool
reads (`/query`, limit 50).
- **End**: a threaded reply, under the root that NIP-10 `resolve()`
gives. Then reaction removal: query own reactions, then a kind:5
deletion for each.
- One HTTP action at a time runs next to the socket. Socket publishes
(presence, typing, frames) don't wait for their OK: each OK is matched
as it arrives, one missing after 30 s is an error, and at most 128 wait
at once. So the socket and HTTP sides stay on schedule while either is
slow, as in the harness. A 2xx response that doesn't decode counts as an
error. No agent request starts after the stop.
- The request mix approximates sustained turns: every turn reads like a
first prompt. A harness already holding a thread reads 25 replies (not
13), batches profile lookups and caches project-home lookups for 30 s,
so this reads profiles and project homes more often than a busy harness.
- `agent.channel_subscriptions`: one live mentions REQ per channel, like
the harness. `agent.presence_interval_secs`: presence heartbeats.
- `agent.query_interval_secs = 0` turns polling off.
- **Sustained holds**: ramp steps may now repeat a level. A 60-minute
hold can be six 600 s steps, each one reported and judged, so drift
shows up as a trend.
- New op `turn_context`. New scenario
`perf/scenarios/local-agent-turns.toml`. `perf/LOADGEN.md` documents the
model.

All new fields are off by default (`turn.per_hour = 0`, presence 0, no
channel subscriptions), so existing scenarios run unchanged.

## What got simpler

- The agent loop's per-action branches now go through one `sent` result.
The observer frame build, the HTTP submit and the `/query` body read are
shared helpers. The old inline frame code is gone.
- "Hold for an hour, judge every 10 minutes" is now a ramp of repeated
levels. No new abort-window concept was needed.
- Identified debt, not touched here: abort is still judged only at step
end, so a collapsed relay is not detected until the step closes.
Repeated short steps reduce that gap.

## Testing

- On a Blox workstation at 6bb151d1: `cargo fmt --check`, `cargo clippy
-p buzz-test-client --all-targets -D warnings`, and `cargo test -p
buzz-test-client --lib` (103 passed). The five turn tests passed 10/10
repeated runs. Three mutants (awaiting the OK again, dropping the stop
guard in `submit`, dropping it in `query`) each fail a test.
- `local-agent-turns` against a local relay at 6bb151d1: both steps 0
errors, 0 clients cancelled.
- `agent_turns_read_react_stream_reply_and_remove_their_reactions` runs
an agent against a fake socket and a fake HTTP relay. It asserts the
exact thread-context filters and count, the metadata, project and
profile reads, the reactions, typing, frames and presence, and that
typing or frames arrive while a tool read is held. It also asserts the
threaded reply under the right root and the deletion of exactly the
reactions the agent added.
- `agent_turns_keep_moving_while_socket_oks_lag`: the same journey with
every socket OK 1.5 s late, past the 1 s heartbeat.
- `a_turn_stopped_while_adding_a_reaction_sends_nothing_more` and
`a_turn_stopped_while_reading_its_reactions_deletes_none`: the stop
lands while a reaction POST or removal query is held, and no request
follows it.
- `agent_turn_reads_that_do_not_decode_are_errors`: 200 responses that
are not events count as errors, and no turn reacts.
- `a_timed_out_route_is_recovered_again_after_a_renewal` (desktop) fails
2 times in 10 on unmodified main as well. That flake is pre-existing and
unrelated to this change.
- Live: a single-relay run with 10k humans and 20k turn-taking agents
(held at 1x for 60 min, then 1.5x and 2x). Results will be added here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Luke Tornquist
2026-11-06 11:33:28 -05:00
committed by GitHub
parent b329e05c5b
commit 3fdd8c6ff1
5 changed files with 1515 additions and 141 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1441
crates/buzz-test-client/src/loadgen/run.rs
View File

File diff suppressed because it is too large Load Diff

102
crates/buzz-test-client/src/loadgen/scenario.rs
View File

@@ -184,12 +184,19 @@ impl Default for HumanProfile {
#[derive(Clone, Debug, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct AgentProfile {
/// Mean interval between HTTP `/query` polls.
/// Mean interval between HTTP `/query` polls (0 disables polling).
pub query_interval_secs: f64,
/// Messages posted through HTTP `/events` per hour.
pub events_per_hour: f64,
/// Encrypted kind:24200 observer frames sent to the owner per hour.
/// Encrypted kind:24200 observer frames sent to the owner per hour,
/// outside turns.
pub observer_frames_per_hour: f64,
/// Seconds between presence heartbeats (0 sends none).
pub presence_interval_secs: f64,
/// Open one live REQ per channel for the agent's mentions there, as the
/// agent harness does, instead of relying on HTTP polling.
pub channel_subscriptions: bool,
pub turn: AgentTurn,
}
impl Default for AgentProfile {
@@ -198,10 +205,52 @@ impl Default for AgentProfile {
query_interval_secs: 5.0,
events_per_hour: 60.0,
observer_frames_per_hour: 60.0,
presence_interval_secs: 0.0,
channel_subscriptions: false,
turn: AgentTurn::default(),
}
}
}
/// Agent turns: bursts of work in one channel, as an agent harness runs a
/// prompt. A turn starts with context reads and two reactions, streams
/// typing and observer frames while it runs, makes `reads` tool reads, and
/// ends with a reply and the reactions' removal. Disabled at 0 `per_hour`.
#[derive(Clone, Debug, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct AgentTurn {
/// Turns started per hour while the agent is idle. An agent runs one
/// turn at a time.
pub per_hour: f64,
/// Mean turn length. Each turn lasts 0.5–1.5 times this.
pub mean_secs: f64,
/// Seconds between observer frames during a turn (0 sends none).
pub frame_secs: f64,
/// Plaintext size of each observer frame, before encryption.
pub frame_bytes: usize,
/// Seconds between typing indicators during a turn (0 sends none).
pub typing_secs: f64,
/// Channel reads over HTTP `/query` during a turn, as tool calls.
pub reads: usize,
}
impl Default for AgentTurn {
fn default() -> Self {
Self {
per_hour: 0.0,
mean_secs: 60.0,
frame_secs: 1.0,
frame_bytes: 1024,
typing_secs: 3.0,
reads: 2,
}
}
}
/// Largest observer frame plaintext: NIP-44 encrypts at most 65,535 bytes,
/// and the frame's JSON envelope needs some of them.
pub const MAX_FRAME_BYTES: usize = 60_000;
/// One thread in community 0, channel 0 that keeps growing: seeded with
/// many direct replies, then replied to and read throughout the run.
/// Disabled while `seed_replies` and `authors` are both 0.
@@ -416,11 +465,13 @@ impl Scenario {
if self.ramp.is_empty() {
return Err("ramp needs at least one step".into());
}
// Clients are only ever added, so a step can never shrink the population.
// Clients are only ever added, so a step can never shrink the
// population. Repeating a level splits a long hold into reported,
// judged windows.
let mut previous = 0.0;
for step in &self.ramp {
if !step.users.is_finite() || step.users <= previous {
return Err("ramp users must be finite and increase every step".into());
if !step.users.is_finite() || step.users <= 0.0 || step.users < previous {
return Err("ramp users must be finite, above 0 and never decrease".into());
}
previous = step.users;
}
@@ -434,7 +485,7 @@ impl Scenario {
"population.connect_rate_per_sec",
self.population.connect_rate_per_sec,
),
("agent.query_interval_secs", a.query_interval_secs),
("agent.turn.mean_secs", a.turn.mean_secs),
("abort.max_p99_ms", self.abort.max_p99_ms),
("desktop.backstop_secs", self.desktop.backstop_secs),
];
@@ -455,6 +506,26 @@ impl Scenario {
if !d.renewal_secs.is_finite() || !(d.renewal_secs == 0.0 || d.renewal_secs >= 1.0) {
return Err("desktop.renewal_secs must be 0 or a finite number >= 1".into());
}
// Agent intervals are 0 (off) or slow enough to schedule.
let intervals = [
(
"agent.presence_interval_secs",
a.presence_interval_secs,
1.0,
),
("agent.turn.frame_secs", a.turn.frame_secs, 0.1),
("agent.turn.typing_secs", a.turn.typing_secs, 0.1),
];
for (name, value, min) in intervals {
if !value.is_finite() || !(value == 0.0 || value >= min) {
return Err(format!("{name} must be 0 or a finite number >= {min}"));
}
}
if a.turn.frame_bytes > MAX_FRAME_BYTES {
return Err(format!(
"agent.turn.frame_bytes must be at most {MAX_FRAME_BYTES}"
));
}
let non_negative = [
("human.typing_per_hour", h.typing_per_hour),
("human.messages_per_hour", h.messages_per_hour),
@@ -462,6 +533,8 @@ impl Scenario {
("human.mean_session_secs", h.mean_session_secs),
("agent.events_per_hour", a.events_per_hour),
("agent.observer_frames_per_hour", a.observer_frames_per_hour),
("agent.query_interval_secs", a.query_interval_secs),
("agent.turn.per_hour", a.turn.per_hour),
(
"hot_thread.replies_per_hour",
self.hot_thread.replies_per_hour,
@@ -700,12 +773,18 @@ hold_secs = 10
"[human]\nthread_reply_fraction = 1.5\n",
"[agent]\nquery_interval_secs = nan\n",
"[agent]\nobserver_frames_per_hour = inf\n",
"[agent]\nquery_interval_secs = -1.0\n",
"[agent]\npresence_interval_secs = 0.5\n",
"[agent.turn]\nper_hour = -1.0\n",
"[agent.turn]\nmean_secs = 0.0\n",
"[agent.turn]\nframe_secs = 0.05\n",
"[agent.turn]\ntyping_secs = nan\n",
"[agent.turn]\nframe_bytes = 60001\n",
"[abort]\nmax_error_rate = nan\n",
"[abort]\nmax_p99_ms = inf\n",
"[[ramp]]\nusers = inf\nhold_secs = 10\n",
// A down-ramp would report a smaller population than is running.
"[[ramp]]\nusers = 0.5\nhold_secs = 10\n",
"[[ramp]]\nusers = 1.0\nhold_secs = 10\n",
];
for case in cases {
let text = format!("{MINIMAL}{case}");
@@ -718,6 +797,15 @@ hold_secs = 10
assert!(Scenario::parse(&text).is_err());
let up = format!("{MINIMAL}[[ramp]]\nusers = 2.0\nhold_secs = 10\n");
assert!(Scenario::parse(&up).is_ok());
// A repeated level splits a long hold into judged windows.
let held = format!("{MINIMAL}[[ramp]]\nusers = 1.0\nhold_secs = 10\n");
assert!(Scenario::parse(&held).is_ok());
// Polling, presence, frames and typing can each be switched off.
let off = format!(
"{MINIMAL}[agent]\nquery_interval_secs = 0.0\npresence_interval_secs = 0.0\n\
[agent.turn]\nper_hour = 6.0\nframe_secs = 0.0\ntyping_secs = 0.0\n"
);
assert!(Scenario::parse(&off).is_ok());
}
#[test]

6
crates/buzz-test-client/src/loadgen/stats.rs
View File

@@ -21,6 +21,8 @@ pub enum Op {
Subscribe,
HotReply,
ThreadFetch,
/// Agent turn: an HTTP `/query` context read at the start of a turn.
TurnContext,
/// Desktop client: NIP-11 `GET /` on connect.
DesktopNip11,
/// Desktop client: EOSE of a live subscription (global or channel batch).
@@ -46,7 +48,7 @@ pub enum Op {
}
impl Op {
pub const ALL: [Op; 28] = [
pub const ALL: [Op; 29] = [
Op::WsConnect,
Op::History,
Op::PublishEphemeral,
@@ -58,6 +60,7 @@ impl Op {
Op::Subscribe,
Op::HotReply,
Op::ThreadFetch,
Op::TurnContext,
Op::DesktopNip11,
Op::DesktopLive,
Op::DesktopReplay,
@@ -90,6 +93,7 @@ impl Op {
Op::Subscribe => "subscribe_eose",
Op::HotReply => "hot_reply",
Op::ThreadFetch => "thread_fetch",
Op::TurnContext => "turn_context",
Op::DesktopNip11 => "desktop_nip11",
Op::DesktopLive => "desktop_live_eose",
Op::DesktopReplay => "desktop_replay_eose",

57
perf/LOADGEN.md
View File

@@ -77,6 +77,10 @@ Members are assigned deterministically by global index, so every shard and
- **Agent**: authenticates with a NIP-OA tag from its owner, polls HTTP
`/query` (channel window plus mentions), posts messages through HTTP
`/events`, and sends encrypted kind:24200 observer frames to its owner.
With `agent.channel_subscriptions` it also opens one live REQ per channel
for its mentions there, and with `agent.presence_interval_secs` it sends
presence heartbeats. `agent.turn` adds turns of work, described under
[Agent turns](#agent-turns).
- **Desktop human**: the `desktop.fraction` of humans that model the desktop
app instead of the basic human above. They keep the basic human's presence,
typing, messages and reactions; the subscriptions and HTTP reads are the
@@ -97,6 +101,55 @@ results as the scenario's rates. Then a ramp step's `users` reads as a
multiple of that deployment's population. Keep such scenarios with that
deployment's own configuration.
## Agent turns
An agent harness is idle between prompts and then works in bursts. Set
`agent.turn.per_hour` above 0 to model that. An idle agent starts a turn in
one of its channels on a Poisson schedule and runs one turn at a time. A
turn lasts 0.5–1.5 times `agent.turn.mean_secs` and does what the harness
does around a prompt:
1. **Start**: an HTTP `/query` for the channel's newest message, the one
the turn answers. A harness is handed it by its subscription, so this
read is `http_query`, not `turn_context`. Then, as `turn_context`, the
harness's own reads. For a message in a thread, one `/query` for the
root, the thread's newest 13 replies and the agent's own newest reply,
then a `/count` of the replies when all 13 came back. After that, the
channel's kind:39000 metadata, its project home (kind:30621 and
kind:30617, one read each) and the message author's profile. Then two
kind:7 reactions on the message through HTTP `/events`.
2. **Work**: a typing indicator every `typing_secs` and an encrypted
observer frame of `frame_bytes` plaintext every `frame_secs` over the
socket, plus `reads` HTTP `/query` reads of the channel's newest 50
messages at random points in the turn, as tool calls.
3. **End**: a timed reply through HTTP `/events`, at the first level of the
message's thread (under its root), as a harness answers a person; a
top-level message becomes the root of a new thread. Then each
reaction's removal: a `/query` for the agent's reactions on
the message and a kind:5 deletion of each one found.
An agent makes one HTTP request sequence at a time, and the socket keeps
running beside it: presence, typing and frames are sent without waiting for
their `OK`, so they stay on schedule while a request or an `OK` is slow, as a
harness's do. A publish has 30 seconds from the start of its send: one the
socket cannot write by then fails and drops the connection, and an `OK` not
back by then is an error. A read counts as an error when its response is not
valid JSON of the expected shape. Requests and publishes do not start after
the stop; one under way finishes, and publishes awaiting their `OK` settle
within the shutdown grace or fail the run.
The request mix is an approximation of sustained turns, not a replay of the
harness. Every turn reads as a first prompt does. A harness that keeps
working in one thread reads 25 replies instead of 13, batches profile
lookups across prompts and caches project-home lookups for 30 seconds, so
the turn context here reads profiles and project homes more often than a
busy harness does.
For a harness-like agent, also set `agent.query_interval_secs = 0` (no
polling) and `events_per_hour = observer_frames_per_hour = 0`, so turns are
its only work. A turn under way when the socket drops continues after the
reconnect.
## Targeting and membership
`target.connect_addr` (`host:port`) applies only together with
@@ -258,7 +311,9 @@ session. Each connect plus authentication must finish within
`rate-limited: quota exceeded` text, are reported as `throttled`, not errors.
Every other rejection, including `rate-limited: shared admission unavailable`
(the relay's admission store is down), counts as an error.
Ramp steps must increase `users`, since clients are only ever added.
Ramp steps must never decrease `users`, since clients are only ever added.
Repeat a level to split a long hold into windows that are each reported and
judged, for example six 600-second steps at the same `users` for an hour.
When the run stops, clients get 15 seconds to settle what they have in flight
and disconnect. An operation already under way runs to its own deadline

50
perf/scenarios/local-agent-turns.toml Normal file
View File

@@ -0,0 +1,50 @@
# Agents that work in turns, as an agent harness does, against a local
# relay (`just relay`). Each human owns two agents; agents do not poll.
# cargo run -p buzz-test-client --bin buzz-loadgen -- seed perf/scenarios/local-agent-turns.toml
# cargo run -p buzz-test-client --bin buzz-loadgen -- run perf/scenarios/local-agent-turns.toml
run_id = "local-agent-turns"
[target]
ws_url = "ws://localhost:3000"
http_url = "http://localhost:3000"
[shape]
communities = 1
channels_per_community = 10
history_messages_per_channel = 20
thread_roots_per_channel = 1
thread_depth = 5
[population]
humans = 20
agents = 40
channels_per_member = 3
connect_rate_per_sec = 50.0
[human]
messages_per_hour = 30.0
[agent]
query_interval_secs = 0.0
events_per_hour = 0.0
observer_frames_per_hour = 0.0
presence_interval_secs = 60.0
channel_subscriptions = true
# Short turns, often, so a short run finishes many of them.
[agent.turn]
per_hour = 120.0
mean_secs = 15.0
frame_secs = 1.0
frame_bytes = 1024
typing_secs = 3.0
reads = 2
# The same level twice: a sustained hold reported in two windows.
[[ramp]]
users = 1.0
hold_secs = 60
[[ramp]]
users = 1.0
hold_secs = 60