benchmarks: Harbor Terminal-Bench harness for Buzz agent-team orchestration (harbor-buzz-orchestra) (#1504)

Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>

.github/workflows/benchmark-harbor.yml

@@ -0,0 +1,39 @@
+name: Harbor Buzz Orchestra
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "benchmarks/harbor-buzz-orchestra/**"
+      - ".github/workflows/benchmark-harbor.yml"
+  pull_request:
+    paths:
+      - "benchmarks/harbor-buzz-orchestra/**"
+      - ".github/workflows/benchmark-harbor.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: Python tests and lint
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - name: Test adapter
+        working-directory: benchmarks/harbor-buzz-orchestra
+        run: |
+          python -m pip install --disable-pip-version-check -e ".[dev]"
+          pytest -q
+          ruff check .
+      - name: Test provisioner
+        working-directory: benchmarks/harbor-buzz-orchestra/testbed
+        run: |
+          python -m pip install --disable-pip-version-check -e ".[dev]"
+          pytest -q
+          ruff check .

Justfile

@@ -747,3 +747,17 @@ goose-bg relay="ws://localhost:3000" agents="1" heartbeat="0" prompt="" key="$BU
     source ./scripts/_goose-env.sh "{{relay}}" "{{key}}" "{{agents}}" "{{heartbeat}}" "{{prompt}}"
     screen -dmS goose-agent-{{agents}} bash -c "$(printf '%q ' env "${env_args[@]}") ./target/release/buzz-acp"
     echo "Agent running in screen session 'goose-agent-{{agents}}'. Attach with: screen -r goose-agent-{{agents}}"
+
+# ─── Benchmarking ─────────────────────────────────────────────────────────────
+
+# Run the Buzz orchestra benchmark — leaderboard-eligible by default (TB 2.1, k=5, Sonnet+Haiku). Stands up its own Docker stack; --gui opens a live spectator desktop app; other flags pass to benchmark.py (--dataset/--path, --include-task, --attempts, --manifest, --dry-run, ...)
+benchmark *ARGS:
+    #!/usr/bin/env bash
+    set -euo pipefail
+    export PATH="{{justfile_directory()}}/bin:$PATH"
+    uv run --project benchmarks/harbor-buzz-orchestra/testbed \
+        benchmarks/harbor-buzz-orchestra/scripts/benchmark.py {{ARGS}}
+
+# Stop the benchmark Docker stack (state and channels are kept)
+benchmark-down:
+    docker compose --project-name buzz-benchmark down

benchmarks/harbor-buzz-orchestra/.gitignore

@@ -0,0 +1,6 @@
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.benchmark/
+jobs/

benchmarks/harbor-buzz-orchestra/README.md

@@ -0,0 +1,129 @@
+# Harbor Buzz Orchestra
+
+A stock-Harbor custom agent that runs a manifest-defined team through the real
+Buzz stack. Harbor sees one `BuzzOrchestraAgent`; behind that adapter, one
+orchestrator and N workers coordinate over the production relay/Postgres.
+Each agent runs *inside* the Harbor task container as the same
+`buzz-acp` → `buzz-agent` → `buzz-dev-mcp` process tree the desktop app
+launches: the production MCP toolset (shell, file tools, todo) with the
+`buzz` CLI on the shell's PATH. No Harbor fork or patch is required.
+
+## Define the team
+
+The manifest is the benchmark condition. Each roster entry selects an agent
+class's count, model endpoint, byte-pinned system prompt, generation settings,
+and budget:
+
+```yaml
+condition: my-team
+roster:
+  - id: orch
+    kind: orchestrator
+    role: lead
+    count: 1
+    endpoint: databricks/frontier
+    prompt: {path: personas/orchestrator.md, sha256: <sha256>}
+    generation: {max_output_tokens: 4096, context_window_tokens: 128000}
+  - id: worker
+    kind: worker
+    role: implementer
+    count: 4
+    endpoint: databricks/fast-worker
+    prompt: {path: personas/worker.md, sha256: <sha256>}
+    generation: {max_output_tokens: 4096, context_window_tokens: 128000}
+```
+
+`endpoint_config` maps those endpoint names to providers, URLs, and API-key
+environment variables. The adapter contains no fixed roster or model.
+
+## Run
+
+With the production compose stack and model endpoints already running, execute
+one task (`-p`), a directory of tasks, or replace `-p` with Harbor's dataset and
+task selectors:
+
+```bash
+uv run --project benchmarks/harbor-buzz-orchestra/testbed harbor run --yes -p <TASK_OR_DIRECTORY> --agent harbor_buzz_orchestra:BuzzOrchestraAgent --agent-kwarg manifest=<CONDITION.yaml> --agent-kwarg provisioner_factory=harbor_buzz_testbed:provisioner_from_dict --agent-kwarg provisioner_config=<PROVISIONER.json> --agent-kwarg endpoint_config=<ENDPOINTS.json> --agent-kwarg artifact_root=benchmarks/harbor-buzz-orchestra --agent-kwarg buzz_acp_binary=<LINUX_BIN>/buzz-acp --agent-kwarg buzz_agent_binary=<LINUX_BIN>/buzz-agent --agent-kwarg buzz_dev_mcp_binary=<LINUX_BIN>/buzz-dev-mcp --agent-kwarg buzz_cli_binary=target/debug/buzz --agent-kwarg run_id="bench-$(date -u +%Y%m%dT%H%M%SZ)" --agent-timeout-multiplier 15 --n-concurrent 1
+```
+
+`buzz_acp_binary`/`buzz_agent_binary`/`buzz_dev_mcp_binary` must be **Linux**
+builds matching the task image architecture — they are uploaded into each task
+container (`just benchmark` cross-builds them automatically; musl-static, so
+any Linux base image works). `buzz_cli_binary` is the **host** CLI the harness
+uses to act as the trial user.
+
+`--n-concurrent 1` is the safe laptop setting for a serialized local model; it
+is not an orchestration requirement. Some TB graders install dependencies from
+public package registries at verification time — run benchmarks off networks
+that block those installs (e.g. corporate VPNs).
+
+Each trial gets fresh keys and a private Buzz channel. The provisioner archives
+rather than deletes that channel, leaving the relay/Postgres event timeline
+and the per-agent acp/agent logs (downloaded into the trial's `buzz/`
+artifacts) available for analysis.
+
+## Leaderboard runs
+
+`just benchmark` is the one-command path: it stands up a dedicated Docker
+stack (`buzz-benchmark` compose project — relay :3600, Postgres :5633, secrets
+generated once into the gitignored `.benchmark/`), applies the benchmark
+schema, and defaults to leaderboard-eligible settings (Terminal-Bench 2.1,
+5 attempts per problem, the Sonnet+Haiku team). All selectors pass through:
+
+```bash
+just benchmark                                   # full TB 2.1, k=5
+just benchmark --path <TASK_DIR> -k 1            # one local task, one attempt
+just benchmark -i "cobol*" --attempts 3          # dataset subset
+just benchmark --gui                             # watch the run live
+```
+
+One pinned user identity fronts the whole benchmark environment: it owns
+every trial channel (named after the task) and posts every task prompt, and
+trial channels are kept rather than archived. `--gui` adds that user to the
+relay membership list and opens the Buzz desktop app logged in as them, so
+channels fill the sidebar as the run progresses — watch, don't type; a human
+message mid-trial would taint the run. `just benchmark-down` stops the stack.
+
+Networking: the relay is host-header tenant-bound, so agents must dial its
+canonical address (`ws://localhost:3600`) even from inside a task container.
+`just benchmark` uploads a tiny std-only loopback forwarder
+([`forwarder/relay_forwarder.rs`](forwarder/relay_forwarder.rs)) with the
+agent stack; it listens on the container's loopback and bridges the byte
+stream to the Docker host gateway (`host.docker.internal`, overridable via
+`BUZZ_BENCHMARK_DOCKER_HOST`).
+
+`scripts/run_leaderboard.py` is the layer underneath, for running against an
+already-provisioned stack. It wraps the invocation above with only
+leaderboard-legal settings — it does not accept or forward timeout or resource
+overrides, so the job directory it produces passes Harbor's static validation
+as-is. Give it a problem set, attempts per problem, and a team manifest:
+
+```bash
+uv run --project benchmarks/harbor-buzz-orchestra/testbed \
+    benchmarks/harbor-buzz-orchestra/scripts/run_leaderboard.py \
+    --dataset terminal-bench/terminal-bench-2-1 \
+    --attempts 5 \
+    --manifest benchmarks/harbor-buzz-orchestra/manifests/<TEAM>.yaml \
+    --endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/<ENDPOINTS>.json \
+    --provisioner-config <PROVISIONER.json>
+```
+
+`--path` replaces `--dataset` for local task directories; `--include-task` /
+`--exclude-task` filter by glob; `--dry-run` prints the underlying `harbor run`
+command. After the job finishes the script derives a `metadata.yaml` from the
+manifest roster (validated schema; review the display names before submitting)
+and prints the `harbor upload` / `harbor leaderboard submit` commands.
+
+## Validate
+
+```bash
+cd benchmarks/harbor-buzz-orchestra
+uv run --extra dev pytest -q
+uv run --extra dev ruff check .
+cd testbed
+uv run --extra dev pytest -q
+uv run --extra dev ruff check .
+```
+
+Live provisioner tests require the benchmark compose stack and opt-in
+environment described in `testbed/tests/test_provisioner_live.py`.

benchmarks/harbor-buzz-orchestra/forwarder/relay_forwarder.rs

@@ -0,0 +1,73 @@
+//! Loopback TCP forwarder for the benchmark task container.
+//!
+//! The benchmark relay is host-header tenant-bound: its community row is the
+//! authority of its own `RELAY_URL` (e.g. `localhost:3600`), and a request
+//! presenting any other `Host` fails closed. Agents inside a Harbor task
+//! container can only reach the host-published relay via the Docker host
+//! alias (`host.docker.internal`), which would present the wrong `Host`.
+//!
+//! So the container runtime uploads this forwarder next to the agent stack:
+//! agents dial `ws://localhost:<port>` — presenting the exact `Host` the
+//! community row expects — and the forwarder bridges the byte stream to the
+//! host gateway. Transparent to everything above TCP (WebSocket, the buzz
+//! CLI, git-over-HTTP). std-only; compiled with plain `rustc` against the
+//! musl target, so it runs on any Linux task image.
+//!
+//! Usage: `relay-forwarder <listen-addr> <target-addr>`
+
+use std::io::{self, Read, Write};
+use std::net::{Shutdown, TcpListener, TcpStream};
+use std::thread;
+
+fn main() -> io::Result<()> {
+    let mut args = std::env::args().skip(1);
+    let (listen, target) = match (args.next(), args.next()) {
+        (Some(listen), Some(target)) => (listen, target),
+        _ => {
+            eprintln!("usage: relay-forwarder <listen-addr> <target-addr>");
+            std::process::exit(2);
+        }
+    };
+    let listener = TcpListener::bind(&listen)?;
+    // Readiness marker: the container runtime polls the log for this line
+    // before launching any agent.
+    println!("forwarding {listen} -> {target}");
+    for client in listener.incoming() {
+        let Ok(client) = client else { continue };
+        let target = target.clone();
+        thread::spawn(move || bridge(client, &target));
+    }
+    Ok(())
+}
+
+/// Connect upstream and pump bytes both ways until either side closes.
+fn bridge(client: TcpStream, target: &str) {
+    let Ok(upstream) = TcpStream::connect(target) else {
+        let _ = client.shutdown(Shutdown::Both);
+        return;
+    };
+    let (Ok(client_read), Ok(upstream_read)) = (client.try_clone(), upstream.try_clone())
+    else {
+        return;
+    };
+    let downstream = thread::spawn(move || pipe(upstream_read, client));
+    pipe(client_read, upstream);
+    let _ = downstream.join();
+}
+
+/// Copy until EOF or error, then half-close the write side so protocols
+/// layered on TCP (WebSocket close handshakes) terminate cleanly.
+fn pipe(mut from: TcpStream, mut to: TcpStream) {
+    let mut buf = [0u8; 16 * 1024];
+    loop {
+        match from.read(&mut buf) {
+            Ok(0) | Err(_) => break,
+            Ok(n) => {
+                if to.write_all(&buf[..n]).is_err() {
+                    break;
+                }
+            }
+        }
+    }
+    let _ = to.shutdown(Shutdown::Write);
+}

benchmarks/harbor-buzz-orchestra/manifests/m1-hello-world.yaml

@@ -0,0 +1,44 @@
+# M1 gate manifest: 1 orchestrator + 1 worker over one Buzz channel solving
+# Harbor's examples/tasks/hello-world via vanilla `harbor run`.
+#
+# Endpoints are LOCAL PLACEHOLDERS for the M1 wiring proof; the pilot/2.1
+# manifests will carry exact Databricks serving endpoint revisions and the
+# frozen price table per the canonical plan.
+schema_version: "1"
+condition: M1-hello-world
+roster:
+  - id: orch
+    kind: orchestrator
+    role: lead
+    count: 1
+    endpoint: local/placeholder-orchestrator
+    model_revision: m1-placeholder
+    prompt:
+      path: personas/orchestrator-m1.md
+      sha256: af64b515de914c99fbdfc5a50830909ac10ab3a4f37e8352a354019a6be74151
+    generation:
+      max_output_tokens: 4096
+      context_window_tokens: 128000
+  - id: worker
+    kind: worker
+    role: implementer
+    count: 1
+    endpoint: local/placeholder-worker
+    model_revision: m1-placeholder
+    prompt:
+      path: personas/worker-m1.md
+      sha256: f3fef3d2c42105c256ff019d71ef99506b31a23e8cb252cac727c8bc470eb304
+    generation:
+      max_output_tokens: 4096
+      context_window_tokens: 128000
+prices:
+  local/placeholder-orchestrator:
+    input_per_million_usd: 0
+    cached_input_per_million_usd: 0
+    output_per_million_usd: 0
+  local/placeholder-worker:
+    input_per_million_usd: 0
+    cached_input_per_million_usd: 0
+    output_per_million_usd: 0
+trial_budget:
+  timeout_seconds: 1800

benchmarks/harbor-buzz-orchestra/manifests/tb-cobol-sonnet-haiku.yaml

@@ -0,0 +1,42 @@
+# Terminal-Bench trial: 1 Sonnet 4.6 orchestrator + 2 Haiku 4.5 workers over
+# one Buzz channel. Endpoint names are exact Anthropic model IDs (the runtime
+# passes the endpoint name to the provider as the model).
+# Prices: Anthropic list $/Mtok as of 2026-07 (Sonnet 3/15, Haiku 1/5).
+schema_version: "1"
+condition: tb-cobol-sonnet46-2haiku45
+roster:
+  - id: orch
+    kind: orchestrator
+    role: lead
+    count: 1
+    endpoint: claude-sonnet-4-6
+    model_revision: claude-sonnet-4-6
+    prompt:
+      path: personas/orchestrator-tb.md
+      sha256: 206829331cdd85fb277266901b9489e190ba23097600a8bb69142d044b9a42e6
+    generation:
+      max_output_tokens: 4096
+      context_window_tokens: 200000
+  - id: worker
+    kind: worker
+    role: implementer
+    count: 2
+    endpoint: claude-haiku-4-5
+    model_revision: claude-haiku-4-5
+    prompt:
+      path: personas/worker-tb.md
+      sha256: 664a994ae3ba93f02ebbb8f61f781f6a59ea3327b21e26e7ffc2d06827ed3e45
+    generation:
+      max_output_tokens: 4096
+      context_window_tokens: 200000
+prices:
+  claude-sonnet-4-6:
+    input_per_million_usd: 3
+    cached_input_per_million_usd: 0.3
+    output_per_million_usd: 15
+  claude-haiku-4-5:
+    input_per_million_usd: 1
+    cached_input_per_million_usd: 0.1
+    output_per_million_usd: 5
+trial_budget:
+  timeout_seconds: 900

benchmarks/harbor-buzz-orchestra/personas/orchestrator-m1.md

@@ -0,0 +1,34 @@
+# Orchestrator — M1 hello-world
+
+You are the orchestrator of a small team solving a terminal task. You do not
+run commands yourself; workers do. You coordinate over a Buzz channel.
+
+Your `shell` tool has the `buzz` CLI on PATH, already authenticated as
+you. Nothing you write is visible to anyone unless you publish it: every
+message — step assignments, verification requests, the final `DONE:`
+— must be sent with
+`buzz messages send --channel <channel-id> --content <text>`. Your team, your
+channel id, and the user you report to are listed in the "Your team"
+section below. Your turn is not complete until you have published your
+message.
+
+Rules:
+1. Read the task instruction. Break it into the smallest concrete steps.
+2. Assign each step to a worker with an @mention. One step per message.
+   State the exact goal and the success check, not just the command to run.
+   The task runs in the worker's terminal working directory. Unless the task
+   instruction itself names a path, refer to files by bare relative name
+   (`hello.txt`) — never invent an absolute path. More generally, relay the
+   task's requirements verbatim and do not add constraints the task does not
+   state (paths, encodings, byte-level rules such as forbidding a trailing
+   newline). Where the task is silent, let standard tool defaults apply.
+3. Wait for the worker's report before assigning the next dependent step.
+4. When a worker reports output, verify it against the task's success
+   criteria yourself before moving on.
+5. When the task is complete, report back to the user: publish a final
+   message starting with `DONE:` that @mentions the user and summarizes
+   what was produced and how you verified it. The task is not finished
+   until this message is published — never conclude silently.
+
+Keep messages short. Never fabricate command output. If a worker's report is
+ambiguous, ask them to re-run with the exact verification command.

benchmarks/harbor-buzz-orchestra/personas/orchestrator-tb.md

@@ -0,0 +1,43 @@
+# Orchestrator — Terminal-Bench team
+
+You are the orchestrator of a small team solving a terminal task. You do not
+run commands yourself; your workers do. You coordinate over a Buzz channel.
+Your team, your channel id, and the user you report to are listed in the
+"Your team" section below.
+
+Your `shell` tool has the `buzz` CLI on PATH, already authenticated as
+you. Nothing you write is visible to anyone unless you publish it: every
+message — step assignments, verification requests, the final `DONE:`
+report — must be sent with
+`buzz messages send --channel <channel-id> --content <text>`. Your turn is
+not complete until you have published your message. Do not use the shell
+for task work — that is your workers' job.
+
+Tasks arrive as a channel message from the user @mentioning you. Address
+each assignment to a specific worker by @mention, exactly one worker per
+step. You may assign independent steps to different workers, but never give
+two workers overlapping or conflicting work — they share one task
+environment and one filesystem.
+
+Rules:
+1. Read the task instruction. Break it into the smallest concrete steps.
+2. Assign each step to a worker with an @mention. One step per message.
+   State the exact goal and the success check, not just the command to run.
+   Relay the task's requirements verbatim — use the paths the task states,
+   and do not add constraints the task does not state (paths, encodings,
+   byte-level rules). Where the task is silent, let standard tool defaults
+   apply.
+3. Wait for the worker's report before assigning the next dependent step.
+4. When a worker reports output, verify it against the task's success
+   criteria before moving on: assign a verification step that runs the
+   task's own success check and shows real output. Assign each verification
+   step to a different worker than the one whose work is being verified —
+   independent verification, never self-review. Do not report completion on
+   a worker's claim alone.
+5. When the task is complete and verified, report back to the user: publish
+   a final message starting with `DONE:` that @mentions the user and
+   summarizes what was produced and how it was verified. The task is not
+   finished until this message is published — never conclude silently.
+
+Keep messages short. Never fabricate command output. If a worker's report is
+ambiguous, ask them to re-run with the exact verification command.

benchmarks/harbor-buzz-orchestra/personas/worker-m1.md

@@ -0,0 +1,32 @@
+# Worker — M1 hello-world
+
+You are a worker agent with terminal access, coordinating over a Buzz channel.
+
+You work directly in the task environment: your `shell` tool runs
+commands in it, and your file tools read and edit its files. The same
+shell has the `buzz` CLI on PATH, already authenticated as you; reports go
+to the channel with
+`buzz messages send --channel <channel-id> --content <text>`. Your team and your
+channel id are listed in the "Your team" section below. Your turn is not
+complete until you have published your report.
+
+The orchestrator only wakes for messages that @mention it. Every report
+you publish must start with an @mention of the agent that assigned you the
+step (use their name exactly as it appears in their message). If the
+assignment's event id is visible in your context, also pass
+`--reply-to <event-id>` to thread the report. A report that mentions
+nobody is invisible and the task will stall.
+
+Rules:
+1. Act only on steps assigned to you by the orchestrator's @mention.
+2. Execute the requested step in the terminal. Prefer the smallest command
+   that achieves the stated goal. Create files in your terminal's current
+   working directory unless the task itself names a path. If an assignment
+   gives an absolute path your working directory does not contain, report
+   the mismatch instead of creating the directory — `mkdir -p` on an
+   invented path silently puts the file where the grader will never look.
+3. Report back in one message: the command you ran, its exit code, and the
+   relevant output (trimmed, never invented).
+4. If a command fails, report the failure verbatim and stop — do not
+   improvise a different approach without the orchestrator's direction.
+5. Never claim success without showing the verifying output.

benchmarks/harbor-buzz-orchestra/personas/worker-tb.md

@@ -0,0 +1,32 @@
+# Worker — Terminal-Bench team
+
+You are a worker agent with terminal access, coordinating over a Buzz
+channel. Your team, your channel id, and your orchestrator are listed in
+the "Your team" section below.
+
+You work directly in the task environment: your `shell` tool runs
+commands in it, and your file tools read and edit its files. The same
+shell has the `buzz` CLI on PATH, already authenticated as you; reports go
+to the channel with
+`buzz messages send --channel <channel-id> --content <text>`. Your turn is
+not complete until you have published your report.
+
+The orchestrator only wakes for messages that @mention it. Every report
+you publish must start with an @mention of the agent that assigned you the
+step (use their name exactly as it appears in their message). If the
+assignment's event id is visible in your context, also pass
+`--reply-to <event-id>` to thread the report. A report that mentions
+nobody is invisible and the task will stall.
+
+Rules:
+1. Act only on steps assigned to you by the orchestrator's @mention. If a
+   message assigns a step to a different worker, ignore it.
+2. Execute the requested step in the terminal BEFORE writing any report.
+   Never describe output you have not yet produced. Prefer the smallest
+   command that achieves the stated goal. Use the paths the task or the
+   assignment states; do not invent paths.
+3. Report back in one message: the command you ran, its exit code, and the
+   relevant output (trimmed, never invented).
+4. If a command fails, report the failure verbatim and stop — do not
+   improvise a different approach without the orchestrator's direction.
+5. Never claim success without showing the verifying output.

benchmarks/harbor-buzz-orchestra/pyproject.toml

@@ -0,0 +1,29 @@
+[project]
+name = "harbor-buzz-orchestra"
+version = "0.1.0"
+description = "Harbor custom agent for Buzz-coordinated multi-agent benchmarks"
+requires-python = ">=3.12"
+dependencies = [
+    "harbor>=0.16.1,<0.18",
+    "pydantic>=2.11",
+    "pyyaml>=6.0",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.4",
+    "pytest-asyncio>=1.2",
+    "ruff>=0.15",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+asyncio_mode = "auto"
+
+[tool.ruff]
+target-version = "py312"
+line-length = 88

benchmarks/harbor-buzz-orchestra/scripts/benchmark.py

@@ -0,0 +1,550 @@
+#!/usr/bin/env python3
+"""One-command benchmark: bring up the Buzz stack in Docker and run it.
+
+``just benchmark`` wraps this script. Defaults are leaderboard-eligible out
+of the box (Terminal-Bench 2.1, 5 attempts per problem, the Sonnet+Haiku
+team); every ``run_leaderboard.py`` selector passes through unchanged. The
+script owns everything around the run:
+
+- A dedicated ``buzz-benchmark`` compose project reusing the production
+  bundle (``deploy/compose/compose.yml``) plus the benchmark port overlay,
+  on its own ports (relay :3600, Postgres :5633, metrics :9602) so it never
+  collides with a dev stack. Secrets and identities are generated once into
+  the gitignored ``.benchmark/`` state dir and reused across runs.
+- One pinned *user* identity for the whole benchmark environment: it owns
+  every trial channel and posts every task, like one human running many
+  teams. Channels are kept (not archived) after each trial.
+- ``--gui`` adds that user to the relay membership list and opens the Buzz
+  desktop app logged in as them, so a human can watch the teams work live.
+
+Run inside the testbed environment (the just recipe does this):
+
+    uv run --project benchmarks/harbor-buzz-orchestra/testbed \
+        benchmarks/harbor-buzz-orchestra/scripts/benchmark.py [--gui] [...]
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import os
+import secrets
+import shutil
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+PACKAGE_ROOT = Path(__file__).resolve().parent.parent
+REPO_ROOT = PACKAGE_ROOT.parents[1]
+STATE_DIR = PACKAGE_ROOT / ".benchmark"
+
+COMPOSE_PROJECT = "buzz-benchmark"
+COMPOSE_FILES = (
+    REPO_ROOT / "deploy" / "compose" / "compose.yml",
+    PACKAGE_ROOT / "testbed" / "compose.benchmark.yml",
+)
+RELAY_HTTP_PORT = 3600
+PG_HOST_PORT = 5633
+METRICS_HOST_PORT = 9602
+GUI_BUNDLE_IDENTIFIER = "xyz.block.buzz.app.benchmark"
+
+DEFAULT_DATASET = "terminal-bench/terminal-bench-2-1"
+DEFAULT_ATTEMPTS = 5
+DEFAULT_MANIFEST = PACKAGE_ROOT / "manifests" / "tb-cobol-sonnet-haiku.yaml"
+DEFAULT_ENDPOINTS = PACKAGE_ROOT / "testbed" / "endpoints" / "anthropic-live.json"
+SCHEMA_SQL = PACKAGE_ROOT / "testbed" / "sql" / "benchmark_schema.sql"
+
+# Linux builds of the production agent stack, uploaded into each task
+# container per trial. Built once in a rust:alpine container (musl → fully
+# static, runs on any Linux task image of the same architecture) and cached.
+AGENT_BINARIES = ("buzz-acp", "buzz-agent", "buzz-dev-mcp")
+# Std-only loopback forwarder (not a workspace crate): agents dial the
+# relay's canonical localhost address inside the task container and the
+# forwarder bridges to the Docker host gateway. Compiled with plain rustc
+# in the same cross-build step.
+FORWARDER_SOURCE = PACKAGE_ROOT / "forwarder" / "relay_forwarder.rs"
+FORWARDER_BINARY = "relay-forwarder"
+LINUX_TARGET_DIR = STATE_DIR / "linux-target"
+RUST_IMAGE = "rust:1.95-alpine"
+
+_spec = importlib.util.spec_from_file_location(
+    "run_leaderboard", Path(__file__).resolve().parent / "run_leaderboard.py"
+)
+run_leaderboard = importlib.util.module_from_spec(_spec)
+sys.modules.setdefault("run_leaderboard", run_leaderboard)
+_spec.loader.exec_module(run_leaderboard)
+
+
+def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=__doc__.splitlines()[0],
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    problems = parser.add_mutually_exclusive_group()
+    problems.add_argument(
+        "--dataset", "-d", default=None,
+        help=f"Registry dataset (default: {DEFAULT_DATASET})",
+    )
+    problems.add_argument(
+        "--path", "-p", type=Path, help="Local task or dataset directory"
+    )
+    parser.add_argument(
+        "--include-task", "-i", action="append", default=[],
+        help="Task name to include (glob, repeatable)",
+    )
+    parser.add_argument(
+        "--exclude-task", "-x", action="append", default=[],
+        help="Task name to exclude (glob, repeatable)",
+    )
+    parser.add_argument(
+        "--attempts", "-k", type=int, default=DEFAULT_ATTEMPTS,
+        help=f"Runs per problem (default: {DEFAULT_ATTEMPTS}, the leaderboard requirement)",
+    )
+    parser.add_argument(
+        "--manifest", type=Path, default=DEFAULT_MANIFEST,
+        help=f"Team manifest YAML (default: {DEFAULT_MANIFEST.name})",
+    )
+    parser.add_argument(
+        "--endpoint-config", type=Path, default=DEFAULT_ENDPOINTS,
+        help=f"Endpoint provider/API-key mapping (default: {DEFAULT_ENDPOINTS.name})",
+    )
+    parser.add_argument("--n-concurrent", "-n", type=int, default=4, help="Concurrent trials")
+    parser.add_argument(
+        "--jobs-dir", type=Path, default=PACKAGE_ROOT / "jobs", help="Job output root"
+    )
+    parser.add_argument("--job-name", default=None, help="Job name (default: lb-<condition>-<UTC>)")
+    parser.add_argument(
+        "--upload", action="store_true", help="Upload to Harbor Hub when the job finishes"
+    )
+    parser.add_argument(
+        "--gui", action="store_true",
+        help="Open the Buzz desktop app as the benchmark user to watch the run live",
+    )
+    parser.add_argument(
+        "--fresh", action="store_true",
+        help="Reset first: drop the stack's Docker volumes and the benchmark "
+             "GUI's app state (keys in state.json are kept)",
+    )
+    parser.add_argument(
+        "--dry-run", action="store_true",
+        help="Print the underlying harbor command and exit (no stack bring-up)",
+    )
+    return parser.parse_args(argv)
+
+
+# -- state: secrets and identities, generated once --------------------------
+
+
+def load_state() -> dict[str, str]:
+    """Generate-or-load the benchmark environment's keys and secrets."""
+    from harbor_buzz_testbed.keys import generate_keypair, keypair_from_secret
+
+    STATE_DIR.mkdir(mode=0o700, exist_ok=True)
+    state_path = STATE_DIR / "state.json"
+    if state_path.is_file():
+        state = json.loads(state_path.read_text())
+    else:
+        owner = generate_keypair()
+        user = generate_keypair()
+        state = {
+            "owner_secret_key": owner.secret_key,
+            "user_secret_key": user.secret_key,
+            "postgres_password": secrets.token_urlsafe(24),
+            "redis_password": secrets.token_urlsafe(24),
+            "typesense_api_key": secrets.token_hex(16),
+            "s3_access_key": secrets.token_hex(10),
+            "s3_secret_key": secrets.token_hex(20),
+            "git_hook_hmac_secret": secrets.token_hex(32),
+            "relay_private_key": generate_keypair().secret_key,
+        }
+        state_path.touch(mode=0o600)
+        state_path.write_text(json.dumps(state, indent=2))
+    state["owner_pubkey"] = keypair_from_secret(state["owner_secret_key"]).pubkey
+    state["user_pubkey"] = keypair_from_secret(state["user_secret_key"]).pubkey
+    return state
+
+
+def print_user_identity(state: dict[str, str]) -> None:
+    """Show the pinned benchmark user's key so a human can import it during
+    the desktop GUI's onboarding (this stack is local-only; the key guards
+    nothing beyond it)."""
+    from harbor_buzz_testbed.keys import encode_nsec
+
+    print(
+        f"benchmark user pubkey: {state['user_pubkey']}\n"
+        f"benchmark user nsec:   {encode_nsec(state['user_secret_key'])} "
+        "(import this in the GUI onboarding to watch as the benchmark user)"
+    )
+
+
+def write_env_file(state: dict[str, str]) -> Path:
+    """Compose interpolation env — regenerated from state on every run."""
+    env_path = STATE_DIR / ".env"
+    lines = {
+        "BUZZ_IMAGE": os.environ.get("BUZZ_IMAGE", "ghcr.io/block/buzz:main"),
+        "BUZZ_DOMAIN": "localhost",
+        "RELAY_URL": f"ws://localhost:{RELAY_HTTP_PORT}",
+        "BUZZ_MEDIA_BASE_URL": f"http://localhost:{RELAY_HTTP_PORT}/media",
+        "BUZZ_MEDIA_SERVER_DOMAIN": "localhost",
+        "BUZZ_CORS_ORIGINS": f"http://localhost:{RELAY_HTTP_PORT}",
+        "BUZZ_REQUIRE_AUTH_TOKEN": "true",
+        "BUZZ_REQUIRE_RELAY_MEMBERSHIP": "true",
+        "BUZZ_ALLOW_NIP_OA_AUTH": "true",
+        "BUZZ_AUTO_MIGRATE": "true",
+        "BUZZ_GIT_CONFORMANCE_PROBE": "true",
+        "RUST_LOG": "buzz_relay=info,buzz_db=info,buzz_auth=info",
+        "RELAY_OWNER_PUBKEY": state["owner_pubkey"],
+        "BUZZ_RELAY_PRIVATE_KEY": state["relay_private_key"],
+        "BUZZ_GIT_HOOK_HMAC_SECRET": state["git_hook_hmac_secret"],
+        "POSTGRES_DB": "buzz",
+        "POSTGRES_USER": "buzz",
+        "POSTGRES_PASSWORD": state["postgres_password"],
+        "REDIS_PASSWORD": state["redis_password"],
+        "TYPESENSE_API_KEY": state["typesense_api_key"],
+        "BUZZ_S3_ACCESS_KEY": state["s3_access_key"],
+        "BUZZ_S3_SECRET_KEY": state["s3_secret_key"],
+        "BUZZ_S3_BUCKET": "buzz-media",
+        "BUZZ_HTTP_PORT": str(RELAY_HTTP_PORT),
+        "BUZZ_PG_HOST_PORT": str(PG_HOST_PORT),
+        "BUZZ_METRICS_HOST_PORT": str(METRICS_HOST_PORT),
+    }
+    env_path.touch(mode=0o600)
+    env_path.write_text("".join(f"{k}={v}\n" for k, v in lines.items()))
+    return env_path
+
+
+def postgres_dsn(state: dict[str, str]) -> str:
+    return (
+        f"postgresql://buzz:{state['postgres_password']}"
+        f"@127.0.0.1:{PG_HOST_PORT}/buzz"
+    )
+
+
+def write_provisioner_config(
+    state: dict[str, str], endpoint_config: Path
+) -> Path:
+    """Resolve per-endpoint API keys from the environment and write the
+    provisioner config: pinned user, keep-channels teardown."""
+    endpoints = json.loads(endpoint_config.read_text())
+    llm_api_keys: dict[str, str] = {}
+    for name, entry in endpoints.items():
+        env_var = entry["api_key_env"]
+        key = os.environ.get(env_var)
+        if not key:
+            raise SystemExit(
+                f"endpoint {name!r} needs the {env_var} environment variable"
+            )
+        llm_api_keys[name] = key
+    config = {
+        "relay_http_url": f"http://localhost:{RELAY_HTTP_PORT}",
+        # The agents dial the relay's CANONICAL address — the relay is
+        # host-header tenant-bound, and its community row is the authority
+        # of RELAY_URL (localhost:3600). Inside the task container that
+        # loopback address is served by a tiny forwarder bridging to the
+        # Docker host gateway (see --relay-gateway below).
+        "relay_ws_url": f"ws://localhost:{RELAY_HTTP_PORT}",
+        "owner_secret_key": state["owner_secret_key"],
+        "postgres_dsn": postgres_dsn(state),
+        "llm_api_keys": llm_api_keys,
+        "user_secret_key": state["user_secret_key"],
+        "archive_on_teardown": False,
+    }
+    path = STATE_DIR / "provisioner.json"
+    path.touch(mode=0o600)
+    path.write_text(json.dumps(config, indent=2))
+    return path
+
+
+# -- docker stack ------------------------------------------------------------
+
+
+def compose_command(*args: str) -> list[str]:
+    command = [
+        "docker", "compose",
+        "--project-name", COMPOSE_PROJECT,
+        "--project-directory", str(STATE_DIR),
+        "--env-file", str(STATE_DIR / ".env"),
+    ]
+    for file in COMPOSE_FILES:
+        command += ["-f", str(file)]
+    return command + list(args)
+
+
+def stale_credential_volume(state: dict[str, str]) -> bool:
+    """True when Postgres is up but rejects THIS clone's password — the
+    volume was initialized by another checkout's ``.benchmark/`` state
+    (compose project name is machine-global, state dir is per-clone)."""
+    import psycopg
+
+    try:
+        psycopg.connect(postgres_dsn(state), connect_timeout=5).close()
+    except psycopg.OperationalError as error:
+        return "password authentication failed" in str(error)
+    return False
+
+
+def bring_up_stack(state: dict[str, str]) -> None:
+    """Compose bring-up (idempotent), self-healing the one known-fatal
+    failure: a stale Postgres volume from a different clone. Nothing in
+    that volume is usable (we can't even authenticate to it), so drop the
+    volumes and retry once rather than aborting with instructions."""
+    try:
+        subprocess.run(compose_command("up", "-d", "--wait"), check=True)
+    except subprocess.CalledProcessError:
+        if not stale_credential_volume(state):
+            raise
+        print(
+            "benchmark Postgres volume was initialized by a different "
+            "checkout's .benchmark/ state — dropping the stale volumes and "
+            "retrying..."
+        )
+        subprocess.run(compose_command("down", "-v"), check=True)
+        subprocess.run(compose_command("up", "-d", "--wait"), check=True)
+
+
+def reset_environment() -> None:
+    """--fresh: drop the stack's Docker volumes and the benchmark GUI's
+    app state, together — GUI records (workspaces, read state) only stay
+    coherent as long as the database they reference exists. Keys in
+    ``state.json`` are kept, so the same nsec works after the reset."""
+    subprocess.run(compose_command("down", "-v"), check=True)
+    if sys.platform == "darwin":
+        for domain in ("WebKit", "Caches", "Application Support"):
+            shutil.rmtree(
+                Path.home() / "Library" / domain / GUI_BUNDLE_IDENTIFIER,
+                ignore_errors=True,
+            )
+
+
+def ensure_stack(state: dict[str, str]) -> None:
+    """Bring the compose stack up (idempotent) and apply the benchmark schema."""
+    import psycopg
+
+    bring_up_stack(state)
+
+    deadline = time.monotonic() + 60
+    last_error: Exception | None = None
+    while time.monotonic() < deadline:
+        try:
+            with psycopg.connect(postgres_dsn(state)) as conn:
+                conn.execute(SCHEMA_SQL.read_text())
+                conn.commit()
+            return
+        except psycopg.Error as error:  # containers healthy but PG settling
+            last_error = error
+            time.sleep(2)
+    raise SystemExit(f"benchmark schema apply failed: {last_error}")
+
+
+# -- buzz binaries -----------------------------------------------------------
+
+
+def ensure_binaries() -> dict[str, Path]:
+    """Find the host buzz CLI, building it once if missing."""
+    try:
+        return run_leaderboard.find_binaries(None)
+    except SystemExit:
+        print("host buzz CLI missing — building (cargo build, first run only)...")
+    cargo = REPO_ROOT / "bin" / "cargo"
+    subprocess.run(
+        [str(cargo), "build", "-p", "buzz-cli"],
+        cwd=REPO_ROOT,
+        check=True,
+    )
+    return run_leaderboard.find_binaries(None)
+
+
+def linux_triple() -> str:
+    """The musl triple matching the Docker engine that runs task containers."""
+    arch = subprocess.run(
+        ["docker", "version", "--format", "{{.Server.Arch}}"],
+        capture_output=True, text=True, check=True,
+    ).stdout.strip()
+    try:
+        return {
+            "arm64": "aarch64-unknown-linux-musl",
+            "amd64": "x86_64-unknown-linux-musl",
+        }[arch]
+    except KeyError:
+        raise SystemExit(f"unsupported Docker architecture: {arch!r}") from None
+
+
+def ensure_agent_binaries() -> Path:
+    """Cross-build the static Linux agent stack once, cached in .benchmark/.
+
+    The agents run *inside* each Harbor task container as the real
+    buzz-acp → buzz-agent → buzz-dev-mcp stack, so the binaries must be
+    Linux ELF for the task image architecture. musl-static means they run
+    on any Linux base image (glibc or not). The relay loopback forwarder
+    is compiled in the same step with plain rustc (std-only, no deps).
+    """
+    triple = linux_triple()
+    bin_dir = LINUX_TARGET_DIR / triple / "release"
+    targets = AGENT_BINARIES + (FORWARDER_BINARY,)
+    if all((bin_dir / name).is_file() for name in targets):
+        return bin_dir
+    print(f"Linux agent binaries missing — cross-building for {triple} "
+          f"in {RUST_IMAGE} (first run only, ~2 min)...")
+    LINUX_TARGET_DIR.mkdir(parents=True, exist_ok=True)
+    (STATE_DIR / "cargo-registry").mkdir(exist_ok=True)
+    packages = [arg for name in AGENT_BINARIES for arg in ("-p", name)]
+    forwarder_src = FORWARDER_SOURCE.relative_to(REPO_ROOT)
+    subprocess.run(
+        [
+            "docker", "run", "--rm",
+            "-v", f"{REPO_ROOT}:/src:ro",
+            "-v", f"{LINUX_TARGET_DIR}:/target",
+            "-v", f"{STATE_DIR / 'cargo-registry'}:/usr/local/cargo/registry",
+            "-e", "CARGO_TARGET_DIR=/target",
+            "-w", "/src",
+            RUST_IMAGE,
+            "sh", "-c",
+            "apk add --no-cache musl-dev >/dev/null && "
+            f"cargo build --release --locked --target {triple} "
+            + " ".join(packages)
+            + f" && rustc --edition 2021 -O --target {triple}"
+            f" -o /target/{triple}/release/{FORWARDER_BINARY}"
+            f" /src/{forwarder_src}",
+        ],
+        check=True,
+    )
+    missing = [n for n in targets if not (bin_dir / n).is_file()]
+    if missing:
+        raise SystemExit(f"cross-build produced no {', '.join(missing)} in {bin_dir}")
+    return bin_dir
+
+
+# -- GUI ---------------------------------------------------------------------
+
+
+def launch_gui(state: dict[str, str]) -> subprocess.Popen:
+    """Open the Buzz desktop app logged in as the benchmark user.
+
+    The relay runs closed (membership required), so the user pubkey is first
+    added to the relay membership list via buzz-admin inside the container —
+    NIP-OA auth tags cover the agents, but the GUI authenticates as a plain
+    member, exactly like a human.
+    """
+    subprocess.run(
+        compose_command(
+            "exec", "-T", "relay",
+            "buzz-admin", "add-member", "--pubkey", state["user_pubkey"],
+        ),
+        check=True,
+    )
+
+    desktop_dir = REPO_ROOT / "desktop"
+    if not (desktop_dir / "node_modules").is_dir():
+        subprocess.run(["pnpm", "install"], cwd=desktop_dir, check=True)
+
+    # tauri dev needs sidecar files present; stub them and drop in the real
+    # CLI binary (mirrors the just staging recipe).
+    target = subprocess.run(
+        ["rustc", "-vV"], capture_output=True, text=True, check=True
+    ).stdout
+    triple = next(
+        line.split(": ", 1)[1] for line in target.splitlines() if line.startswith("host: ")
+    )
+    sidecar_dir = desktop_dir / "src-tauri" / "binaries"
+    sidecar_dir.mkdir(parents=True, exist_ok=True)
+    binaries = ensure_binaries()
+    for name in ("buzz-acp", "buzz-agent", "buzz-dev-mcp", "git-credential-nostr", "buzz"):
+        stub = sidecar_dir / f"{name}-{triple}"
+        if not stub.exists():
+            stub.touch()
+    real_cli = sidecar_dir / f"buzz-{triple}"
+    real_cli.write_bytes(binaries["buzz"].read_bytes())
+    real_cli.chmod(0o755)
+
+    print(
+        f"Opening Buzz GUI as the benchmark user ({state['user_pubkey'][:16]}…).\n"
+        "Watch, don't type — a message from you mid-trial would taint the run."
+    )
+    # Distinct bundle identifier: the desktop app persists workspaces (incl.
+    # their relay URLs) in per-identifier WebKit localStorage, and a stored
+    # workspace's relay URL overrides BUZZ_RELAY_URL by design. Reusing the
+    # default identifier means any past local-dev session's ws://localhost:3000
+    # workspace silently shadows the benchmark relay. An identifier of our own
+    # keeps that state isolated both ways.
+    tauri_config = json.dumps(
+        {"identifier": GUI_BUNDLE_IDENTIFIER, "productName": "Buzz Benchmark"}
+    )
+    return subprocess.Popen(
+        ["pnpm", "exec", "tauri", "dev", "--config", tauri_config],
+        cwd=desktop_dir,
+        env={
+            **os.environ,
+            "BUZZ_RELAY_URL": f"ws://localhost:{RELAY_HTTP_PORT}",
+            "BUZZ_PRIVATE_KEY": state["user_secret_key"],
+        },
+    )
+
+
+# -- main ---------------------------------------------------------------------
+
+
+def leaderboard_argv(
+    args: argparse.Namespace, provisioner_config: Path, agent_bin_dir: Path
+) -> list[str]:
+    argv: list[str] = []
+    if args.path:
+        argv += ["--path", str(args.path)]
+    else:
+        argv += ["--dataset", args.dataset or DEFAULT_DATASET]
+    for pattern in args.include_task:
+        argv += ["--include-task", pattern]
+    for pattern in args.exclude_task:
+        argv += ["--exclude-task", pattern]
+    argv += [
+        "--attempts", str(args.attempts),
+        "--manifest", str(args.manifest),
+        "--endpoint-config", str(args.endpoint_config),
+        "--provisioner-config", str(provisioner_config),
+        "--agent-bin-dir", str(agent_bin_dir),
+        # The relay as reachable from inside a task container: Docker's
+        # host alias, bridged to the canonical localhost address by the
+        # uploaded forwarder. Override the alias with
+        # BUZZ_BENCHMARK_DOCKER_HOST if your engine exposes the host
+        # differently.
+        "--relay-gateway",
+        f"{os.environ.get('BUZZ_BENCHMARK_DOCKER_HOST', 'host.docker.internal')}"
+        f":{RELAY_HTTP_PORT}",
+        "--n-concurrent", str(args.n_concurrent),
+        "--jobs-dir", str(args.jobs_dir),
+    ]
+    if args.job_name:
+        argv += ["--job-name", args.job_name]
+    if args.upload:
+        argv.append("--upload")
+    if args.dry_run:
+        argv.append("--dry-run")
+    return argv
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = parse_args(argv)
+    state = load_state()
+    print_user_identity(state)
+    write_env_file(state)
+    provisioner_config = write_provisioner_config(state, args.endpoint_config)
+
+    if args.dry_run:
+        agent_bin_dir = LINUX_TARGET_DIR / linux_triple() / "release"
+    else:
+        ensure_binaries()
+        agent_bin_dir = ensure_agent_binaries()
+        if args.fresh:
+            reset_environment()
+        ensure_stack(state)
+        if args.gui:
+            launch_gui(state)
+
+    return run_leaderboard.main(
+        leaderboard_argv(args, provisioner_config, agent_bin_dir)
+    )
+
+
+if __name__ == "__main__":
+    sys.exit(main())

benchmarks/harbor-buzz-orchestra/scripts/run_leaderboard.py

@@ -0,0 +1,271 @@
+#!/usr/bin/env python3
+"""Run a problem set with a team manifest and produce leaderboard-ready results.
+
+One command wraps ``harbor run`` with only leaderboard-legal settings — no
+timeout or resource overrides are accepted or forwarded, so the resulting job
+directory passes Harbor's static validation as produced. After the run it
+writes a ``metadata.yaml`` template derived from the manifest and prints the
+exact upload/submit commands.
+
+Run inside the testbed environment so ``harbor`` and the adapter are
+importable:
+
+    uv run --project benchmarks/harbor-buzz-orchestra/testbed \
+        benchmarks/harbor-buzz-orchestra/scripts/run_leaderboard.py \
+        --dataset terminal-bench/terminal-bench-2-1 \
+        --attempts 5 \
+        --manifest benchmarks/harbor-buzz-orchestra/manifests/<TEAM>.yaml \
+        --endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/<ENDPOINTS>.json \
+        --provisioner-config <PROVISIONER.json> \
+        --agent-bin-dir <DIR with Linux buzz-acp/buzz-agent/buzz-dev-mcp>
+"""
+
+from __future__ import annotations
+
+import argparse
+import datetime as dt
+import json
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+import yaml
+
+PACKAGE_ROOT = Path(__file__).resolve().parent.parent
+AGENT_IMPORT = "harbor_buzz_orchestra:BuzzOrchestraAgent"
+PROVISIONER_FACTORY = "harbor_buzz_testbed:provisioner_from_dict"
+# Host-side: the harness speaks to the relay as the trial user via this CLI.
+BINARIES = ("buzz",)
+# Container-side: the production stack uploaded into each task container.
+# These must be Linux builds matching the task image architecture.
+AGENT_BINARIES = ("buzz-acp", "buzz-agent", "buzz-dev-mcp")
+# Uploaded alongside the stack when --relay-gateway is set: bridges the
+# agents' canonical relay address to the host gateway (the relay is
+# host-header tenant-bound, so agents must present its canonical Host).
+FORWARDER_BINARY = "relay-forwarder"
+
+PROVIDER_ORGS = {"anthropic": "Anthropic", "openai": "OpenAI", "databricks": "Databricks"}
+
+
+def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=__doc__.splitlines()[0], formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    problems = parser.add_mutually_exclusive_group(required=True)
+    problems.add_argument(
+        "--dataset", "-d", help="Registry dataset (e.g. terminal-bench/terminal-bench-2-1)"
+    )
+    problems.add_argument(
+        "--path", "-p", type=Path, help="Local task or dataset directory"
+    )
+    parser.add_argument(
+        "--include-task", "-i", action="append", default=[],
+        help="Task name to include from the dataset (glob, repeatable)",
+    )
+    parser.add_argument(
+        "--exclude-task", "-x", action="append", default=[],
+        help="Task name to exclude from the dataset (glob, repeatable)",
+    )
+    parser.add_argument(
+        "--attempts", "-k", type=int, required=True,
+        help="Runs per problem (leaderboards require 5)",
+    )
+    parser.add_argument("--manifest", type=Path, required=True, help="Team manifest YAML")
+    parser.add_argument(
+        "--endpoint-config", type=Path, required=True,
+        help="JSON mapping manifest endpoint names to providers/API keys",
+    )
+    parser.add_argument(
+        "--provisioner-config", type=Path, required=True,
+        help="JSON config for the Buzz relay/Postgres provisioner",
+    )
+    parser.add_argument(
+        "--buzz-bin-dir", type=Path, default=None,
+        help="Directory with the host buzz CLI (default: repo target/release, then target/debug)",
+    )
+    parser.add_argument(
+        "--agent-bin-dir", type=Path, required=True,
+        help="Directory with Linux builds of buzz-acp/buzz-agent/buzz-dev-mcp "
+        "to upload into each task container",
+    )
+    parser.add_argument(
+        "--relay-gateway", default="",
+        help="host:port of the benchmark relay as reachable from inside the "
+        "task container (e.g. host.docker.internal:3600). When set, a "
+        "loopback forwarder from --agent-bin-dir bridges the canonical "
+        "relay address to this gateway",
+    )
+    parser.add_argument("--n-concurrent", "-n", type=int, default=4, help="Concurrent trials")
+    parser.add_argument("--jobs-dir", type=Path, default=Path("jobs"), help="Job output root")
+    parser.add_argument("--job-name", default=None, help="Job name (default: lb-<condition>-<UTC>)")
+    parser.add_argument(
+        "--upload", action="store_true", help="Upload to Harbor Hub when the job finishes"
+    )
+    parser.add_argument("--dry-run", action="store_true", help="Print the harbor command and exit")
+    return parser.parse_args(argv)
+
+
+def find_binaries(bin_dir: Path | None) -> dict[str, Path]:
+    candidates = (
+        [bin_dir]
+        if bin_dir is not None
+        else [PACKAGE_ROOT.parents[1] / "target" / kind for kind in ("release", "debug")]
+    )
+    for candidate in candidates:
+        found = {name: candidate / name for name in BINARIES}
+        if all(path.is_file() for path in found.values()):
+            return found
+    searched = ", ".join(str(c) for c in candidates)
+    raise SystemExit(
+        f"buzz binaries not found (need {', '.join(BINARIES)}; searched {searched}). "
+        "Build them with `cargo build` or pass --buzz-bin-dir."
+    )
+
+
+def find_agent_binaries(bin_dir: Path, with_forwarder: bool = False) -> dict[str, Path]:
+    """The Linux agent stack uploaded into each task container."""
+    names = AGENT_BINARIES + ((FORWARDER_BINARY,) if with_forwarder else ())
+    found = {name: bin_dir / name for name in names}
+    missing = [name for name, path in found.items() if not path.is_file()]
+    if missing:
+        raise SystemExit(
+            f"Linux agent binaries not found in {bin_dir}: {', '.join(missing)}. "
+            "`just benchmark` builds them; or cross-compile with "
+            "cargo --target <arch>-unknown-linux-musl and pass --agent-bin-dir."
+        )
+    return found
+
+
+def build_command(
+    args: argparse.Namespace,
+    binaries: dict[str, Path],
+    agent_binaries: dict[str, Path],
+) -> list[str]:
+    """Compose the harbor invocation. Standard settings only: any timeout or
+    resource override would fail leaderboard static validation, so none are
+    accepted or forwarded."""
+    command = [
+        "harbor", "run", "--yes",
+        "--job-name", args.job_name,
+        "--jobs-dir", str(args.jobs_dir),
+        "-k", str(args.attempts),
+        "--n-concurrent", str(args.n_concurrent),
+    ]
+    if args.dataset:
+        command += ["--dataset", args.dataset]
+    else:
+        command += ["--path", str(args.path)]
+    for pattern in args.include_task:
+        command += ["--include-task-name", pattern]
+    for pattern in args.exclude_task:
+        command += ["--exclude-task-name", pattern]
+    if args.upload:
+        command.append("--upload")
+    command += ["--agent", AGENT_IMPORT]
+    kwargs = {
+        "manifest": args.manifest,
+        "provisioner_factory": PROVISIONER_FACTORY,
+        "provisioner_config": args.provisioner_config,
+        "artifact_root": PACKAGE_ROOT,
+        "endpoint_config": args.endpoint_config,
+        "buzz_acp_binary": agent_binaries["buzz-acp"],
+        "buzz_agent_binary": agent_binaries["buzz-agent"],
+        "buzz_dev_mcp_binary": agent_binaries["buzz-dev-mcp"],
+        "buzz_cli_binary": binaries["buzz"],
+        "run_id": args.job_name,
+    }
+    if args.relay_gateway:
+        kwargs["relay_gateway"] = args.relay_gateway
+        kwargs["forwarder_binary"] = agent_binaries[FORWARDER_BINARY]
+    for key, value in kwargs.items():
+        command += ["--agent-kwarg", f"{key}={value}"]
+    return command
+
+
+def write_metadata_template(args: argparse.Namespace, job_dir: Path) -> Path:
+    """Derive a metadata.yaml template (harbor.leaderboard.metadata schema)
+    from the manifest roster; display-name placeholders are for the submitter
+    to confirm."""
+    manifest = yaml.safe_load(args.manifest.read_text())
+    endpoints = json.loads(args.endpoint_config.read_text())
+    models, seen = [], set()
+    for entry in manifest.get("roster", []):
+        model = entry.get("model_revision") or entry["endpoint"]
+        if model in seen:
+            continue
+        seen.add(model)
+        provider = endpoints.get(entry["endpoint"], {}).get("provider", "FILL_ME")
+        models.append(
+            {
+                "model_name": model,
+                "model_provider": provider,
+                "model_display_name": model,
+                "model_org_display_name": PROVIDER_ORGS.get(provider, "FILL_ME"),
+            }
+        )
+    metadata = {
+        "agent_url": "https://github.com/block/buzz",
+        "agent_display_name": f"Buzz Orchestra ({manifest.get('condition', 'team')})",
+        "agent_org_display_name": "Block",
+        "models": models,
+    }
+    path = job_dir / "metadata.yaml"
+    path.write_text(yaml.safe_dump(metadata, sort_keys=False))
+    return path
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = parse_args(argv)
+    for label, path in (
+        ("manifest", args.manifest),
+        ("endpoint config", args.endpoint_config),
+        ("provisioner config", args.provisioner_config),
+    ):
+        if not path.is_file():
+            raise SystemExit(f"{label} not found: {path}")
+    if args.job_name is None:
+        condition = yaml.safe_load(args.manifest.read_text()).get("condition", "team")
+        stamp = dt.datetime.now(dt.UTC).strftime("%Y%m%dT%H%M%SZ")
+        args.job_name = f"lb-{condition}-{stamp}"
+
+    if args.dry_run:
+        # Dry runs print the command without requiring built binaries.
+        bin_dir = args.buzz_bin_dir or PACKAGE_ROOT.parents[1] / "target" / "release"
+        binaries = {name: bin_dir / name for name in BINARIES}
+        agent_binaries = {
+            name: args.agent_bin_dir / name
+            for name in AGENT_BINARIES + (FORWARDER_BINARY,)
+        }
+        print(" ".join(build_command(args, binaries, agent_binaries)))
+        return 0
+    binaries = find_binaries(args.buzz_bin_dir)
+    agent_binaries = find_agent_binaries(
+        args.agent_bin_dir, with_forwarder=bool(args.relay_gateway)
+    )
+    command = build_command(args, binaries, agent_binaries)
+    if shutil.which("harbor") is None:
+        raise SystemExit(
+            "harbor not on PATH — run via: uv run --project "
+            f"{PACKAGE_ROOT / 'testbed'} {Path(__file__).resolve()} ..."
+        )
+
+    result = subprocess.run(command)
+    job_dir = args.jobs_dir / args.job_name
+    if result.returncode != 0:
+        print(f"harbor run failed (exit {result.returncode}); job dir: {job_dir}")
+        return result.returncode
+
+    metadata_path = write_metadata_template(args, job_dir)
+    print("\nLeaderboard-ready job complete.")
+    print(f"  1. Review submitter details in {metadata_path}")
+    print(f"  2. harbor upload {job_dir}")
+    print(
+        "  3. harbor leaderboard submit -l terminal-bench/terminal-bench-2-1 "
+        f"-j <job UUID from upload> -m {metadata_path}"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

benchmarks/harbor-buzz-orchestra/src/harbor_buzz_orchestra/__init__.py

@@ -0,0 +1,25 @@
+"""Buzz orchestra custom agent for Harbor."""
+
+from .agent import BuzzOrchestraAgent
+from .manifest import ExperimentManifest, ManifestError
+from .provisioning import AgentCredential, TrialHandle, TrialProvisioner
+from .runtime import OrchestraRuntime, RuntimeResult
+from .container_runtime import (
+    BuzzContainerRuntime,
+    EndpointLaunchConfig,
+    RuntimeLaunchError,
+)
+
+__all__ = [
+    "AgentCredential",
+    "BuzzOrchestraAgent",
+    "BuzzContainerRuntime",
+    "EndpointLaunchConfig",
+    "ExperimentManifest",
+    "ManifestError",
+    "OrchestraRuntime",
+    "RuntimeResult",
+    "RuntimeLaunchError",
+    "TrialHandle",
+    "TrialProvisioner",
+]

benchmarks/harbor-buzz-orchestra/src/harbor_buzz_orchestra/agent.py

@@ -0,0 +1,200 @@
+"""Harbor custom-agent entry point for Buzz orchestration."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from harbor.agents.base import BaseAgent
+from harbor.environments.base import BaseEnvironment
+from harbor.models.agent.context import AgentContext
+
+from .manifest import ExperimentManifest
+from .provisioning import TrialProvisioner
+from .runtime import OrchestraRuntime
+from .container_runtime import BuzzContainerRuntime, EndpointLaunchConfig
+
+
+class BuzzOrchestraAgent(BaseAgent):
+    """Coordinate an arbitrary manifest-defined team through a Buzz trial."""
+
+    # Set True only once the runtime writes a validated agent/trajectory.json.
+    SUPPORTS_ATIF = False
+
+    def __init__(
+        self,
+        logs_dir: Path,
+        model_name: str | None = None,
+        *,
+        manifest: str | Path | dict[str, Any],
+        provisioner: TrialProvisioner | None = None,
+        runtime: OrchestraRuntime | None = None,
+        provisioner_factory: str | None = None,
+        provisioner_config: str | Path | dict[str, Any] | None = None,
+        artifact_root: str | Path | None = None,
+        endpoint_config: str | Path | dict[str, Any] | None = None,
+        buzz_acp_binary: str = "buzz-acp",
+        buzz_agent_binary: str = "buzz-agent",
+        buzz_dev_mcp_binary: str = "buzz-dev-mcp",
+        buzz_cli_binary: str = "buzz",
+        relay_gateway: str = "",
+        forwarder_binary: str = "relay-forwarder",
+        run_id: str | None = None,
+        **kwargs: Any,
+    ) -> None:
+        super().__init__(logs_dir=logs_dir, model_name=model_name, **kwargs)
+        self.manifest = ExperimentManifest.load(manifest)
+        self.provisioner = provisioner or self._build_provisioner(
+            provisioner_factory, provisioner_config
+        )
+        self.runtime = runtime or self._build_runtime(
+            logs_dir,
+            artifact_root,
+            endpoint_config,
+            buzz_acp_binary,
+            buzz_agent_binary,
+            buzz_dev_mcp_binary,
+            buzz_cli_binary,
+            relay_gateway,
+            forwarder_binary,
+        )
+        self.run_id = run_id
+
+    @staticmethod
+    def name() -> str:
+        return "buzz-orchestra"
+
+    def version(self) -> str:
+        return "0.1.0"
+
+    @staticmethod
+    def _load_mapping(
+        source: str | Path | dict[str, Any] | None,
+    ) -> dict[str, Any] | None:
+        if source is None:
+            return None
+        if isinstance(source, dict):
+            return source
+        import json
+
+        path = Path(source).expanduser()
+        try:
+            value = json.loads(path.read_text(encoding="utf-8"))
+        except (OSError, json.JSONDecodeError) as error:
+            raise ValueError(f"cannot load JSON config {path}: {error}") from error
+        if not isinstance(value, dict):
+            raise ValueError(f"JSON config {path} must contain an object")
+        return value
+
+    @classmethod
+    def _build_provisioner(
+        cls,
+        factory_path: str | None,
+        config_source: str | Path | dict[str, Any] | None,
+    ) -> TrialProvisioner | None:
+        config = cls._load_mapping(config_source)
+        if factory_path is None and config is None:
+            return None
+        if factory_path is None or config is None:
+            raise ValueError(
+                "provisioner_factory and provisioner_config must be provided together"
+            )
+        from harbor.utils.import_path import import_symbol
+
+        factory = import_symbol(factory_path)
+        return factory(config)
+
+    @classmethod
+    def _build_runtime(
+        cls,
+        logs_dir: Path,
+        artifact_root: str | Path | None,
+        endpoint_source: str | Path | dict[str, Any] | None,
+        buzz_acp_binary: str,
+        buzz_agent_binary: str,
+        buzz_dev_mcp_binary: str,
+        buzz_cli_binary: str,
+        relay_gateway: str,
+        forwarder_binary: str,
+    ) -> OrchestraRuntime | None:
+        endpoint_data = cls._load_mapping(endpoint_source)
+        if endpoint_data is None and artifact_root is None:
+            return None
+        if endpoint_data is None or artifact_root is None:
+            raise ValueError(
+                "artifact_root and endpoint_config must be provided together"
+            )
+        endpoints = {
+            name: EndpointLaunchConfig(
+                provider=value["provider"],
+                api_key_env=value["api_key_env"],
+                env=value.get("env", {}),
+            )
+            for name, value in endpoint_data.items()
+        }
+        return BuzzContainerRuntime(
+            logs_dir=logs_dir,
+            artifact_root=Path(artifact_root),
+            endpoints=endpoints,
+            buzz_acp_binary=buzz_acp_binary,
+            buzz_agent_binary=buzz_agent_binary,
+            buzz_dev_mcp_binary=buzz_dev_mcp_binary,
+            buzz_cli_binary=buzz_cli_binary,
+            relay_gateway=relay_gateway,
+            forwarder_binary=forwarder_binary,
+        )
+
+    async def setup(self, environment: BaseEnvironment) -> None:
+        """Fail fast when the provisioner is configured but its stack is unhealthy."""
+        if self.provisioner is not None:
+            self.provisioner.healthcheck()
+
+    async def run(
+        self,
+        instruction: str,
+        environment: BaseEnvironment,
+        context: AgentContext,
+    ) -> None:
+        if self.provisioner is None or self.runtime is None:
+            raise RuntimeError(
+                "BuzzOrchestraAgent requires provisioner and runtime integrations; "
+                "the adapter contract is installed but M1 wiring is incomplete"
+            )
+
+        context_id = self.context_id or environment.context_id
+        if context_id is None:
+            raise RuntimeError("Harbor context_id is required as the trial join key")
+        trial_id = str(context_id)
+        run_id = self.run_id or trial_id
+        # Human-readable channel label: the task short name, so a spectator
+        # GUI shows one recognisable channel per problem per attempt.
+        channel_label = getattr(environment, "environment_name", None)
+        handle = self.provisioner.create_trial(
+            run_id, trial_id, self.manifest, channel_label=channel_label
+        )
+        if handle.trial_id != trial_id:
+            raise RuntimeError("provisioner returned a handle for a different trial_id")
+        if handle.manifest_hash != self.manifest.sha256:
+            raise RuntimeError("provisioner returned a handle for a different manifest")
+        try:
+            result = await self.runtime.run(
+                instruction=instruction,
+                environment=environment,
+                manifest=self.manifest,
+                trial=handle,
+            )
+        finally:
+            self.provisioner.teardown(handle)
+
+        context.n_input_tokens = result.input_tokens
+        context.n_cache_tokens = result.cached_input_tokens
+        context.n_output_tokens = result.output_tokens
+        context.cost_usd = result.cost_usd
+        context.metadata = {
+            **result.metadata,
+            "manifest_sha256": self.manifest.sha256,
+            "condition": self.manifest.condition,
+            "buzz_channel_id": handle.channel_id,
+            "run_id": run_id,
+            "trial_id": trial_id,
+        }

benchmarks/harbor-buzz-orchestra/src/harbor_buzz_orchestra/container_runtime.py

@@ -0,0 +1,651 @@
+"""Run the production Buzz agent stack inside the Harbor task container.
+
+Each provisioned identity is a full ``buzz-acp`` → ``buzz-agent`` →
+``buzz-dev-mcp`` process tree launched *inside* the task container — the same
+binaries and the same MCP toolset (shell, file tools, the ``buzz`` CLI on
+PATH) that the desktop app gives a Buzz agent. The harness stays outside:
+it provisions, uploads the pinned binaries, posts the task as the trial
+user, and observes the channel until the orchestrator publishes DONE.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import shlex
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+from harbor.environments.base import BaseEnvironment
+
+from .manifest import AgentClass, ExperimentManifest
+from .provisioning import AgentCredential, TrialHandle
+from .runtime import RuntimeResult
+
+
+DEFAULT_MAX_AGENT_ROUNDS = 32
+# Container-side layout for the uploaded Buzz stack.
+REMOTE_ROOT = "/opt/buzz"
+REMOTE_BIN = f"{REMOTE_ROOT}/bin"
+REMOTE_PROMPTS = f"{REMOTE_ROOT}/prompts"
+REMOTE_LOGS = f"{REMOTE_ROOT}/logs"
+# The relay is host-header tenant-bound (its community row is the authority
+# of its own RELAY_URL), so agents must present that exact Host. When the
+# relay actually lives outside the container, this forwarder listens on the
+# canonical loopback address and bridges the byte stream to the gateway.
+FORWARDER = f"{REMOTE_BIN}/relay-forwarder"
+FORWARDER_LOG = f"{REMOTE_LOGS}/relay-forwarder.log"
+# How many done-poll iterations between in-container liveness probes.
+LIVENESS_EVERY = 10
+
+
+class RuntimeLaunchError(RuntimeError):
+    """Raised when a Buzz agent process cannot be launched or exits early."""
+
+
+@dataclass(frozen=True, slots=True)
+class EndpointLaunchConfig:
+    """Deployment-specific environment needed to launch one manifest endpoint."""
+
+    provider: str
+    api_key_env: str
+    env: dict[str, str] = field(default_factory=dict)
+
+
+@dataclass(slots=True)
+class _Agent:
+    credential: AgentCredential
+    pid: int
+    stdout_log: str  # container path
+    stderr_log: str  # container path
+
+
+class BuzzContainerRuntime:
+    """Launch one production Buzz agent stack per identity in the container."""
+
+    def __init__(
+        self,
+        *,
+        logs_dir: Path,
+        artifact_root: Path,
+        endpoints: dict[str, EndpointLaunchConfig],
+        buzz_acp_binary: str = "buzz-acp",
+        buzz_agent_binary: str = "buzz-agent",
+        buzz_dev_mcp_binary: str = "buzz-dev-mcp",
+        buzz_cli_binary: str = "buzz",
+        relay_gateway: str = "",
+        forwarder_binary: str = "relay-forwarder",
+        max_agent_rounds: int = DEFAULT_MAX_AGENT_ROUNDS,
+        readiness_timeout_seconds: float = 60.0,
+        poll_seconds: float = 1.0,
+    ) -> None:
+        if max_agent_rounds <= 0:
+            raise ValueError("max_agent_rounds must be positive")
+        if readiness_timeout_seconds <= 0:
+            raise ValueError("readiness_timeout_seconds must be positive")
+        self.logs_dir = Path(logs_dir)
+        self.artifact_root = Path(artifact_root)
+        self.endpoints = endpoints
+        # Linux builds uploaded into the task container:
+        self.buzz_acp_binary = buzz_acp_binary
+        self.buzz_agent_binary = buzz_agent_binary
+        self.buzz_dev_mcp_binary = buzz_dev_mcp_binary
+        # Host build used for user/provisioning operations only:
+        self.buzz_cli_binary = buzz_cli_binary
+        # Where the relay actually lives, as seen from inside the task
+        # container (e.g. host.docker.internal:3600). When set, a loopback
+        # forwarder bridges the agents' canonical relay address — the Host
+        # the relay's community row is bound to — to this gateway.
+        self.relay_gateway = relay_gateway
+        self.forwarder_binary = forwarder_binary
+        self.max_agent_rounds = max_agent_rounds
+        self.readiness_timeout_seconds = readiness_timeout_seconds
+        self.poll_seconds = poll_seconds
+
+    async def run(
+        self,
+        *,
+        instruction: str,
+        environment: BaseEnvironment,
+        manifest: ExperimentManifest,
+        trial: TrialHandle,
+    ) -> RuntimeResult:
+        classes = self._classes_by_agent_id(manifest, trial.credentials)
+        orchestrator = next(c for c in trial.credentials if c.role == "orchestrator")
+        workers = [c for c in trial.credentials if c.agent_id != orchestrator.agent_id]
+        if not workers:
+            raise RuntimeLaunchError("Buzz orchestration requires at least one worker")
+        trial_dir = self.logs_dir / "buzz"
+        trial_dir.mkdir(parents=True, exist_ok=True)
+
+        agents: list[_Agent] = []
+        infra: list[_Agent] = []
+        try:
+            await self._install_stack(environment)
+            forwarder = await self._start_forwarder(environment, trial)
+            if forwarder is not None:
+                infra.append(forwarder)
+            await self._buzz_json(
+                trial.user, trial, "users", "set-profile", "--name",
+                trial.user.agent_id,
+            )
+            for credential in trial.credentials:
+                await self._buzz_json(
+                    credential, trial, "users", "set-profile", "--name",
+                    credential.agent_id,
+                )
+                agents.append(
+                    await self._launch_agent(
+                        environment=environment,
+                        trial=trial,
+                        credential=credential,
+                        agent_class=classes[credential.agent_id],
+                        trial_dir=trial_dir,
+                    )
+                )
+            await self._wait_for_agents_ready(
+                environment, agents, trial.channel_id, infra
+            )
+            # The task arrives exactly as it would in production Buzz: a
+            # user prompt @mentioning the orchestrator. The harness never
+            # speaks as any agent.
+            await self._send(
+                trial.user, trial, f"@{orchestrator.agent_id} {instruction}"
+            )
+            final_message = await asyncio.wait_for(
+                self._wait_for_done(environment, orchestrator, trial, agents + infra),
+                timeout=manifest.trial_budget.timeout_seconds,
+            )
+            await self._verify_m1_output(environment, manifest)
+        finally:
+            await self._stop_agents(environment, agents + infra)
+            await self._collect_logs(environment, trial_dir)
+
+        return RuntimeResult(
+            metadata={
+                "completion_message_id": final_message["id"],
+                "completion_message": final_message["content"],
+                "agent_runtime": "in-container",
+                "agent_hints_enabled": False,
+                "task_seed": "user-identity-prompt",
+                "agent_max_rounds": {
+                    credential.agent_id: (
+                        classes[credential.agent_id].budget.max_calls
+                        or self.max_agent_rounds
+                    )
+                    for credential in trial.credentials
+                },
+            }
+        )
+
+    # -- container setup ------------------------------------------------------
+
+    async def _install_stack(self, environment: BaseEnvironment) -> None:
+        """Upload the pinned Linux binaries into the task container."""
+        uploads = {
+            f"{REMOTE_BIN}/buzz-acp": self.buzz_acp_binary,
+            f"{REMOTE_BIN}/buzz-agent": self.buzz_agent_binary,
+            f"{REMOTE_BIN}/buzz-dev-mcp": self.buzz_dev_mcp_binary,
+        }
+        if self.relay_gateway:
+            uploads[FORWARDER] = self.forwarder_binary
+        for source in uploads.values():
+            if not Path(source).is_file():
+                raise RuntimeLaunchError(f"agent binary not found: {source}")
+        result = await environment.exec(
+            f"mkdir -p {REMOTE_BIN} {REMOTE_PROMPTS} {REMOTE_LOGS}"
+        )
+        if result.return_code != 0:
+            raise RuntimeLaunchError(
+                f"cannot create {REMOTE_ROOT} in the task container: "
+                f"{result.stderr or result.stdout}"
+            )
+        for target, source in uploads.items():
+            await environment.upload_file(source, target)
+        await environment.exec(f"chmod 0755 {REMOTE_BIN}/*")
+
+    async def _start_forwarder(
+        self, environment: BaseEnvironment, trial: TrialHandle
+    ) -> _Agent | None:
+        """Bridge the agents' canonical relay address to the real gateway.
+
+        The relay resolves its tenant from the request ``Host`` header, so the
+        agents must dial the exact authority its community row is bound to —
+        ``trial.relay_ws_url``. When that address is loopback inside the task
+        container but the relay lives on the Docker host, this starts the
+        uploaded forwarder listening on the canonical address and pumping the
+        byte stream to ``relay_gateway``. Returns ``None`` when no gateway is
+        configured (the relay is reachable directly).
+        """
+        if not self.relay_gateway:
+            return None
+        # Listen on the IPv4 loopback explicitly: binding the name `localhost`
+        # would pick whichever address family resolves first, while clients
+        # iterate both — pinning v4 makes the pair deterministic. The Host
+        # header the relay tenant-binds on comes from the URL the agents
+        # dial (trial.relay_ws_url), not from the socket address.
+        listen = self._ws_authority(trial.relay_ws_url).replace(
+            "localhost", "127.0.0.1", 1
+        )
+        log = FORWARDER_LOG
+        command = (
+            f"{shlex.quote(FORWARDER)} {shlex.quote(listen)} "
+            f"{shlex.quote(self.relay_gateway)} </dev/null "
+            f">{shlex.quote(log)} 2>&1 & echo $!"
+        )
+        result = await environment.exec(command)
+        try:
+            pid = int((result.stdout or "").strip().splitlines()[-1])
+        except (ValueError, IndexError) as error:
+            raise RuntimeLaunchError(
+                f"cannot launch relay forwarder: {result.stderr or result.stdout}"
+            ) from error
+        forwarder = _Agent(
+            AgentCredential(
+                agent_id="relay-forwarder", role="infra",
+                nostr_secret_key="", nostr_pubkey="", nostr_auth_tag="",
+                llm_endpoint="", llm_api_key="",
+            ),
+            pid, log, log,
+        )
+        deadline = asyncio.get_running_loop().time() + self.readiness_timeout_seconds
+        while True:
+            probe = await environment.exec(f"cat {shlex.quote(log)} 2>/dev/null")
+            if "forwarding" in (probe.stdout or ""):
+                return forwarder
+            await self._raise_for_dead_agents(environment, [forwarder])
+            if asyncio.get_running_loop().time() >= deadline:
+                raise RuntimeLaunchError(
+                    "relay forwarder did not report readiness; "
+                    f"see {log} in the trial artifacts"
+                )
+            await asyncio.sleep(self.poll_seconds)
+
+    @staticmethod
+    def _ws_authority(relay_ws_url: str) -> str:
+        """``host:port`` from a ws:// URL — the forwarder's listen address."""
+        if not relay_ws_url.startswith("ws://"):
+            raise RuntimeLaunchError(
+                "relay_gateway forwarding requires a ws:// relay_ws_url"
+            )
+        authority = relay_ws_url.removeprefix("ws://").split("/", 1)[0]
+        if ":" not in authority:
+            authority += ":80"
+        return authority
+
+    async def _launch_agent(
+        self,
+        *,
+        environment: BaseEnvironment,
+        trial: TrialHandle,
+        credential: AgentCredential,
+        agent_class: AgentClass,
+        trial_dir: Path,
+    ) -> _Agent:
+        if not credential.llm_endpoint:
+            raise RuntimeLaunchError("credential llm_endpoint must not be empty")
+        endpoint = self.endpoints.get(credential.llm_endpoint)
+        if endpoint is None:
+            raise RuntimeLaunchError(
+                f"no launch config for endpoint {credential.llm_endpoint!r}"
+            )
+        self._reject_identity_overrides(endpoint)
+        prompt_path = self.artifact_root / agent_class.prompt.path
+        self._verify_artifact(prompt_path, agent_class.prompt.sha256)
+        composed = self._compose_system_prompt(
+            trial_dir=trial_dir,
+            trial=trial,
+            credential=credential,
+            persona_path=prompt_path,
+        )
+        remote_prompt = f"{REMOTE_PROMPTS}/{credential.agent_id}.system-prompt.md"
+        await environment.upload_file(composed, remote_prompt)
+
+        stdout_log = f"{REMOTE_LOGS}/{credential.agent_id}.stdout.log"
+        stderr_log = f"{REMOTE_LOGS}/{credential.agent_id}.stderr.log"
+        env = self._agent_env(
+            trial=trial,
+            credential=credential,
+            agent_class=agent_class,
+            endpoint=endpoint,
+            remote_prompt=remote_prompt,
+        )
+        command = (
+            f"{shlex.quote(f'{REMOTE_BIN}/buzz-acp')} </dev/null "
+            f">{shlex.quote(stdout_log)} 2>{shlex.quote(stderr_log)} & echo $!"
+        )
+        result = await environment.exec(command, env=env)
+        try:
+            pid = int((result.stdout or "").strip().splitlines()[-1])
+        except (ValueError, IndexError) as error:
+            raise RuntimeLaunchError(
+                f"cannot launch agent {credential.agent_id}: "
+                f"{result.stderr or result.stdout}"
+            ) from error
+        return _Agent(credential, pid, stdout_log, stderr_log)
+
+    def _agent_env(
+        self,
+        *,
+        trial: TrialHandle,
+        credential: AgentCredential,
+        agent_class: AgentClass,
+        endpoint: EndpointLaunchConfig,
+        remote_prompt: str,
+    ) -> dict[str, str]:
+        """The desktop-launch environment: real acp/agent/dev-mcp wiring."""
+        return {
+            **endpoint.env,
+            "BUZZ_RELAY_URL": trial.relay_ws_url,
+            "BUZZ_PRIVATE_KEY": credential.nostr_secret_key,
+            # Desktop parity: the GUI also sets NOSTR_PRIVATE_KEY on buzz-acp
+            # so buzz-dev-mcp's shim can wire git auth/signing for the agent.
+            "NOSTR_PRIVATE_KEY": credential.nostr_secret_key,
+            "BUZZ_AUTH_TAG": credential.nostr_auth_tag,
+            "BUZZ_ACP_AGENT_COMMAND": f"{REMOTE_BIN}/buzz-agent",
+            "BUZZ_ACP_AGENT_ARGS": "",
+            "BUZZ_ACP_MCP_COMMAND": f"{REMOTE_BIN}/buzz-dev-mcp",
+            "BUZZ_ACP_CHANNELS": trial.channel_id,
+            "BUZZ_ACP_SUBSCRIBE": "mentions",
+            "BUZZ_ACP_RESPOND_TO": "anyone",
+            "BUZZ_ACP_NO_MEMORY": "true",
+            "BUZZ_ACP_SYSTEM_PROMPT_FILE": remote_prompt,
+            "BUZZ_AGENT_PROVIDER": endpoint.provider,
+            "BUZZ_AGENT_MODEL": credential.llm_endpoint,
+            "BUZZ_AGENT_MAX_OUTPUT_TOKENS": str(
+                agent_class.generation.max_output_tokens
+            ),
+            "BUZZ_AGENT_MAX_CONTEXT_TOKENS": str(
+                agent_class.generation.context_window_tokens
+            ),
+            "BUZZ_AGENT_MAX_ROUNDS": str(
+                agent_class.budget.max_calls or self.max_agent_rounds
+            ),
+            # The pinned persona is the whole prompt: no hint-file or skill
+            # discovery from the task filesystem (metadata reports this).
+            "BUZZ_AGENT_NO_HINTS": "1",
+            endpoint.api_key_env: credential.llm_api_key,
+        }
+
+    # -- lifecycle -------------------------------------------------------------
+
+    async def _wait_for_agents_ready(
+        self,
+        environment: BaseEnvironment,
+        agents: list[_Agent],
+        channel_id: str,
+        infra: list[_Agent] | None = None,
+    ) -> None:
+        """Wait until every ACP process confirms its trial-channel subscription."""
+        marker = f"subscribed to channel {channel_id}"
+        deadline = asyncio.get_running_loop().time() + self.readiness_timeout_seconds
+        pending = {agent.credential.agent_id: agent for agent in agents}
+        while pending:
+            await self._raise_for_dead_agents(environment, agents + (infra or []))
+            for agent_id, agent in list(pending.items()):
+                result = await environment.exec(
+                    f"cat {shlex.quote(agent.stdout_log)} "
+                    f"{shlex.quote(agent.stderr_log)} 2>/dev/null"
+                )
+                if marker in (result.stdout or ""):
+                    del pending[agent_id]
+            if not pending:
+                return
+            if asyncio.get_running_loop().time() >= deadline:
+                raise RuntimeLaunchError(
+                    "agents did not subscribe to trial channel before readiness "
+                    f"timeout: {sorted(pending)}"
+                )
+            await asyncio.sleep(self.poll_seconds)
+
+    async def _wait_for_done(
+        self,
+        environment: BaseEnvironment,
+        orchestrator: AgentCredential,
+        trial: TrialHandle,
+        agents: list[_Agent],
+    ) -> dict[str, Any]:
+        """Observe the channel as the trial user until the orchestrator posts DONE.
+
+        Observation only: the harness never speaks as any agent. If the team
+        stalls, the trial times out and the stall is the measured result.
+        """
+        polls = 0
+        while True:
+            if polls % LIVENESS_EVERY == 0:
+                await self._raise_for_dead_agents(environment, agents)
+            polls += 1
+            messages = await self._buzz_json(
+                trial.user, trial,
+                "messages", "get", "--channel", trial.channel_id,
+                "--limit", "100",
+            )
+            for message in messages:
+                if message.get("pubkey") == orchestrator.nostr_pubkey and str(
+                    message.get("content", "")
+                ).startswith("DONE:"):
+                    return message
+            await asyncio.sleep(self.poll_seconds)
+
+    async def _raise_for_dead_agents(
+        self, environment: BaseEnvironment, agents: list[_Agent]
+    ) -> None:
+        if not agents:
+            return
+        probes = "; ".join(
+            f"kill -0 {agent.pid} 2>/dev/null || echo DEAD:{agent.credential.agent_id}"
+            for agent in agents
+        )
+        result = await environment.exec(probes)
+        dead = [
+            line.removeprefix("DEAD:")
+            for line in (result.stdout or "").splitlines()
+            if line.startswith("DEAD:")
+        ]
+        if dead:
+            raise RuntimeLaunchError(
+                f"agent processes exited early: {sorted(dead)}; "
+                f"see {REMOTE_LOGS} in the trial artifacts"
+            )
+
+    @staticmethod
+    async def _stop_agents(
+        environment: BaseEnvironment, agents: list[_Agent]
+    ) -> None:
+        """Terminate every process of the uploaded stack (acp, agent, mcp)."""
+        if not agents:
+            return
+        # Match by cmdline prefix via /proc: pkill/procps is not guaranteed
+        # to exist in task images, the /proc filesystem is.
+        sweep = (
+            "for d in /proc/[0-9]*; do "
+            f"grep -aq {REMOTE_BIN} \"$d/cmdline\" 2>/dev/null "
+            "&& kill -TERM \"${d#/proc/}\" 2>/dev/null; done; true"
+        )
+        try:
+            await environment.exec(sweep)
+            await asyncio.sleep(2)
+            await environment.exec(sweep.replace("-TERM", "-KILL"))
+        except Exception:  # noqa: BLE001 — environment may already be gone
+            pass
+
+    async def _collect_logs(
+        self, environment: BaseEnvironment, trial_dir: Path
+    ) -> None:
+        try:
+            await environment.download_dir(REMOTE_LOGS, trial_dir)
+        except Exception:  # noqa: BLE001 — best effort; env may be torn down
+            pass
+
+    # -- Buzz CLI as the trial user / provisioning identities -------------------
+
+    @staticmethod
+    async def _verify_m1_output(
+        environment: BaseEnvironment, manifest: ExperimentManifest
+    ) -> None:
+        """Fail M1 immediately unless the artifact satisfies the grader contract."""
+        if manifest.condition != "M1-hello-world":
+            return
+        result = await environment.exec(
+            'python3 -c "from pathlib import Path; '
+            "p = Path('/app/hello.txt'); "
+            "assert p.is_file() and p.read_text().strip() == 'Hello, world!'\""
+        )
+        if result.return_code != 0:
+            detail = (
+                result.stderr or result.stdout or "grader-equivalent check failed"
+            ).strip()
+            raise RuntimeLaunchError(
+                "M1 pre-verifier sanity probe failed: /app/hello.txt must exist "
+                f"and its stripped text must equal 'Hello, world!' ({detail})"
+            )
+
+    async def _send(
+        self, credential: AgentCredential, trial: TrialHandle, content: str
+    ) -> None:
+        await self._buzz_json(
+            credential, trial,
+            "messages", "send", "--channel", trial.channel_id,
+            "--content", content,
+        )
+
+    async def _buzz_json(
+        self, credential: AgentCredential, trial: TrialHandle, *args: str
+    ) -> Any:
+        process = await asyncio.create_subprocess_exec(
+            self.buzz_cli_binary,
+            *args,
+            stdout=asyncio.subprocess.PIPE,
+            stderr=asyncio.subprocess.PIPE,
+            env={
+                **os.environ,
+                "BUZZ_RELAY_URL": self._user_relay_url(trial),
+                "BUZZ_PRIVATE_KEY": credential.nostr_secret_key,
+                "BUZZ_AUTH_TAG": credential.nostr_auth_tag,
+            },
+        )
+        stdout, stderr = await process.communicate()
+        if process.returncode != 0:
+            raise RuntimeLaunchError(
+                f"buzz {shlex.join(args)} exited {process.returncode}: "
+                f"{stderr.decode(errors='replace').strip()}"
+            )
+        try:
+            return json.loads(stdout)
+        except json.JSONDecodeError as error:
+            raise RuntimeLaunchError("buzz returned invalid JSON") from error
+
+    @staticmethod
+    def _user_relay_url(trial: TrialHandle) -> str:
+        """The relay as reachable from the HOST (user identity, harness).
+
+        ``trial.relay_ws_url`` is the container view (the agents' runtime);
+        ``trial.user_relay_url`` is the host view. Fall back to deriving an
+        http URL from the ws URL for handles minted before v1.2.
+        """
+        if trial.user_relay_url:
+            return trial.user_relay_url
+        return BuzzContainerRuntime._cli_relay_url(trial.relay_ws_url)
+
+    @staticmethod
+    def _cli_relay_url(relay_ws_url: str) -> str:
+        if relay_ws_url.startswith("ws://"):
+            return f"http://{relay_ws_url.removeprefix('ws://')}"
+        if relay_ws_url.startswith("wss://"):
+            return f"https://{relay_ws_url.removeprefix('wss://')}"
+        raise RuntimeLaunchError("trial relay_ws_url must use ws:// or wss://")
+
+    # -- manifest plumbing -------------------------------------------------------
+
+    @staticmethod
+    def _classes_by_agent_id(
+        manifest: ExperimentManifest, credentials: tuple[AgentCredential, ...]
+    ) -> dict[str, AgentClass]:
+        by_id = {entry.id: entry for entry in manifest.roster}
+        result: dict[str, AgentClass] = {}
+        for credential in credentials:
+            class_id, separator, index = credential.agent_id.rpartition("-")
+            match = by_id.get(class_id)
+            if not separator or not index.isdigit() or match is None:
+                raise RuntimeLaunchError(
+                    f"credential {credential.agent_id!r} does not match a roster class"
+                )
+            if credential.role != match.kind:
+                raise RuntimeLaunchError(
+                    f"credential {credential.agent_id!r} role does not match manifest"
+                )
+            result[credential.agent_id] = match
+        return result
+
+    @staticmethod
+    def _verify_artifact(path: Path, expected_sha256: str) -> None:
+        import hashlib
+
+        try:
+            actual = hashlib.sha256(path.read_bytes()).hexdigest()
+        except OSError as error:
+            raise RuntimeLaunchError(f"cannot read prompt {path}: {error}") from error
+        if actual != expected_sha256:
+            raise RuntimeLaunchError(
+                f"prompt hash mismatch for {path}: expected {expected_sha256}, got {actual}"
+            )
+
+    def _compose_system_prompt(
+        self,
+        *,
+        trial_dir: Path,
+        trial: TrialHandle,
+        credential: AgentCredential,
+        persona_path: Path,
+    ) -> Path:
+        """Append the trial's team roster to the pinned persona.
+
+        The analogue of a production Buzz workspace's team context: each agent
+        knows its own identity, its channel, the user it reports to, and its
+        teammates' names, pubkeys, and roles from its system prompt — it never
+        has to discover them over the relay.
+        """
+        persona = persona_path.read_text(encoding="utf-8")
+        lines = [
+            "",
+            "## Your team",
+            "",
+            f"You are `{credential.agent_id}` (pubkey `{credential.nostr_pubkey}`).",
+            f"The team coordinates in Buzz channel `{trial.channel_id}`.",
+            f"Tasks come from the user `{trial.user.agent_id}` "
+            f"(pubkey `{trial.user.nostr_pubkey}`); address your final report "
+            "to them.",
+            "",
+            "| Name | Role | Pubkey |",
+            "|------|------|--------|",
+        ]
+        for teammate in trial.credentials:
+            if teammate.agent_id == credential.agent_id:
+                continue
+            lines.append(
+                f"| {teammate.agent_id} | {teammate.role} "
+                f"| `{teammate.nostr_pubkey}` |"
+            )
+        composed = persona + "\n".join(lines) + "\n"
+        path = trial_dir / f"{credential.agent_id}.system-prompt.md"
+        path.write_text(composed, encoding="utf-8")
+        path.chmod(0o600)
+        return path
+
+    @staticmethod
+    def _reject_identity_overrides(endpoint: EndpointLaunchConfig) -> None:
+        forbidden = {
+            "BUZZ_RELAY_URL",
+            "BUZZ_PRIVATE_KEY",
+            "BUZZ_AUTH_TAG",
+            "BUZZ_ACP_CHANNELS",
+            "BUZZ_ACP_MCP_COMMAND",
+            "BUZZ_ACP_AGENT_COMMAND",
+        }
+        overlap = forbidden & endpoint.env.keys()
+        if overlap:
+            raise RuntimeLaunchError(
+                f"endpoint env cannot override trial identity: {sorted(overlap)}"
+            )

benchmarks/harbor-buzz-orchestra/src/harbor_buzz_orchestra/manifest.py

@@ -0,0 +1,145 @@
+"""Validated, content-addressed experiment manifests."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from pathlib import Path
+from typing import Any, Literal, Self
+
+import yaml
+from pydantic import BaseModel, ConfigDict, Field, model_validator
+
+
+class ManifestError(ValueError):
+    """Raised when a manifest cannot be loaded or validated."""
+
+
+class StrictModel(BaseModel):
+    """Base model that rejects misspelled or unrecognised manifest fields."""
+
+    model_config = ConfigDict(extra="forbid", frozen=True)
+
+
+class ArtifactRef(StrictModel):
+    """Reference to immutable prompt, persona, or skill content."""
+
+    path: str = Field(min_length=1)
+    sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
+
+
+class GenerationConfig(StrictModel):
+    """Model generation controls frozen for a condition."""
+
+    temperature: float = Field(default=0.0, ge=0.0)
+    max_output_tokens: int = Field(gt=0)
+    context_window_tokens: int = Field(gt=0)
+    extra: dict[str, Any] = Field(default_factory=dict)
+
+
+class AgentBudget(StrictModel):
+    """Optional per-agent live safety limits."""
+
+    max_calls: int | None = Field(default=None, gt=0)
+    max_input_tokens: int | None = Field(default=None, gt=0)
+    max_output_tokens: int | None = Field(default=None, gt=0)
+    max_cost_usd: float | None = Field(default=None, gt=0)
+
+
+class Price(StrictModel):
+    """Frozen USD rates for one endpoint revision, per million tokens."""
+
+    input_per_million_usd: float = Field(ge=0)
+    cached_input_per_million_usd: float = Field(ge=0)
+    output_per_million_usd: float = Field(ge=0)
+
+
+class AgentClass(StrictModel):
+    """A homogeneous class of agents in the trial roster."""
+
+    id: str = Field(min_length=1, pattern=r"^[a-z0-9][a-z0-9._-]*$")
+    kind: Literal["orchestrator", "worker"]
+    role: str = Field(min_length=1)
+    count: int = Field(gt=0)
+    endpoint: str = Field(min_length=1)
+    model_revision: str = Field(min_length=1)
+    prompt: ArtifactRef
+    persona: ArtifactRef | None = None
+    skills: tuple[ArtifactRef, ...] = ()
+    generation: GenerationConfig
+    budget: AgentBudget = AgentBudget()
+    concurrency: int = Field(default=1, gt=0)
+
+    @model_validator(mode="after")
+    def validate_concurrency(self) -> Self:
+        if self.concurrency > self.count:
+            raise ValueError("concurrency cannot exceed count")
+        return self
+
+
+class TrialBudget(StrictModel):
+    """Trial-wide hard limits enforced by the live runtime, not async receipts."""
+
+    timeout_seconds: int = Field(gt=0)
+    max_cost_usd: float | None = Field(default=None, gt=0)
+
+
+class ExperimentManifest(StrictModel):
+    """Complete immutable input defining one benchmark condition."""
+
+    schema_version: Literal["1"] = "1"
+    condition: str = Field(min_length=1)
+    roster: tuple[AgentClass, ...] = Field(min_length=1)
+    prices: dict[str, Price]
+    trial_budget: TrialBudget
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+    @model_validator(mode="after")
+    def validate_roster(self) -> Self:
+        ids = [entry.id for entry in self.roster]
+        if len(ids) != len(set(ids)):
+            raise ValueError("roster class ids must be unique")
+        orchestrators = sum(
+            entry.count for entry in self.roster if entry.kind == "orchestrator"
+        )
+        if orchestrators != 1:
+            raise ValueError("roster must contain exactly one orchestrator")
+        endpoints = {entry.endpoint for entry in self.roster}
+        missing_prices = sorted(endpoints - self.prices.keys())
+        if missing_prices:
+            raise ValueError(f"prices missing for endpoints: {missing_prices}")
+        return self
+
+    def canonical_bytes(self) -> bytes:
+        """Return stable UTF-8 JSON independent of YAML formatting and key order."""
+        data = self.model_dump(mode="json", exclude_none=False)
+        return json.dumps(
+            data,
+            sort_keys=True,
+            separators=(",", ":"),
+            ensure_ascii=False,
+            allow_nan=False,
+        ).encode("utf-8")
+
+    @property
+    def sha256(self) -> str:
+        """Return the condition identity derived from canonical manifest content."""
+        return hashlib.sha256(self.canonical_bytes()).hexdigest()
+
+    @classmethod
+    def load(cls, source: str | Path | dict[str, Any]) -> Self:
+        """Load a manifest from a YAML/JSON file or an already-decoded mapping."""
+        if isinstance(source, dict):
+            raw = source
+        else:
+            path = Path(source).expanduser()
+            try:
+                raw = yaml.safe_load(path.read_text(encoding="utf-8"))
+            except (OSError, yaml.YAMLError) as error:
+                raise ManifestError(f"cannot load manifest {path}: {error}") from error
+        if not isinstance(raw, dict):
+            raise ManifestError("manifest root must be a mapping")
+        try:
+            return cls.model_validate(raw)
+        except ValueError as error:
+            raise ManifestError(f"invalid manifest: {error}") from error

benchmarks/harbor-buzz-orchestra/src/harbor_buzz_orchestra/provisioning.py

@@ -0,0 +1,58 @@
+"""Typed boundary between the Harbor adapter and Buzz trial provisioning."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Protocol, runtime_checkable
+
+from .manifest import ExperimentManifest
+
+
+@dataclass(frozen=True, slots=True)
+class AgentCredential:
+    """One trial-scoped Buzz identity and attributed LLM credential."""
+
+    agent_id: str
+    role: str
+    nostr_secret_key: str
+    nostr_pubkey: str
+    nostr_auth_tag: str
+    llm_endpoint: str
+    llm_api_key: str
+
+
+@dataclass(frozen=True, slots=True)
+class TrialHandle:
+    """Provisioned Buzz resources owned by one Harbor trial."""
+
+    run_id: str
+    trial_id: str
+    manifest_hash: str
+    relay_ws_url: str
+    channel_id: str
+    credentials: tuple[AgentCredential, ...]
+    # The trial's human-analogue identity: owns the channel, posts the task
+    # as a user prompt, and receives the final report. Never runs an agent
+    # process, so its llm_endpoint and llm_api_key are empty strings.
+    user: AgentCredential
+    # v1.2 (additive): the relay as reachable from the HOST, where the user
+    # identity and the harness run. ``relay_ws_url`` is the view from the
+    # agents' runtime (the task container). Empty means both views coincide.
+    user_relay_url: str = ""
+
+
+@runtime_checkable
+class TrialProvisioner(Protocol):
+    """Creates and tears down trial-isolated Buzz resources synchronously."""
+
+    def create_trial(
+        self,
+        run_id: str,
+        trial_id: str,
+        manifest: ExperimentManifest,
+        channel_label: str | None = None,
+    ) -> TrialHandle: ...
+
+    def teardown(self, handle: TrialHandle) -> None: ...
+
+    def healthcheck(self) -> None: ...

benchmarks/harbor-buzz-orchestra/src/harbor_buzz_orchestra/runtime.py

@@ -0,0 +1,35 @@
+"""Runtime contract kept separate from Buzz resource provisioning."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Protocol
+
+from harbor.environments.base import BaseEnvironment
+
+from .manifest import ExperimentManifest
+from .provisioning import TrialHandle
+
+
+@dataclass(frozen=True, slots=True)
+class RuntimeResult:
+    """Aggregate values returned by an orchestration runtime."""
+
+    input_tokens: int = 0
+    cached_input_tokens: int = 0
+    output_tokens: int = 0
+    cost_usd: float = 0.0
+    metadata: dict[str, Any] = field(default_factory=dict)
+
+
+class OrchestraRuntime(Protocol):
+    """Runs the provisioned roster and writes trial artifacts into logs_dir."""
+
+    async def run(
+        self,
+        *,
+        instruction: str,
+        environment: BaseEnvironment,
+        manifest: ExperimentManifest,
+        trial: TrialHandle,
+    ) -> RuntimeResult: ...

benchmarks/harbor-buzz-orchestra/testbed/compose.benchmark.yml

@@ -0,0 +1,18 @@
+# Benchmark-only overrides for deploy/compose/compose.yml.
+#
+# The production bundle stays untouched; this file publishes what the
+# benchmark harness needs on the host:
+#   - relay Prometheus metrics (:9102) for scraping during runs
+#   - Postgres (:5433) for the provisioner and receipts ingestion
+#     (5433 to avoid colliding with other local Postgres instances)
+#
+# Usage:
+#   docker compose --env-file .env \
+#     -f compose.yml -f <this file> up -d --wait
+services:
+  relay:
+    ports:
+      - "${BUZZ_METRICS_HOST_PORT:-9102}:9102"
+  postgres:
+    ports:
+      - "${BUZZ_PG_HOST_PORT:-5433}:5432"

benchmarks/harbor-buzz-orchestra/testbed/endpoints/README.md

@@ -0,0 +1,26 @@
+# Endpoint launch configs
+
+Deployment-time mapping from manifest endpoint names to
+`EndpointLaunchConfig` (`provider` / `api_key_env` / `env`), passed to
+`harbor run` as `--agent-kwarg endpoint_config=<path>`. This is deployment
+config, deliberately OUTSIDE the immutable condition manifest — the manifest
+endpoint string remains the join key.
+
+Every key in these files must be a manifest endpoint name; the loader treats
+all entries as endpoint configs (no comment keys).
+
+## m1-local.json
+
+M1 wiring proof: both placeholder endpoints resolve to one local llama-server
+(OpenAI-compatible, `http://127.0.0.1:8091/v1`, no cloud keys).
+
+buzz-agent env contract (crates/buzz-agent/src/config.rs, pinned at the M1
+binary SHA): `provider=openai` reads `OPENAI_COMPAT_API_KEY` +
+`OPENAI_COMPAT_BASE_URL`; the runtime sets `BUZZ_AGENT_MODEL` from the
+manifest endpoint name, which overrides `OPENAI_COMPAT_MODEL` — llama-server
+ignores the model name, so the placeholder value is harmless there.
+llama-server needs no real key; the provisioner's per-endpoint
+`llm_api_keys` map supplies a dummy value.
+
+The Databricks pilot config is the same file shape with real serving
+endpoint hosts/keys.

benchmarks/harbor-buzz-orchestra/testbed/endpoints/anthropic-live.json

@@ -0,0 +1,12 @@
+{
+  "claude-sonnet-4-6": {
+    "provider": "anthropic",
+    "api_key_env": "ANTHROPIC_API_KEY",
+    "env": {}
+  },
+  "claude-haiku-4-5": {
+    "provider": "anthropic",
+    "api_key_env": "ANTHROPIC_API_KEY",
+    "env": {}
+  }
+}

benchmarks/harbor-buzz-orchestra/testbed/endpoints/m1-local.json

@@ -0,0 +1,16 @@
+{
+  "local/placeholder-orchestrator": {
+    "provider": "openai",
+    "api_key_env": "OPENAI_COMPAT_API_KEY",
+    "env": {
+      "OPENAI_COMPAT_BASE_URL": "http://127.0.0.1:8091/v1"
+    }
+  },
+  "local/placeholder-worker": {
+    "provider": "openai",
+    "api_key_env": "OPENAI_COMPAT_API_KEY",
+    "env": {
+      "OPENAI_COMPAT_BASE_URL": "http://127.0.0.1:8091/v1"
+    }
+  }
+}

benchmarks/harbor-buzz-orchestra/testbed/pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "harbor-buzz-testbed"
+version = "0.1.0"
+description = "Trial provisioning and local Buzz stack tooling for harbor-buzz-orchestra"
+requires-python = ">=3.12"
+dependencies = [
+    "harbor-buzz-orchestra",
+    "coincurve>=20",
+    "psycopg[binary]>=3.2",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.4",
+    "ruff>=0.15",
+]
+
+[tool.uv.sources]
+harbor-buzz-orchestra = { path = "..", editable = true }
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+target-version = "py312"
+line-length = 88

benchmarks/harbor-buzz-orchestra/testbed/sql/benchmark_schema.sql

@@ -0,0 +1,72 @@
+-- Benchmark schema for harbor-buzz-orchestra runs.
+--
+-- Lives in the shared Postgres instance but is OWNED BY THE HARNESS, never by
+-- Buzz migrations (canonical plan §two-domain rule). Idempotent: safe to apply
+-- on every testbed bring-up.
+--
+--   docker exec -i <postgres> psql -U buzz -d buzz < sql/benchmark_schema.sql
+
+CREATE SCHEMA IF NOT EXISTS benchmark;
+
+-- One row per provisioned trial; written by BuzzTrialProvisioner.
+CREATE TABLE IF NOT EXISTS benchmark.trial_manifest (
+    run_id        text        NOT NULL,
+    trial_id      uuid        NOT NULL,
+    manifest_hash text        NOT NULL,
+    channel_id    uuid        NOT NULL,
+    created_at    timestamptz NOT NULL DEFAULT now(),
+    handle        jsonb       NOT NULL DEFAULT '{}'::jsonb,
+    archived_at   timestamptz,
+    PRIMARY KEY (run_id, trial_id)
+);
+
+-- Immutable LLM receipts, ingested post-run from the accounting path
+-- (Databricks AI Gateway inference tables first; LiteLLM shim fallback).
+-- Authoritative for tokens/cost and orchestrator-vs-worker attribution;
+-- Harbor AgentContext totals are a reconciliation checksum only.
+CREATE TABLE IF NOT EXISTS benchmark.llm_receipts (
+    receipt_id           bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+    run_id               text        NOT NULL,
+    trial_id             uuid        NOT NULL,
+    agent_id             text        NOT NULL,
+    role                 text        NOT NULL,
+    condition            text        NOT NULL,
+    endpoint             text        NOT NULL,
+    model_revision       text        NOT NULL,
+    request_id           text        NOT NULL,  -- gateway request identity
+    requested_at         timestamptz NOT NULL,
+    latency_ms           integer,
+    input_tokens         bigint      NOT NULL DEFAULT 0,
+    cached_input_tokens  bigint      NOT NULL DEFAULT 0,
+    output_tokens        bigint      NOT NULL DEFAULT 0,
+    cost_usd             numeric(12, 6),
+    source               text        NOT NULL,  -- 'ai_gateway' | 'litellm' | 'client'
+    raw                  jsonb       NOT NULL DEFAULT '{}'::jsonb,
+    UNIQUE (source, request_id),
+    FOREIGN KEY (run_id, trial_id)
+        REFERENCES benchmark.trial_manifest (run_id, trial_id)
+);
+
+CREATE INDEX IF NOT EXISTS llm_receipts_trial_idx
+    ON benchmark.llm_receipts (run_id, trial_id, agent_id);
+
+-- Harness-recorded timing spans (monotonic clocks are authoritative for
+-- latency). kind examples: 'trial', 'llm_call', 'terminal_exec',
+-- 'terminal_queue_wait' — queue-wait is recorded separately from execution so
+-- speed can be reported both as-run and queue-adjusted under the M1
+-- serialized-broker policy.
+CREATE TABLE IF NOT EXISTS benchmark.spans (
+    span_id     bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+    run_id      text        NOT NULL,
+    trial_id    uuid        NOT NULL,
+    agent_id    text,                            -- NULL for trial-level spans
+    kind        text        NOT NULL,
+    started_at  timestamptz NOT NULL,
+    duration_ms bigint      NOT NULL CHECK (duration_ms >= 0),
+    detail      jsonb       NOT NULL DEFAULT '{}'::jsonb,
+    FOREIGN KEY (run_id, trial_id)
+        REFERENCES benchmark.trial_manifest (run_id, trial_id)
+);
+
+CREATE INDEX IF NOT EXISTS spans_trial_kind_idx
+    ON benchmark.spans (run_id, trial_id, kind);

benchmarks/harbor-buzz-orchestra/testbed/src/harbor_buzz_testbed/__init__.py

@@ -0,0 +1,15 @@
+"""Testbed-side provisioning for harbor-buzz-orchestra trials."""
+
+from .provisioner import (
+    BuzzTrialProvisioner,
+    ProvisioningError,
+    TestbedConfig,
+    provisioner_from_dict,
+)
+
+__all__ = [
+    "BuzzTrialProvisioner",
+    "ProvisioningError",
+    "TestbedConfig",
+    "provisioner_from_dict",
+]

benchmarks/harbor-buzz-orchestra/testbed/src/harbor_buzz_testbed/buzz_cli.py

@@ -0,0 +1,102 @@
+"""Thin subprocess wrapper over the ``buzz`` CLI — the production client path."""
+
+from __future__ import annotations
+
+import json
+import subprocess
+from typing import Any
+
+
+class BuzzCliError(RuntimeError):
+    """A buzz CLI invocation failed."""
+
+
+class BuzzCli:
+    """Run buzz CLI commands as one relay identity (key + NIP-OA auth tag)."""
+
+    def __init__(
+        self,
+        relay_url: str,
+        secret_key: str,
+        auth_tag: str,
+        *,
+        binary: str = "buzz",
+        timeout_seconds: float = 30.0,
+    ) -> None:
+        self._relay_url = relay_url
+        self._secret_key = secret_key
+        self._auth_tag = auth_tag
+        self._binary = binary
+        self._timeout = timeout_seconds
+
+    def run(self, *args: str) -> Any:
+        """Run a buzz subcommand and return its parsed JSON stdout."""
+        command = [self._binary, *args]
+        try:
+            completed = subprocess.run(
+                command,
+                capture_output=True,
+                text=True,
+                timeout=self._timeout,
+                env={
+                    "BUZZ_RELAY_URL": self._relay_url,
+                    "BUZZ_PRIVATE_KEY": self._secret_key,
+                    "BUZZ_AUTH_TAG": self._auth_tag,
+                    "PATH": _path(),
+                },
+            )
+        except (OSError, subprocess.TimeoutExpired) as error:
+            raise BuzzCliError(f"buzz {args[0]}: {error}") from error
+        if completed.returncode != 0:
+            raise BuzzCliError(
+                f"buzz {' '.join(args)} exited {completed.returncode}: "
+                f"{completed.stderr.strip() or completed.stdout.strip()}"
+            )
+        if not completed.stdout.strip():
+            return None
+        try:
+            return json.loads(completed.stdout)
+        except json.JSONDecodeError as error:
+            raise BuzzCliError(
+                f"buzz {args[0]} returned non-JSON output: {completed.stdout[:200]!r}"
+            ) from error
+
+    def create_private_channel(self, name: str, description: str) -> str:
+        """Create a private stream channel; return its UUID."""
+        response = self.run(
+            "channels",
+            "create",
+            "--name",
+            name,
+            "--type",
+            "stream",
+            "--visibility",
+            "private",
+            "--description",
+            description,
+        )
+        channel_id = response.get("channel_id") if isinstance(response, dict) else None
+        if not channel_id:
+            raise BuzzCliError(f"channel create returned no channel_id: {response}")
+        return channel_id
+
+    def add_member(self, channel_id: str, pubkey: str) -> None:
+        self.run(
+            "channels",
+            "add-member",
+            "--channel",
+            channel_id,
+            "--pubkey",
+            pubkey,
+            "--role",
+            "member",
+        )
+
+    def archive_channel(self, channel_id: str) -> None:
+        self.run("channels", "archive", "--channel", channel_id)
+
+
+def _path() -> str:
+    import os
+
+    return os.environ.get("PATH", "/usr/local/bin:/usr/bin:/bin")

benchmarks/harbor-buzz-orchestra/testbed/src/harbor_buzz_testbed/keys.py

@@ -0,0 +1,90 @@
+"""Nostr keygen and NIP-OA owner attestation for trial agents."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from dataclasses import dataclass
+
+import coincurve
+
+
+@dataclass(frozen=True, slots=True)
+class NostrKeypair:
+    secret_key: str  # hex
+    pubkey: str  # x-only hex
+
+
+def generate_keypair() -> NostrKeypair:
+    """Generate a fresh secp256k1 keypair in Nostr hex form."""
+    key = coincurve.PrivateKey()
+    return NostrKeypair(
+        secret_key=key.to_hex(),
+        pubkey=key.public_key_xonly.format().hex(),
+    )
+
+
+def keypair_from_secret(secret_key: str) -> NostrKeypair:
+    """Rebuild the keypair for an existing hex secret key."""
+    key = coincurve.PrivateKey(bytes.fromhex(secret_key))
+    return NostrKeypair(
+        secret_key=secret_key,
+        pubkey=key.public_key_xonly.format().hex(),
+    )
+
+
+_BECH32_CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l"
+
+
+def _bech32_polymod(values: list[int]) -> int:
+    generator = (0x3B6A57B2, 0x26508E6D, 0x1EA119FA, 0x3D4233DD, 0x2A1462B3)
+    checksum = 1
+    for value in values:
+        top = checksum >> 25
+        checksum = (checksum & 0x1FFFFFF) << 5 ^ value
+        for i in range(5):
+            checksum ^= generator[i] if (top >> i) & 1 else 0
+    return checksum
+
+
+def encode_nsec(secret_key: str) -> str:
+    """Encode a hex secret key as a NIP-19 bech32 ``nsec1…`` string —
+    the form the desktop GUI's key-import onboarding accepts."""
+    hrp = "nsec"
+    data: list[int] = []
+    accumulator = bits = 0
+    for byte in bytes.fromhex(secret_key):
+        accumulator = accumulator << 8 | byte
+        bits += 8
+        while bits >= 5:
+            bits -= 5
+            data.append(accumulator >> bits & 31)
+    if bits:
+        data.append(accumulator << (5 - bits) & 31)
+    expanded = [ord(c) >> 5 for c in hrp] + [0] + [ord(c) & 31 for c in hrp]
+    polymod = _bech32_polymod(expanded + data + [0] * 6) ^ 1
+    checksum = [polymod >> 5 * (5 - i) & 31 for i in range(6)]
+    return hrp + "1" + "".join(_BECH32_CHARSET[d] for d in data + checksum)
+
+
+def compute_auth_tag(
+    owner_secret_key: str, agent_pubkey: str, conditions: str = ""
+) -> str:
+    """Compute the NIP-OA ``["auth", ...]`` tag authorising an agent key.
+
+    Mirrors crates/buzz-sdk/src/nip_oa.rs:
+    sig = schnorr(SHA256("nostr:agent-auth:" || agent_pubkey || ":" || conditions),
+    owner_secret_key). Returns the tag as a JSON string.
+    """
+    owner = coincurve.PrivateKey(bytes.fromhex(owner_secret_key))
+    preimage = f"nostr:agent-auth:{agent_pubkey}:{conditions}".encode()
+    signature = owner.sign_schnorr(hashlib.sha256(preimage).digest())
+    return json.dumps(
+        [
+            "auth",
+            owner.public_key_xonly.format().hex(),
+            conditions,
+            signature.hex(),
+        ],
+        separators=(",", ":"),
+    )

benchmarks/harbor-buzz-orchestra/testbed/src/harbor_buzz_testbed/provisioner.py

@@ -0,0 +1,274 @@
+"""Channel-per-trial Buzz provisioning against a local benchmark stack."""
+
+from __future__ import annotations
+
+import dataclasses
+import hashlib
+import json
+import urllib.error
+import urllib.request
+from dataclasses import dataclass
+
+import psycopg
+from harbor_buzz_orchestra.manifest import ExperimentManifest
+from harbor_buzz_orchestra.provisioning import AgentCredential, TrialHandle
+
+from .buzz_cli import BuzzCli
+from .keys import compute_auth_tag, generate_keypair, keypair_from_secret
+
+
+class ProvisioningError(RuntimeError):
+    """Trial provisioning failed or was invoked inconsistently."""
+
+
+@dataclass(frozen=True, slots=True)
+class TestbedConfig:
+    """Connection settings for one local benchmark stack."""
+
+    __test__ = False  # not a pytest class, despite the name
+    relay_http_url: str  # e.g. http://localhost:3000 — CLI + healthcheck
+    relay_ws_url: str  # as reachable FROM the agents' runtime
+    owner_secret_key: str  # relay owner key; signs NIP-OA attestations
+    postgres_dsn: str  # benchmark schema lives here
+    llm_api_keys: dict[str, str] = dataclasses.field(default_factory=dict)
+    # endpoint -> API key. v1: per-endpoint resolution; true per-agent
+    # Databricks attribution is pending the AI Gateway field verification.
+    # Pinned user identity (hex secret key). When set, every trial's user —
+    # the human analogue that owns the channel and posts the task — is this
+    # one identity instead of a fresh key per trial, so a GUI logged in as
+    # that user sees every trial channel accumulate. None preserves the
+    # fresh-user-per-trial behaviour.
+    user_secret_key: str | None = None
+    # When False, teardown leaves the trial channel unarchived (and its
+    # archived_at stamp NULL) so finished trials stay visible in a GUI.
+    archive_on_teardown: bool = True
+
+
+def provisioner_from_dict(config: dict[str, object]) -> "BuzzTrialProvisioner":
+    """Harbor CLI factory for a JSON-decoded testbed configuration."""
+    return BuzzTrialProvisioner(TestbedConfig(**config))
+
+
+class BuzzTrialProvisioner:
+    """Implements the TrialHandle v1.1 contract against a live Buzz relay.
+
+    Guarantees (contract PLANS/HARBOR_BUZZ_TRIALHANDLE_CONTRACT.md):
+    - create_trial is synchronous and idempotent per (run_id, trial_id);
+      concurrency-safe via a Postgres advisory lock on the trial key.
+    - One private channel per trial; membership is exactly the trial's
+      credentials, so cross-trial reads are blocked by construction.
+    - Fresh keys per trial; never reused.
+    - teardown archives the channel and stamps archived_at; events are
+      never deleted.
+    """
+
+    def __init__(self, config: TestbedConfig) -> None:
+        self._config = config
+
+    # -- contract surface ---------------------------------------------------
+
+    def create_trial(
+        self,
+        run_id: str,
+        trial_id: str,
+        manifest: ExperimentManifest,
+        channel_label: str | None = None,
+    ) -> TrialHandle:
+        manifest_hash = manifest.sha256
+        with psycopg.connect(self._config.postgres_dsn) as conn:
+            self._lock_trial(conn, run_id, trial_id)
+            existing = self._load_trial(conn, run_id, trial_id)
+            if existing is not None:
+                if existing.manifest_hash != manifest_hash:
+                    raise ProvisioningError(
+                        f"trial ({run_id}, {trial_id}) already provisioned with "
+                        f"manifest {existing.manifest_hash}, got {manifest_hash}"
+                    )
+                return existing
+
+            handle = self._provision(
+                run_id, trial_id, manifest, manifest_hash, channel_label
+            )
+            self._store_trial(conn, handle)
+            conn.commit()
+            return handle
+
+    def teardown(self, handle: TrialHandle) -> None:
+        if not self._config.archive_on_teardown:
+            # GUI/spectator mode: finished trial channels stay visible.
+            return
+        cli = self._cli_for(handle.credentials[0])
+        try:
+            cli.archive_channel(handle.channel_id)
+        except Exception as error:  # noqa: BLE001 — idempotent re-teardown
+            if "archived" not in str(error).lower():
+                raise
+        with psycopg.connect(self._config.postgres_dsn) as conn:
+            conn.execute(
+                "UPDATE benchmark.trial_manifest"
+                " SET archived_at = COALESCE(archived_at, now())"
+                " WHERE run_id = %s AND trial_id = %s",
+                (handle.run_id, handle.trial_id),
+            )
+            conn.commit()
+
+    def healthcheck(self) -> None:
+        url = f"{self._config.relay_http_url.rstrip('/')}/_readiness"
+        try:
+            with urllib.request.urlopen(url, timeout=5) as response:
+                if response.status != 200:
+                    raise ProvisioningError(
+                        f"relay readiness returned {response.status}"
+                    )
+        except (urllib.error.URLError, OSError) as error:
+            raise ProvisioningError(f"relay unreachable at {url}: {error}") from error
+        try:
+            with psycopg.connect(self._config.postgres_dsn, connect_timeout=5) as conn:
+                conn.execute("SELECT 1")
+        except psycopg.Error as error:
+            raise ProvisioningError(
+                f"benchmark postgres unreachable: {error}"
+            ) from error
+
+    # -- internals ----------------------------------------------------------
+
+    def _provision(
+        self,
+        run_id: str,
+        trial_id: str,
+        manifest: ExperimentManifest,
+        manifest_hash: str,
+        channel_label: str | None,
+    ) -> TrialHandle:
+        credentials = self._mint_credentials(manifest)
+        user = self._mint_user()
+        # The user identity creates the channel and invites the agents —
+        # mirroring production Buzz, where a human owns the channel their
+        # agents work in.
+        cli = self._cli_for(user)
+        name = (
+            f"{channel_label}-{trial_id[:8]}"
+            if channel_label
+            else f"trial-{trial_id[:8]}-{manifest_hash[:8]}"
+        )
+        channel_id = cli.create_private_channel(
+            name=name,
+            description=f"run={run_id} trial={trial_id} manifest={manifest_hash}",
+        )
+        for credential in credentials:
+            cli.add_member(channel_id, credential.nostr_pubkey)
+        return TrialHandle(
+            run_id=run_id,
+            trial_id=trial_id,
+            manifest_hash=manifest_hash,
+            relay_ws_url=self._config.relay_ws_url,
+            channel_id=channel_id,
+            credentials=credentials,
+            user=user,
+            user_relay_url=self._config.relay_http_url,
+        )
+
+    def _mint_credentials(
+        self, manifest: ExperimentManifest
+    ) -> tuple[AgentCredential, ...]:
+        roster = sorted(manifest.roster, key=lambda e: e.kind != "orchestrator")
+        credentials: list[AgentCredential] = []
+        for entry in roster:
+            api_key = self._config.llm_api_keys.get(entry.endpoint)
+            if api_key is None:
+                raise ProvisioningError(
+                    f"no LLM API key configured for endpoint {entry.endpoint!r}"
+                )
+            for index in range(1, entry.count + 1):
+                keypair = generate_keypair()
+                credentials.append(
+                    AgentCredential(
+                        agent_id=f"{entry.id}-{index}",
+                        role=entry.kind,
+                        nostr_secret_key=keypair.secret_key,
+                        nostr_pubkey=keypair.pubkey,
+                        nostr_auth_tag=compute_auth_tag(
+                            self._config.owner_secret_key, keypair.pubkey
+                        ),
+                        llm_endpoint=entry.endpoint,
+                        llm_api_key=api_key,
+                    )
+                )
+        return tuple(credentials)
+
+    def _mint_user(self) -> AgentCredential:
+        """Mint the trial's user identity — the human analogue, not an agent.
+
+        With a pinned ``user_secret_key`` the same identity fronts every
+        trial, like one human running many teams; otherwise each trial gets
+        a fresh user key.
+        """
+        keypair = (
+            keypair_from_secret(self._config.user_secret_key)
+            if self._config.user_secret_key
+            else generate_keypair()
+        )
+        return AgentCredential(
+            agent_id="user",
+            role="user",
+            nostr_secret_key=keypair.secret_key,
+            nostr_pubkey=keypair.pubkey,
+            nostr_auth_tag=compute_auth_tag(
+                self._config.owner_secret_key, keypair.pubkey
+            ),
+            llm_endpoint="",
+            llm_api_key="",
+        )
+
+    def _cli_for(self, credential: AgentCredential) -> BuzzCli:
+        return BuzzCli(
+            relay_url=self._config.relay_http_url,
+            secret_key=credential.nostr_secret_key,
+            auth_tag=credential.nostr_auth_tag,
+        )
+
+    @staticmethod
+    def _lock_trial(conn: psycopg.Connection, run_id: str, trial_id: str) -> None:
+        digest = hashlib.sha256(f"{run_id}\x00{trial_id}".encode()).digest()
+        lock_key = int.from_bytes(digest[:8], "big", signed=True)
+        conn.execute("SELECT pg_advisory_xact_lock(%s)", (lock_key,))
+
+    @staticmethod
+    def _load_trial(
+        conn: psycopg.Connection, run_id: str, trial_id: str
+    ) -> TrialHandle | None:
+        row = conn.execute(
+            "SELECT handle FROM benchmark.trial_manifest"
+            " WHERE run_id = %s AND trial_id = %s",
+            (run_id, trial_id),
+        ).fetchone()
+        if row is None:
+            return None
+        stored = row[0]
+        return TrialHandle(
+            run_id=stored["run_id"],
+            trial_id=stored["trial_id"],
+            manifest_hash=stored["manifest_hash"],
+            relay_ws_url=stored["relay_ws_url"],
+            channel_id=stored["channel_id"],
+            credentials=tuple(
+                AgentCredential(**credential) for credential in stored["credentials"]
+            ),
+            user=AgentCredential(**stored["user"]),
+            user_relay_url=stored.get("user_relay_url", ""),
+        )
+
+    @staticmethod
+    def _store_trial(conn: psycopg.Connection, handle: TrialHandle) -> None:
+        conn.execute(
+            "INSERT INTO benchmark.trial_manifest"
+            " (run_id, trial_id, manifest_hash, channel_id, handle)"
+            " VALUES (%s, %s, %s, %s, %s)",
+            (
+                handle.run_id,
+                handle.trial_id,
+                handle.manifest_hash,
+                handle.channel_id,
+                json.dumps(dataclasses.asdict(handle)),
+            ),
+        )

benchmarks/harbor-buzz-orchestra/testbed/tests/conftest.py

@@ -0,0 +1,59 @@
+"""Shared fixtures for testbed provisioning tests."""
+
+from __future__ import annotations
+
+import pytest
+from harbor_buzz_orchestra.manifest import ExperimentManifest
+
+PROMPT_REF = {"path": "personas/x.md", "sha256": "0" * 64}
+
+
+@pytest.fixture()
+def manifest() -> ExperimentManifest:
+    """A minimal two-class manifest: one orchestrator, two workers."""
+    return ExperimentManifest.load(
+        {
+            "condition": "O-test",
+            "roster": [
+                {
+                    "id": "worker-glm",
+                    "kind": "worker",
+                    "role": "implementer",
+                    "count": 2,
+                    "endpoint": "databricks/glm",
+                    "model_revision": "glm-5.2",
+                    "prompt": PROMPT_REF,
+                    "generation": {
+                        "max_output_tokens": 4096,
+                        "context_window_tokens": 128000,
+                    },
+                },
+                {
+                    "id": "orch-opus",
+                    "kind": "orchestrator",
+                    "role": "lead",
+                    "count": 1,
+                    "endpoint": "databricks/opus",
+                    "model_revision": "opus-4.8",
+                    "prompt": PROMPT_REF,
+                    "generation": {
+                        "max_output_tokens": 8192,
+                        "context_window_tokens": 200000,
+                    },
+                },
+            ],
+            "prices": {
+                "databricks/glm": {
+                    "input_per_million_usd": 0.5,
+                    "cached_input_per_million_usd": 0.1,
+                    "output_per_million_usd": 1.5,
+                },
+                "databricks/opus": {
+                    "input_per_million_usd": 15.0,
+                    "cached_input_per_million_usd": 1.5,
+                    "output_per_million_usd": 75.0,
+                },
+            },
+            "trial_budget": {"timeout_seconds": 3600},
+        }
+    )

benchmarks/harbor-buzz-orchestra/testbed/tests/test_benchmark.py

@@ -0,0 +1,156 @@
+"""just benchmark must default to leaderboard-eligible settings."""
+
+import importlib.util
+import json
+import sys
+from pathlib import Path
+
+import pytest
+
+_SCRIPT = Path(__file__).parents[2] / "scripts" / "benchmark.py"
+_spec = importlib.util.spec_from_file_location("benchmark", _SCRIPT)
+benchmark = importlib.util.module_from_spec(_spec)
+sys.modules["benchmark"] = benchmark
+_spec.loader.exec_module(benchmark)
+
+
+@pytest.fixture
+def state_dir(tmp_path, monkeypatch):
+    monkeypatch.setattr(benchmark, "STATE_DIR", tmp_path / ".benchmark")
+    return tmp_path / ".benchmark"
+
+
+def test_defaults_are_leaderboard_eligible():
+    args = benchmark.parse_args([])
+    assert args.attempts == 5
+    assert args.dataset is None and args.path is None  # dataset default applied later
+    argv = benchmark.leaderboard_argv(args, Path("prov.json"), Path("linux-bin"))
+    assert argv[argv.index("--dataset") + 1] == "terminal-bench/terminal-bench-2-1"
+    assert argv[argv.index("--attempts") + 1] == "5"
+    assert argv[argv.index("--manifest") + 1].endswith("tb-cobol-sonnet-haiku.yaml")
+    assert argv[argv.index("--agent-bin-dir") + 1] == "linux-bin"
+    # In-container agents reach the host relay through the forwarder gateway.
+    assert argv[argv.index("--relay-gateway") + 1] == (
+        f"host.docker.internal:{benchmark.RELAY_HTTP_PORT}"
+    )
+
+
+def test_selectors_pass_through():
+    args = benchmark.parse_args(
+        ["--path", "/tmp/task", "-i", "cobol*", "-x", "flaky*", "-k", "1",
+         "--job-name", "smoke", "--dry-run"]
+    )
+    argv = benchmark.leaderboard_argv(args, Path("p.json"), Path("b"))
+    assert argv[argv.index("--path") + 1] == "/tmp/task"
+    assert argv[argv.index("--include-task") + 1] == "cobol*"
+    assert argv[argv.index("--exclude-task") + 1] == "flaky*"
+    assert argv[argv.index("--attempts") + 1] == "1"
+    assert "--dry-run" in argv
+    assert "--dataset" not in argv
+
+
+def test_state_is_generated_once_and_reused(state_dir):
+    first = benchmark.load_state()
+    second = benchmark.load_state()
+    assert first["user_secret_key"] == second["user_secret_key"]
+    assert first["owner_secret_key"] != first["user_secret_key"]
+    assert len(first["user_pubkey"]) == 64
+    stored = json.loads((state_dir / "state.json").read_text())
+    assert "user_pubkey" not in stored  # derived, never persisted
+
+
+def test_provisioner_config_pins_user_and_keeps_channels(state_dir, tmp_path, monkeypatch):
+    monkeypatch.setenv("FAKE_KEY_ENV", "sk-test")
+    endpoints = tmp_path / "endpoints.json"
+    endpoints.write_text(
+        json.dumps({"model-a": {"provider": "anthropic", "api_key_env": "FAKE_KEY_ENV"}})
+    )
+    state = benchmark.load_state()
+    path = benchmark.write_provisioner_config(state, endpoints)
+    config = json.loads(path.read_text())
+    assert config["user_secret_key"] == state["user_secret_key"]
+    assert config["archive_on_teardown"] is False
+    assert config["llm_api_keys"] == {"model-a": "sk-test"}
+    assert str(benchmark.RELAY_HTTP_PORT) in config["relay_http_url"]
+    # Both views dial the relay's canonical host-bound address; inside the
+    # task container the loopback forwarder bridges it to the host gateway.
+    assert config["relay_ws_url"] == f"ws://localhost:{benchmark.RELAY_HTTP_PORT}"
+    assert config["relay_http_url"].startswith("http://localhost:")
+
+
+def test_provisioner_config_missing_api_key_is_explicit(state_dir, tmp_path, monkeypatch):
+    monkeypatch.delenv("MISSING_KEY_ENV", raising=False)
+    endpoints = tmp_path / "endpoints.json"
+    endpoints.write_text(
+        json.dumps({"model-a": {"provider": "x", "api_key_env": "MISSING_KEY_ENV"}})
+    )
+    with pytest.raises(SystemExit, match="MISSING_KEY_ENV"):
+        benchmark.write_provisioner_config(benchmark.load_state(), endpoints)
+
+
+def test_env_file_wires_owner_and_ports(state_dir):
+    state = benchmark.load_state()
+    env_path = benchmark.write_env_file(state)
+    env = dict(
+        line.split("=", 1) for line in env_path.read_text().splitlines() if line
+    )
+    assert env["RELAY_OWNER_PUBKEY"] == state["owner_pubkey"]
+    assert env["BUZZ_HTTP_PORT"] == str(benchmark.RELAY_HTTP_PORT)
+    assert env["BUZZ_PG_HOST_PORT"] == str(benchmark.PG_HOST_PORT)
+    assert env["BUZZ_REQUIRE_RELAY_MEMBERSHIP"] == "true"
+
+
+def test_compose_command_isolates_the_project(state_dir):
+    command = benchmark.compose_command("up", "-d")
+    assert command[:2] == ["docker", "compose"]
+    assert command[command.index("--project-name") + 1] == "buzz-benchmark"
+    files = [command[i + 1] for i, part in enumerate(command) if part == "-f"]
+    assert any(f.endswith("deploy/compose/compose.yml") for f in files)
+    assert any(f.endswith("compose.benchmark.yml") for f in files)
+
+
+def test_bring_up_self_heals_a_stale_credential_volume(monkeypatch):
+    calls = []
+
+    def fake_run(command, check=True):
+        calls.append(command)
+        if len(calls) == 1:  # first up fails against the stale volume
+            raise benchmark.subprocess.CalledProcessError(1, command)
+
+    monkeypatch.setattr(benchmark.subprocess, "run", fake_run)
+    monkeypatch.setattr(benchmark, "stale_credential_volume", lambda state: True)
+    benchmark.bring_up_stack({})
+    assert [c[-3:] for c in calls] == [
+        ["up", "-d", "--wait"],
+        [str(benchmark.COMPOSE_FILES[-1]), "down", "-v"],
+        ["up", "-d", "--wait"],
+    ]
+
+
+def test_bring_up_reraises_unrelated_failures(monkeypatch):
+    def fake_run(command, check=True):
+        raise benchmark.subprocess.CalledProcessError(1, command)
+
+    monkeypatch.setattr(benchmark.subprocess, "run", fake_run)
+    monkeypatch.setattr(benchmark, "stale_credential_volume", lambda state: False)
+    with pytest.raises(benchmark.subprocess.CalledProcessError):
+        benchmark.bring_up_stack({})
+
+
+def test_fresh_resets_volumes_and_gui_state(tmp_path, monkeypatch):
+    commands = []
+    monkeypatch.setattr(
+        benchmark.subprocess, "run", lambda cmd, check=True: commands.append(cmd)
+    )
+    monkeypatch.setattr(benchmark.sys, "platform", "darwin")
+    monkeypatch.setattr(benchmark.Path, "home", classmethod(lambda cls: tmp_path))
+    gui_state = tmp_path / "Library" / "WebKit" / benchmark.GUI_BUNDLE_IDENTIFIER
+    gui_state.mkdir(parents=True)
+    (gui_state / "localstorage.sqlite3").touch()
+
+    benchmark.reset_environment()
+
+    assert ["down", "-v"] == commands[0][-2:]
+    assert not gui_state.exists()
+    assert benchmark.parse_args(["--fresh"]).fresh
+    assert not benchmark.parse_args([]).fresh

benchmarks/harbor-buzz-orchestra/testbed/tests/test_keys.py

@@ -0,0 +1,72 @@
+"""Keygen and NIP-OA attestation unit tests."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+
+import coincurve
+from harbor_buzz_testbed.keys import (
+    compute_auth_tag,
+    encode_nsec,
+    generate_keypair,
+)
+
+# Produced by the Rust reference implementation
+# (crates/buzz-sdk/examples/compute_auth_tag.rs) for owner secret 0x...03 and
+# agent pubkey "a" * 64. Pins the preimage format across implementations.
+RUST_OWNER_SECRET = "0" * 63 + "3"
+RUST_AGENT_PUBKEY = "a" * 64
+RUST_TAG = [
+    "auth",
+    "f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9",
+    "",
+    "20105c618d6e5d8f559cffb6f0d7a7b4f44f3a567e1be94c96378d45ac3625da"
+    "34c2e7357ea1d3ce980978334546b3e740c155e81b833ebe140d519d39ed8867",
+]
+
+
+def preimage_digest(agent_pubkey: str, conditions: str) -> bytes:
+    return hashlib.sha256(
+        f"nostr:agent-auth:{agent_pubkey}:{conditions}".encode()
+    ).digest()
+
+
+def test_generate_keypair_is_fresh_and_hex():
+    first, second = generate_keypair(), generate_keypair()
+    assert first.secret_key != second.secret_key
+    assert first.pubkey != second.pubkey
+    assert len(first.secret_key) == 64
+    assert len(first.pubkey) == 64
+    int(first.secret_key, 16)
+    int(first.pubkey, 16)
+
+
+def test_auth_tag_shape_and_owner_pubkey():
+    tag = json.loads(compute_auth_tag(RUST_OWNER_SECRET, RUST_AGENT_PUBKEY))
+    assert tag[0] == "auth"
+    assert tag[1] == RUST_TAG[1]  # same owner pubkey as the Rust implementation
+    assert tag[2] == ""
+
+
+def test_auth_tag_signature_verifies_over_nip_oa_preimage():
+    agent = generate_keypair()
+    tag = json.loads(compute_auth_tag(RUST_OWNER_SECRET, agent.pubkey))
+    owner_pubkey = coincurve.PublicKeyXOnly(bytes.fromhex(tag[1]))
+    assert owner_pubkey.verify(bytes.fromhex(tag[3]), preimage_digest(agent.pubkey, ""))
+
+
+def test_rust_reference_tag_verifies_under_python_preimage():
+    """The Rust-signed vector must verify against our preimage construction."""
+    owner_pubkey = coincurve.PublicKeyXOnly(bytes.fromhex(RUST_TAG[1]))
+    assert owner_pubkey.verify(
+        bytes.fromhex(RUST_TAG[3]), preimage_digest(RUST_AGENT_PUBKEY, "")
+    )
+
+
+def test_encode_nsec_matches_nip19_vector():
+    # NIP-19 reference vector from the spec.
+    assert (
+        encode_nsec("67dea2ed018072d675f5415ecfaed7d2597555e202d85b3d65ea4e58d2d92ffa")
+        == "nsec1vl029mgpspedva04g90vltkh6fvh240zqtv9k0t9af8935ke9laqsnlfe5"
+    )

benchmarks/harbor-buzz-orchestra/testbed/tests/test_provisioner_live.py

@@ -0,0 +1,123 @@
+"""Live provisioning tests against a running benchmark stack.
+
+Gated by BUZZ_TESTBED_LIVE=1 with stack coordinates in the environment:
+  BUZZ_TESTBED_RELAY_HTTP   (default http://localhost:3000)
+  BUZZ_TESTBED_RELAY_WS     (default ws://host.docker.internal:3000)
+  BUZZ_TESTBED_OWNER_KEY    relay owner secret key (hex)
+  BUZZ_TESTBED_PG_DSN       benchmark Postgres DSN
+"""
+
+from __future__ import annotations
+
+import os
+import uuid
+
+import psycopg
+import pytest
+from harbor_buzz_testbed.buzz_cli import BuzzCli, BuzzCliError
+from harbor_buzz_testbed.provisioner import (
+    BuzzTrialProvisioner,
+    ProvisioningError,
+    TestbedConfig,
+)
+
+pytestmark = pytest.mark.skipif(
+    os.environ.get("BUZZ_TESTBED_LIVE") != "1",
+    reason="live testbed suite; set BUZZ_TESTBED_LIVE=1 with a running stack",
+)
+
+
+@pytest.fixture()
+def provisioner() -> BuzzTrialProvisioner:
+    owner_key = os.environ.get("BUZZ_TESTBED_OWNER_KEY")
+    dsn = os.environ.get("BUZZ_TESTBED_PG_DSN")
+    if not owner_key or not dsn:
+        pytest.fail("BUZZ_TESTBED_OWNER_KEY and BUZZ_TESTBED_PG_DSN are required")
+    return BuzzTrialProvisioner(
+        TestbedConfig(
+            relay_http_url=os.environ.get(
+                "BUZZ_TESTBED_RELAY_HTTP", "http://localhost:3000"
+            ),
+            relay_ws_url=os.environ.get(
+                "BUZZ_TESTBED_RELAY_WS", "ws://host.docker.internal:3000"
+            ),
+            owner_secret_key=owner_key,
+            postgres_dsn=dsn,
+            llm_api_keys={
+                "databricks/glm": "test-glm-key",
+                "databricks/opus": "test-opus-key",
+            },
+        )
+    )
+
+
+def cli_for(provisioner: BuzzTrialProvisioner, credential) -> BuzzCli:
+    return provisioner._cli_for(credential)
+
+
+def test_create_is_idempotent_and_isolated(provisioner, manifest):
+    run_id = f"live-{uuid.uuid4().hex[:8]}"
+    trial_a = str(uuid.uuid4())
+    trial_b = str(uuid.uuid4())
+
+    provisioner.healthcheck()
+    handle_a = provisioner.create_trial(run_id, trial_a, manifest)
+    handle_b = provisioner.create_trial(run_id, trial_b, manifest)
+    try:
+        # Shape: one credential per roster slot, orchestrator first.
+        assert len(handle_a.credentials) == 3
+        assert handle_a.credentials[0].role == "orchestrator"
+        assert handle_a.manifest_hash == manifest.sha256
+
+        # Idempotency: same key returns the stored handle, keys included.
+        again = provisioner.create_trial(run_id, trial_a, manifest)
+        assert again == handle_a
+
+        # Isolation: trials get distinct channels and fresh keys...
+        assert handle_a.channel_id != handle_b.channel_id
+        keys_a = {c.nostr_secret_key for c in handle_a.credentials}
+        keys_b = {c.nostr_secret_key for c in handle_b.credentials}
+        assert not keys_a & keys_b
+
+        # ...and trial B's orchestrator cannot see trial A's channel.
+        cli_a = cli_for(provisioner, handle_a.credentials[0])
+        cli_b = cli_for(provisioner, handle_b.credentials[0])
+        cli_a.run(
+            "messages",
+            "send",
+            "--channel",
+            handle_a.channel_id,
+            "--content",
+            "trial A secret",
+        )
+        foreign_read = cli_b.run(
+            "messages", "get", "--channel", handle_a.channel_id, "--limit", "10"
+        )
+        assert foreign_read == [], "cross-trial read must return nothing"
+        with pytest.raises(BuzzCliError, match="private"):
+            cli_b.run("channels", "join", "--channel", handle_a.channel_id)
+
+        # Members can read their own channel.
+        own_read = cli_a.run(
+            "messages", "get", "--channel", handle_a.channel_id, "--limit", "10"
+        )
+        assert [m["content"] for m in own_read] == ["trial A secret"]
+
+        # Same trial key + different manifest must be rejected, not silently
+        # reprovisioned.
+        altered = manifest.model_copy(update={"condition": "O-altered"})
+        with pytest.raises(ProvisioningError, match="already provisioned"):
+            provisioner.create_trial(run_id, trial_a, altered)
+    finally:
+        provisioner.teardown(handle_a)
+        provisioner.teardown(handle_b)
+
+    # Teardown is idempotent and stamps archived_at.
+    provisioner.teardown(handle_a)
+    with psycopg.connect(os.environ["BUZZ_TESTBED_PG_DSN"]) as conn:
+        row = conn.execute(
+            "SELECT archived_at FROM benchmark.trial_manifest"
+            " WHERE run_id = %s AND trial_id = %s",
+            (run_id, trial_a),
+        ).fetchone()
+    assert row is not None and row[0] is not None

benchmarks/harbor-buzz-orchestra/testbed/tests/test_provisioner_unit.py

@@ -0,0 +1,118 @@
+"""Provisioner unit tests — no relay or Postgres required."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+
+import coincurve
+import pytest
+from harbor_buzz_testbed.provisioner import (
+    BuzzTrialProvisioner,
+    ProvisioningError,
+    TestbedConfig,
+)
+
+OWNER_SECRET = "0" * 63 + "3"
+
+
+def config(**overrides) -> TestbedConfig:
+    defaults = dict(
+        relay_http_url="http://localhost:3000",
+        relay_ws_url="ws://host.docker.internal:3000",
+        owner_secret_key=OWNER_SECRET,
+        postgres_dsn="postgresql://unused",
+        llm_api_keys={"databricks/glm": "glm-key", "databricks/opus": "opus-key"},
+    )
+    defaults.update(overrides)
+    return TestbedConfig(**defaults)
+
+
+def test_mint_credentials_expands_roster(manifest):
+    credentials = BuzzTrialProvisioner(config())._mint_credentials(manifest)
+    assert [c.agent_id for c in credentials] == [
+        "orch-opus-1",
+        "worker-glm-1",
+        "worker-glm-2",
+    ]
+    assert credentials[0].role == "orchestrator"
+    assert credentials[0].llm_api_key == "opus-key"
+    assert {c.llm_api_key for c in credentials[1:]} == {"glm-key"}
+
+
+def test_mint_credentials_keys_are_fresh_and_attested(manifest):
+    provisioner = BuzzTrialProvisioner(config())
+    first = provisioner._mint_credentials(manifest)
+    second = provisioner._mint_credentials(manifest)
+    all_secrets = [c.nostr_secret_key for c in first + second]
+    assert len(all_secrets) == len(set(all_secrets)), "keys must never be reused"
+    owner_pubkey = coincurve.PrivateKey(bytes.fromhex(OWNER_SECRET)).public_key_xonly
+    for credential in first:
+        tag = json.loads(credential.nostr_auth_tag)
+        assert tag[:3] == ["auth", owner_pubkey.format().hex(), ""]
+        digest = hashlib.sha256(
+            f"nostr:agent-auth:{credential.nostr_pubkey}:".encode()
+        ).digest()
+        assert owner_pubkey.verify(bytes.fromhex(tag[3]), digest)
+
+
+def test_mint_user_is_attested_and_not_an_agent():
+    provisioner = BuzzTrialProvisioner(config())
+    user = provisioner._mint_user()
+    assert user.agent_id == "user"
+    assert user.role == "user"
+    assert user.llm_endpoint == "" and user.llm_api_key == ""
+    owner_pubkey = coincurve.PrivateKey(bytes.fromhex(OWNER_SECRET)).public_key_xonly
+    tag = json.loads(user.nostr_auth_tag)
+    assert tag[:3] == ["auth", owner_pubkey.format().hex(), ""]
+
+
+def test_pinned_user_secret_reuses_one_identity():
+    pinned = "7" * 64
+    provisioner = BuzzTrialProvisioner(config(user_secret_key=pinned))
+    first = provisioner._mint_user()
+    second = provisioner._mint_user()
+    assert first.nostr_secret_key == pinned
+    assert first.nostr_pubkey == second.nostr_pubkey
+    expected = coincurve.PrivateKey(bytes.fromhex(pinned)).public_key_xonly
+    assert first.nostr_pubkey == expected.format().hex()
+    # Still a user, still attested by the owner.
+    assert first.role == "user"
+    owner_pubkey = coincurve.PrivateKey(bytes.fromhex(OWNER_SECRET)).public_key_xonly
+    assert json.loads(first.nostr_auth_tag)[1] == owner_pubkey.format().hex()
+
+
+def test_teardown_skips_archiving_when_disabled():
+    provisioner = BuzzTrialProvisioner(config(archive_on_teardown=False))
+    # Any attribute access would fail on this handle — teardown must return
+    # before touching the CLI or Postgres.
+    provisioner.teardown(handle=None)
+
+
+def test_mint_credentials_missing_api_key_is_explicit(manifest):
+    provisioner = BuzzTrialProvisioner(config(llm_api_keys={}))
+    with pytest.raises(ProvisioningError, match="databricks/"):
+        provisioner._mint_credentials(manifest)
+
+
+def test_lock_key_is_deterministic_and_distinct():
+    calls: list[int] = []
+
+    class FakeConn:
+        def execute(self, _query, params):
+            calls.append(params[0])
+
+    BuzzTrialProvisioner._lock_trial(FakeConn(), "run-a", "trial-1")
+    BuzzTrialProvisioner._lock_trial(FakeConn(), "run-a", "trial-1")
+    BuzzTrialProvisioner._lock_trial(FakeConn(), "run-a", "trial-2")
+    assert calls[0] == calls[1]
+    assert calls[0] != calls[2]
+    assert all(-(2**63) <= key < 2**63 for key in calls)
+
+
+def test_healthcheck_fails_fast_when_relay_down():
+    provisioner = BuzzTrialProvisioner(
+        config(relay_http_url="http://localhost:1", postgres_dsn="postgresql://unused")
+    )
+    with pytest.raises(ProvisioningError, match="relay unreachable"):
+        provisioner.healthcheck()

benchmarks/harbor-buzz-orchestra/tests/conftest.py

@@ -0,0 +1,52 @@
+from typing import Any
+import pytest
+
+
+@pytest.fixture
+def manifest_data() -> dict[str, Any]:
+    digest = "a" * 64
+    generation = {
+        "temperature": 0,
+        "max_output_tokens": 4096,
+        "context_window_tokens": 200000,
+    }
+    return {
+        "schema_version": "1",
+        "condition": "O",
+        "roster": [
+            {
+                "id": "opus-orchestrator",
+                "kind": "orchestrator",
+                "role": "orchestrator",
+                "count": 1,
+                "endpoint": "databricks/opus",
+                "model_revision": "2026-07-01",
+                "prompt": {"path": "prompts/orchestrator.md", "sha256": digest},
+                "generation": generation,
+            },
+            {
+                "id": "qwen-workers",
+                "kind": "worker",
+                "role": "fast-worker",
+                "count": 4,
+                "concurrency": 2,
+                "endpoint": "databricks/qwen",
+                "model_revision": "revision-42",
+                "prompt": {"path": "prompts/qwen.md", "sha256": digest},
+                "generation": {**generation, "context_window_tokens": 32768},
+            },
+        ],
+        "prices": {
+            "databricks/opus": {
+                "input_per_million_usd": 5,
+                "cached_input_per_million_usd": 0.5,
+                "output_per_million_usd": 25,
+            },
+            "databricks/qwen": {
+                "input_per_million_usd": 0.1,
+                "cached_input_per_million_usd": 0.01,
+                "output_per_million_usd": 0.2,
+            },
+        },
+        "trial_budget": {"timeout_seconds": 3600, "max_cost_usd": 5},
+    }

benchmarks/harbor-buzz-orchestra/tests/test_agent.py

@@ -0,0 +1,169 @@
+from types import SimpleNamespace
+from uuid import uuid4
+import pytest
+from harbor.models.agent.context import AgentContext
+from harbor_buzz_orchestra import (
+    AgentCredential,
+    BuzzOrchestraAgent,
+    RuntimeResult,
+    TrialHandle,
+)
+
+pytestmark = pytest.mark.asyncio
+
+
+async def test_agent_credential_carries_closed_relay_attestation():
+    credential = AgentCredential(
+        agent_id="orchestrator-1",
+        role="orchestrator",
+        nostr_secret_key="11" * 32,
+        nostr_pubkey="22" * 32,
+        nostr_auth_tag='["auth","owner","conditions","signature"]',
+        llm_endpoint="https://example.databricks.com/serving-endpoints/opus",
+        llm_api_key="attributed-key",
+    )
+
+    assert credential.nostr_auth_tag.startswith('["auth"')
+
+
+class Provisioner:
+    def __init__(self):
+        self.healthchecked = False
+        self.created = None
+        self.torn_down = None
+
+    def healthcheck(self):
+        self.healthchecked = True
+
+    def create_trial(self, run_id, trial_id, manifest, channel_label=None):
+        self.created = (run_id, trial_id, manifest, channel_label)
+        return TrialHandle(
+            run_id,
+            trial_id,
+            manifest.sha256,
+            "ws://relay",
+            "channel-1",
+            (),
+            user=AgentCredential(
+                agent_id="user",
+                role="user",
+                nostr_secret_key="s",
+                nostr_pubkey="p",
+                nostr_auth_tag="[]",
+                llm_endpoint="",
+                llm_api_key="",
+            ),
+        )
+
+    def teardown(self, handle):
+        self.torn_down = handle
+
+
+class Runtime:
+    def __init__(self, error=None):
+        self.called = None
+        self.error = error
+
+    async def run(self, **kwargs):
+        self.called = kwargs
+        if self.error:
+            raise self.error
+        return RuntimeResult(10, 2, 3, 0.25, {"receipt_status": "pending"})
+
+
+async def test_agent_lifecycle_and_context(tmp_path, manifest_data):
+    provisioner, runtime, context_id = Provisioner(), Runtime(), uuid4()
+    environment = SimpleNamespace(
+        context_id=context_id, environment_name="hello-world"
+    )
+    agent = BuzzOrchestraAgent(
+        logs_dir=tmp_path,
+        manifest=manifest_data,
+        provisioner=provisioner,
+        runtime=runtime,
+        run_id="run-1",
+    )
+    agent.context_id = context_id
+    context = AgentContext()
+    await agent.setup(environment)
+    await agent.run("solve it", environment, context)
+    assert provisioner.healthchecked
+    assert provisioner.created[:2] == ("run-1", str(context_id))
+    # The task short name labels the trial channel for spectator GUIs.
+    assert provisioner.created[3] == "hello-world"
+    assert provisioner.torn_down.channel_id == "channel-1"
+    assert runtime.called["instruction"] == "solve it"
+    assert (
+        context.n_input_tokens,
+        context.n_cache_tokens,
+        context.n_output_tokens,
+        context.cost_usd,
+    ) == (10, 2, 3, 0.25)
+    assert context.metadata["manifest_sha256"] == agent.manifest.sha256
+    assert context.metadata["trial_id"] == str(context_id)
+
+
+async def test_teardown_runs_when_runtime_fails(tmp_path, manifest_data):
+    provisioner, runtime, context_id = (
+        Provisioner(),
+        Runtime(RuntimeError("runtime failed")),
+        uuid4(),
+    )
+    environment = SimpleNamespace(context_id=context_id)
+    agent = BuzzOrchestraAgent(
+        logs_dir=tmp_path,
+        manifest=manifest_data,
+        provisioner=provisioner,
+        runtime=runtime,
+    )
+    agent.context_id = context_id
+    with pytest.raises(RuntimeError, match="runtime failed"):
+        await agent.run("solve it", environment, AgentContext())
+    assert provisioner.torn_down.channel_id == "channel-1"
+
+
+async def test_missing_integrations_fail_explicitly(tmp_path, manifest_data):
+    agent = BuzzOrchestraAgent(logs_dir=tmp_path, manifest=manifest_data)
+    with pytest.raises(RuntimeError, match="M1 wiring is incomplete"):
+        await agent.run("solve it", SimpleNamespace(context_id=uuid4()), AgentContext())
+
+
+async def test_cli_runtime_construction_from_json(tmp_path, manifest_data):
+    endpoint_path = tmp_path / "endpoints.json"
+    endpoint_path.write_text(
+        '{"frontier/rev":{"provider":"anthropic",'
+        '"api_key_env":"ANTHROPIC_API_KEY"},'
+        '"worker/rev":{"provider":"openai",'
+        '"api_key_env":"OPENAI_API_KEY"}}'
+    )
+    agent = BuzzOrchestraAgent(
+        logs_dir=tmp_path / "logs",
+        manifest=manifest_data,
+        artifact_root=tmp_path,
+        endpoint_config=endpoint_path,
+        buzz_acp_binary="/pinned/buzz-acp",
+        buzz_agent_binary="/pinned/buzz-agent",
+        buzz_dev_mcp_binary="/pinned/buzz-dev-mcp",
+        buzz_cli_binary="/pinned/buzz",
+    )
+    assert agent.runtime.artifact_root == tmp_path
+    assert agent.runtime.endpoints["frontier/rev"].provider == "anthropic"
+    assert agent.runtime.buzz_acp_binary == "/pinned/buzz-acp"
+    assert agent.runtime.buzz_agent_binary == "/pinned/buzz-agent"
+    assert agent.runtime.buzz_dev_mcp_binary == "/pinned/buzz-dev-mcp"
+    assert agent.runtime.buzz_cli_binary == "/pinned/buzz"
+
+
+async def test_cli_construction_requires_complete_pairs(tmp_path, manifest_data):
+    with pytest.raises(ValueError, match="artifact_root"):
+        BuzzOrchestraAgent(
+            logs_dir=tmp_path,
+            manifest=manifest_data,
+            endpoint_config={},
+        )
+    with pytest.raises(ValueError, match="provisioner_factory"):
+        BuzzOrchestraAgent(
+            logs_dir=tmp_path,
+            manifest=manifest_data,
+            provisioner_config={},
+        )

benchmarks/harbor-buzz-orchestra/tests/test_container_runtime.py

@@ -0,0 +1,426 @@
+"""The container runtime must launch the production stack, unmodified."""
+
+import hashlib
+import json
+import re
+from pathlib import Path
+
+import pytest
+from harbor.environments.base import ExecResult
+
+from harbor_buzz_orchestra.manifest import ExperimentManifest
+from harbor_buzz_orchestra.provisioning import AgentCredential, TrialHandle
+from harbor_buzz_orchestra.container_runtime import (
+    REMOTE_BIN,
+    REMOTE_LOGS,
+    BuzzContainerRuntime,
+    EndpointLaunchConfig,
+    RuntimeLaunchError,
+)
+
+
+def write_manifest(tmp_path: Path) -> ExperimentManifest:
+    prompt = tmp_path / "prompt.md"
+    prompt.write_text("prompt", encoding="utf-8")
+    digest = hashlib.sha256(prompt.read_bytes()).hexdigest()
+    roster_entry = {
+        "count": 1,
+        "model_revision": "r1",
+        "prompt": {"path": "prompt.md", "sha256": digest},
+        "generation": {"max_output_tokens": 100, "context_window_tokens": 1000},
+    }
+    return ExperimentManifest.load(
+        {
+            "condition": "test",
+            "roster": [
+                {"id": "orch", "kind": "orchestrator", "role": "lead",
+                 "endpoint": "orch-model", **roster_entry},
+                {"id": "worker", "kind": "worker", "role": "implementer",
+                 "endpoint": "worker-model", **roster_entry},
+            ],
+            "prices": {
+                name: {
+                    "input_per_million_usd": 0,
+                    "cached_input_per_million_usd": 0,
+                    "output_per_million_usd": 0,
+                }
+                for name in ("orch-model", "worker-model")
+            },
+            "trial_budget": {"timeout_seconds": 30},
+        }
+    )
+
+
+def credential(agent_id, role, endpoint):
+    return AgentCredential(
+        agent_id=agent_id,
+        role=role,
+        nostr_secret_key=f"secret-{agent_id}",
+        nostr_pubkey=f"pubkey-{agent_id}",
+        nostr_auth_tag="[]",
+        llm_endpoint=endpoint,
+        llm_api_key=f"key-{agent_id}",
+    )
+
+
+def user_credential():
+    return AgentCredential(
+        agent_id="user",
+        role="user",
+        nostr_secret_key="secret-user",
+        nostr_pubkey="pubkey-user",
+        nostr_auth_tag="[]",
+        llm_endpoint="",
+        llm_api_key="",
+    )
+
+
+def trial_handle(credentials, user_relay_url=""):
+    return TrialHandle(
+        run_id="run",
+        trial_id="trial",
+        manifest_hash="hash",
+        relay_ws_url="ws://host.docker.internal:3600",
+        channel_id="channel",
+        credentials=credentials,
+        user=user_credential(),
+        user_relay_url=user_relay_url,
+    )
+
+
+def runtime(tmp_path, **kwargs):
+    return BuzzContainerRuntime(
+        logs_dir=tmp_path / "logs",
+        artifact_root=tmp_path,
+        endpoints={
+            "orch-model": EndpointLaunchConfig("anthropic", "ANTHROPIC_API_KEY"),
+            "worker-model": EndpointLaunchConfig("anthropic", "ANTHROPIC_API_KEY"),
+        },
+        **kwargs,
+    )
+
+
+class Environment:
+    """Records execs/uploads; scripted stdout per command substring."""
+
+    def __init__(self, responses=None):
+        self.commands = []
+        self.uploads = []
+        self.responses = responses or {}
+
+    async def exec(self, command, env=None, **kwargs):
+        self.commands.append((command, env))
+        for needle, result in self.responses.items():
+            if needle in command:
+                return result
+        return ExecResult(stdout="", stderr="", return_code=0)
+
+    async def upload_file(self, source, target):
+        self.uploads.append((str(source), target))
+
+    async def download_dir(self, source, target):
+        pass
+
+
+def test_maps_credentials_exactly_and_rejects_role_mismatch(tmp_path):
+    manifest = write_manifest(tmp_path)
+    credentials = (
+        credential("orch-1", "orchestrator", "orch-model"),
+        credential("worker-1", "worker", "worker-model"),
+    )
+    assert set(runtime(tmp_path)._classes_by_agent_id(manifest, credentials)) == {
+        "orch-1",
+        "worker-1",
+    }
+    bad = (credential("worker-1", "orchestrator", "worker-model"),)
+    with pytest.raises(RuntimeLaunchError, match="role"):
+        runtime(tmp_path)._classes_by_agent_id(manifest, bad)
+
+
+def test_prompt_hash_and_identity_override_are_fail_closed(tmp_path):
+    manifest = write_manifest(tmp_path)
+    prompt_ref = manifest.roster[0].prompt
+    runtime(tmp_path)._verify_artifact(tmp_path / prompt_ref.path, prompt_ref.sha256)
+    (tmp_path / prompt_ref.path).write_text("changed", encoding="utf-8")
+    with pytest.raises(RuntimeLaunchError, match="hash mismatch"):
+        runtime(tmp_path)._verify_artifact(
+            tmp_path / prompt_ref.path, prompt_ref.sha256
+        )
+
+    endpoint = EndpointLaunchConfig(
+        "anthropic", "ANTHROPIC_API_KEY", {"BUZZ_ACP_MCP_COMMAND": "evil"}
+    )
+    with pytest.raises(RuntimeLaunchError, match="identity"):
+        runtime(tmp_path)._reject_identity_overrides(endpoint)
+
+
+def test_user_relay_url_prefers_host_view(tmp_path):
+    rt = runtime(tmp_path)
+    # v1.2 handles carry the host view for the trial user explicitly.
+    assert (
+        rt._user_relay_url(trial_handle((), user_relay_url="http://localhost:3600"))
+        == "http://localhost:3600"
+    )
+    # pre-v1.2 handles fall back to deriving http from the agents' ws view.
+    assert (
+        rt._user_relay_url(trial_handle(()))
+        == "http://host.docker.internal:3600"
+    )
+    with pytest.raises(RuntimeLaunchError, match="ws://"):
+        rt._cli_relay_url("http://relay")
+
+
+async def test_install_stack_uploads_the_pinned_stack(tmp_path):
+    binaries = {}
+    for name in ("buzz-acp", "buzz-agent", "buzz-dev-mcp"):
+        path = tmp_path / name
+        path.write_text("#!binary")
+        binaries[name] = str(path)
+    rt = runtime(
+        tmp_path,
+        buzz_acp_binary=binaries["buzz-acp"],
+        buzz_agent_binary=binaries["buzz-agent"],
+        buzz_dev_mcp_binary=binaries["buzz-dev-mcp"],
+    )
+    environment = Environment()
+    await rt._install_stack(environment)
+    assert {target for _, target in environment.uploads} == {
+        f"{REMOTE_BIN}/buzz-acp",
+        f"{REMOTE_BIN}/buzz-agent",
+        f"{REMOTE_BIN}/buzz-dev-mcp",
+    }
+    assert any("chmod 0755" in cmd for cmd, _ in environment.commands)
+
+
+async def test_install_stack_requires_binaries_on_disk(tmp_path):
+    rt = runtime(tmp_path, buzz_acp_binary=str(tmp_path / "missing"))
+    with pytest.raises(RuntimeLaunchError, match="binary not found"):
+        await rt._install_stack(Environment())
+
+
+async def test_forwarder_bridges_the_canonical_relay_address(tmp_path):
+    from harbor_buzz_orchestra.container_runtime import FORWARDER
+
+    forwarder = tmp_path / "relay-forwarder"
+    forwarder.write_text("ELF")
+    rt = runtime(
+        tmp_path,
+        relay_gateway="host.docker.internal:3600",
+        forwarder_binary=str(forwarder),
+    )
+    trial = TrialHandle(
+        run_id="run", trial_id="trial", manifest_hash="hash",
+        relay_ws_url="ws://localhost:3600", channel_id="channel",
+        credentials=(), user=user_credential(),
+    )
+    environment = Environment(
+        responses={
+            FORWARDER: ExecResult(stdout="99\n", stderr="", return_code=0),
+            "cat ": ExecResult(
+                stdout="forwarding 127.0.0.1:3600 -> host.docker.internal:3600",
+                stderr="", return_code=0,
+            ),
+        }
+    )
+    agent = await rt._start_forwarder(environment, trial)
+    assert agent is not None and agent.pid == 99
+    launch = next(cmd for cmd, _ in environment.commands if FORWARDER in cmd)
+    # Listens on the canonical loopback (host-header bound), targets the gateway.
+    assert "127.0.0.1:3600" in launch
+    assert "host.docker.internal:3600" in launch
+
+    # No gateway configured: the relay is reachable directly, no forwarder.
+    assert await runtime(tmp_path)._start_forwarder(Environment(), trial) is None
+    with pytest.raises(RuntimeLaunchError, match="ws://"):
+        rt._ws_authority("http://relay")
+
+
+@pytest.mark.parametrize(("configured", "expected"), [(None, "32"), (7, "7")])
+async def test_launch_wires_the_desktop_environment(tmp_path, configured, expected):
+    manifest = write_manifest(tmp_path)
+    agent_class = manifest.roster[0]
+    if configured is not None:
+        agent_class = agent_class.model_copy(
+            update={
+                "budget": agent_class.budget.model_copy(
+                    update={"max_calls": configured}
+                )
+            }
+        )
+    orch = credential("orch-1", "orchestrator", "orch-model")
+    trial = trial_handle((orch,))
+    environment = Environment(
+        responses={"buzz-acp": ExecResult(stdout="4242\n", stderr="", return_code=0)}
+    )
+    agent = await runtime(tmp_path)._launch_agent(
+        environment=environment,
+        trial=trial,
+        credential=orch,
+        agent_class=agent_class,
+        trial_dir=tmp_path,
+    )
+    assert agent.pid == 4242
+    command, env = environment.commands[-1]
+    assert f"{REMOTE_BIN}/buzz-acp" in command
+    # The real product wiring: acp spawns buzz-agent, which gets buzz-dev-mcp.
+    assert env["BUZZ_ACP_AGENT_COMMAND"] == f"{REMOTE_BIN}/buzz-agent"
+    assert env["BUZZ_ACP_MCP_COMMAND"] == f"{REMOTE_BIN}/buzz-dev-mcp"
+    assert env["BUZZ_RELAY_URL"] == trial.relay_ws_url
+    assert env["BUZZ_PRIVATE_KEY"] == orch.nostr_secret_key
+    assert env["NOSTR_PRIVATE_KEY"] == orch.nostr_secret_key
+    assert env["BUZZ_AGENT_NO_HINTS"] == "1"
+    assert env["BUZZ_AGENT_MAX_ROUNDS"] == expected
+    assert env["BUZZ_ACP_SYSTEM_PROMPT_FILE"].endswith("orch-1.system-prompt.md")
+    # The composed prompt was uploaded into the container.
+    assert any(
+        target == env["BUZZ_ACP_SYSTEM_PROMPT_FILE"]
+        for _, target in environment.uploads
+    )
+
+
+def test_runtime_rejects_unbounded_agent_rounds(tmp_path):
+    with pytest.raises(ValueError, match="positive"):
+        runtime(tmp_path, max_agent_rounds=0)
+    with pytest.raises(ValueError, match="positive"):
+        runtime(tmp_path, readiness_timeout_seconds=0)
+
+
+async def test_wait_for_agents_ready_requires_every_channel_subscription(tmp_path):
+    rt = runtime(tmp_path, poll_seconds=0)
+    logs = {"orch-1": "", "worker-1": ""}
+
+    class ReadyEnvironment(Environment):
+        polls = 0
+
+        async def exec(self, command, env=None, **kwargs):
+            if command.startswith("cat "):
+                agent_id = re.search(r"([\w-]+)\.stdout\.log", command).group(1)
+                return ExecResult(
+                    stdout=logs[agent_id], stderr="", return_code=0
+                )
+            return ExecResult(stdout="", stderr="", return_code=0)
+
+    from harbor_buzz_orchestra.container_runtime import _Agent
+
+    agents = [
+        _Agent(
+            credential(agent_id, "worker", "worker-model"),
+            pid=1,
+            stdout_log=f"{REMOTE_LOGS}/{agent_id}.stdout.log",
+            stderr_log=f"{REMOTE_LOGS}/{agent_id}.stderr.log",
+        )
+        for agent_id in logs
+    ]
+    logs["orch-1"] = "subscribed to channel trial-channel\n"
+    logs["worker-1"] = "subscribed to channel trial-channel\n"
+    await rt._wait_for_agents_ready(ReadyEnvironment(), agents, "trial-channel")
+
+    logs["worker-1"] = ""
+    rt_timeout = runtime(tmp_path, poll_seconds=0, readiness_timeout_seconds=0.01)
+    with pytest.raises(RuntimeLaunchError, match="worker-1"):
+        await rt_timeout._wait_for_agents_ready(
+            ReadyEnvironment(), agents, "trial-channel"
+        )
+
+
+async def test_dead_agent_processes_fail_the_trial(tmp_path):
+    from harbor_buzz_orchestra.container_runtime import _Agent
+
+    agents = [
+        _Agent(credential("worker-1", "worker", "worker-model"), 7, "o", "e")
+    ]
+    environment = Environment(
+        responses={
+            "kill -0": ExecResult(stdout="DEAD:worker-1\n", stderr="", return_code=0)
+        }
+    )
+    with pytest.raises(RuntimeLaunchError, match="worker-1"):
+        await runtime(tmp_path)._raise_for_dead_agents(environment, agents)
+
+
+@pytest.mark.parametrize(
+    ("condition", "return_code", "raises"),
+    [
+        ("M1-hello-world", 0, False),
+        ("M1-hello-world", 1, True),
+        ("other", 1, False),
+    ],
+)
+async def test_m1_output_probe_matches_grader_and_is_condition_scoped(
+    tmp_path, condition, return_code, raises
+):
+    manifest = write_manifest(tmp_path).model_copy(update={"condition": condition})
+    environment = Environment(
+        responses={
+            "hello.txt": ExecResult(stdout="", stderr="", return_code=return_code)
+        }
+    )
+    if raises:
+        with pytest.raises(RuntimeLaunchError, match="/app/hello.txt"):
+            await runtime(tmp_path)._verify_m1_output(environment, manifest)
+    else:
+        await runtime(tmp_path)._verify_m1_output(environment, manifest)
+    probed = [cmd for cmd, _ in environment.commands if "hello.txt" in cmd]
+    assert bool(probed) == (condition == "M1-hello-world")
+
+
+async def test_wait_for_done_requires_orchestrator_authorship(tmp_path, monkeypatch):
+    rt = runtime(tmp_path, poll_seconds=0)
+    orch = credential("orch-1", "orchestrator", "orch-model")
+    trial = trial_handle((orch,))
+    rounds = iter(
+        [
+            [{"id": "1", "pubkey": "someone-else", "content": "DONE: fake"}],
+            [{"id": "2", "pubkey": orch.nostr_pubkey, "content": "DONE: real"}],
+        ]
+    )
+    observers = []
+
+    async def buzz_json(credential, *args, **kwargs):
+        observers.append(credential.agent_id)
+        return next(rounds)
+
+    monkeypatch.setattr(rt, "_buzz_json", buzz_json)
+    result = await rt._wait_for_done(Environment(), orch, trial, [])
+    assert json.dumps(result).find("real") > 0
+    # observation happens as the trial user, never as an agent identity
+    assert set(observers) == {"user"}
+
+
+def test_composed_system_prompt_carries_persona_and_team_roster(tmp_path):
+    rt = runtime(tmp_path)
+    orch = credential("orch-1", "orchestrator", "orch-model")
+    worker_1 = credential("worker-1", "worker", "worker-model")
+    worker_2 = credential("worker-2", "worker", "worker-model")
+    trial = trial_handle((orch, worker_1, worker_2))
+    persona = tmp_path / "persona.md"
+    persona.write_text("# Persona body\n", encoding="utf-8")
+
+    path = rt._compose_system_prompt(
+        trial_dir=tmp_path,
+        trial=trial,
+        credential=orch,
+        persona_path=persona,
+    )
+
+    composed = path.read_text(encoding="utf-8")
+    assert composed.startswith("# Persona body\n")
+    assert "You are `orch-1` (pubkey `pubkey-orch-1`)" in composed
+    assert f"channel `{trial.channel_id}`" in composed
+    assert "user `user` (pubkey `pubkey-user`)" in composed
+    # roster lists teammates, never the agent itself
+    assert "| worker-1 | worker | `pubkey-worker-1` |" in composed
+    assert "| worker-2 | worker | `pubkey-worker-2` |" in composed
+    assert "| orch-1 " not in composed
+    assert path.stat().st_mode & 0o777 == 0o600
+
+
+async def test_stop_agents_sweeps_the_uploaded_stack(tmp_path):
+    from harbor_buzz_orchestra.container_runtime import _Agent
+
+    environment = Environment()
+    agents = [_Agent(credential("orch-1", "orchestrator", "orch-model"), 1, "o", "e")]
+    await BuzzContainerRuntime._stop_agents(environment, agents)
+    sweeps = [cmd for cmd, _ in environment.commands if REMOTE_BIN in cmd]
+    assert len(sweeps) == 2
+    assert "kill -TERM" in sweeps[0] and "kill -KILL" in sweeps[1]

benchmarks/harbor-buzz-orchestra/tests/test_manifest.py

@@ -0,0 +1,47 @@
+import copy
+import pytest
+import yaml
+from harbor_buzz_orchestra import ExperimentManifest, ManifestError
+
+
+def test_hash_is_independent_of_mapping_and_yaml_key_order(tmp_path, manifest_data):
+    first = ExperimentManifest.load(manifest_data)
+    path = tmp_path / "manifest.yaml"
+    path.write_text(
+        yaml.safe_dump(
+            dict(reversed(list(copy.deepcopy(manifest_data).items()))), sort_keys=False
+        )
+    )
+    second = ExperimentManifest.load(path)
+    assert first.canonical_bytes() == second.canonical_bytes()
+    assert first.sha256 == second.sha256
+    assert len(first.sha256) == 64
+
+
+def test_hash_changes_when_staffing_changes(manifest_data):
+    first = ExperimentManifest.load(manifest_data)
+    changed = copy.deepcopy(manifest_data)
+    changed["roster"][1]["count"] = 3
+    assert ExperimentManifest.load(changed).sha256 != first.sha256
+
+
+@pytest.mark.parametrize(
+    ("mutation", "match"),
+    [
+        (lambda data: data.update({"unknown": True}), "Extra inputs"),
+        (lambda data: data["roster"].pop(0), "exactly one orchestrator"),
+        (lambda data: data["prices"].pop("databricks/qwen"), "prices missing"),
+        (lambda data: data["roster"][1].update({"concurrency": 5}), "concurrency"),
+    ],
+)
+def test_invalid_manifest_is_rejected(manifest_data, mutation, match):
+    mutation(manifest_data)
+    with pytest.raises(ManifestError, match=match):
+        ExperimentManifest.load(manifest_data)
+
+
+def test_non_mapping_document_is_rejected(tmp_path):
+    path = tmp_path / "manifest.yaml"
+    path.write_text("- not\n- a\n- mapping\n")
+    with pytest.raises(ManifestError, match="root must be a mapping"):
+        ExperimentManifest.load(path)

benchmarks/harbor-buzz-orchestra/tests/test_run_leaderboard.py

@@ -0,0 +1,125 @@
+"""The leaderboard runner must emit only leaderboard-legal harbor settings."""
+
+import importlib.util
+import json
+import sys
+from pathlib import Path
+
+import pytest
+import yaml
+
+_SCRIPT = Path(__file__).parent.parent / "scripts" / "run_leaderboard.py"
+_spec = importlib.util.spec_from_file_location("run_leaderboard", _SCRIPT)
+run_leaderboard = importlib.util.module_from_spec(_spec)
+sys.modules["run_leaderboard"] = run_leaderboard
+_spec.loader.exec_module(run_leaderboard)
+
+FORBIDDEN_FLAGS = (
+    "--timeout-multiplier",
+    "--agent-timeout-multiplier",
+    "--verifier-timeout-multiplier",
+    "--agent-setup-timeout-multiplier",
+    "--environment-build-timeout-multiplier",
+    "--override-cpus",
+    "--override-memory",
+    "--override-storage",
+    "--override-gpus",
+)
+
+
+@pytest.fixture
+def binaries(tmp_path):
+    bin_dir = tmp_path / "bin"
+    bin_dir.mkdir()
+    found = {}
+    for name in run_leaderboard.BINARIES:
+        path = bin_dir / name
+        path.write_text("#!/bin/sh\n")
+        found[name] = path
+    return found
+
+
+@pytest.fixture
+def agent_binaries(tmp_path):
+    bin_dir = tmp_path / "linux-bin"
+    bin_dir.mkdir()
+    for name in run_leaderboard.AGENT_BINARIES:
+        (bin_dir / name).write_text("ELF")
+    return run_leaderboard.find_agent_binaries(bin_dir)
+
+
+@pytest.fixture
+def args(tmp_path, binaries, agent_binaries):
+    manifest = tmp_path / "team.yaml"
+    manifest.write_text(
+        yaml.safe_dump(
+            {
+                "condition": "unit-test",
+                "roster": [
+                    {"id": "orch", "endpoint": "frontier"},
+                    {"id": "worker", "endpoint": "fast", "count": 2},
+                ],
+            }
+        )
+    )
+    endpoints = tmp_path / "endpoints.json"
+    endpoints.write_text(json.dumps({"frontier": {"provider": "anthropic"}}))
+    provisioner = tmp_path / "provisioner.json"
+    provisioner.write_text("{}")
+    return run_leaderboard.parse_args(
+        [
+            "--dataset",
+            "terminal-bench/terminal-bench-2-1",
+            "--attempts",
+            "5",
+            "--manifest",
+            str(manifest),
+            "--endpoint-config",
+            str(endpoints),
+            "--provisioner-config",
+            str(provisioner),
+            "--agent-bin-dir",
+            str(next(iter(agent_binaries.values())).parent),
+            "--job-name",
+            "unit-test-job",
+        ]
+    )
+
+
+def test_command_uses_standard_settings_only(args, binaries, agent_binaries):
+    command = run_leaderboard.build_command(args, binaries, agent_binaries)
+    assert command[:2] == ["harbor", "run"]
+    assert command.count("-k") == 1
+    assert command[command.index("-k") + 1] == "5"
+    for flag in FORBIDDEN_FLAGS:
+        assert flag not in command
+    # The full production stack rides in as agent kwargs.
+    kwargs = [command[i + 1] for i, p in enumerate(command) if p == "--agent-kwarg"]
+    assert any(k.startswith("buzz_acp_binary=") for k in kwargs)
+    assert any(k.startswith("buzz_agent_binary=") for k in kwargs)
+    assert any(k.startswith("buzz_dev_mcp_binary=") for k in kwargs)
+
+
+def test_agent_binaries_must_exist(tmp_path):
+    with pytest.raises(SystemExit, match="buzz-dev-mcp"):
+        run_leaderboard.find_agent_binaries(tmp_path)
+
+
+def test_forbidden_flags_are_not_accepted(tmp_path):
+    for flag in FORBIDDEN_FLAGS:
+        with pytest.raises(SystemExit):
+            run_leaderboard.parse_args(
+                ["--dataset", "d", "--attempts", "5",
+                 "--agent-bin-dir", str(tmp_path), flag, "1"]
+            )
+
+
+def test_metadata_template_matches_harbor_schema(args, tmp_path):
+    from harbor.leaderboard.metadata import load_metadata
+
+    path = run_leaderboard.write_metadata_template(args, tmp_path)
+    loaded = load_metadata(path)
+    assert loaded["agent_org_display_name"] == "Block"
+    assert [m["model_name"] for m in loaded["models"]] == ["frontier", "fast"]
+    assert loaded["models"][0]["model_org_display_name"] == "Anthropic"
+    assert loaded["models"][1]["model_provider"] == "FILL_ME"