mirror of
https://github.com/block/buzz.git
synced 2026-11-05 17:01:35 +00:00
f463e726dd
## Why Evaluating buzz agent memory retrieval by seeding a memory then asking the buzz agent a question it needs that memory. **Bug Found**: System prompt had no inclusion of retrieving cold memories and suggested looking in a mem/*.md directory that does not exist. Updated `system-prompt.md` to include memory CLI tools and usage. Eval Before System Prompt Change: 0/3 Eval After System Prompt Change: 3/3 ## What - Add a `memory-retrieval` benchmark that seeds agent memory with `buzz mem set` before asking a direct question. - Grade the observable threaded answer without inspecting tool calls or exposing the answer in channel history. - Teach agents to use `buzz mem set`, `buzz mem ls`, and `buzz mem get` for cold memory. - Add a wire-debug endpoint configuration for diagnosing ACP tool calls in local runs. - Add fixture, seeding, verifier, and prompt coverage. ## Risk Assessment Low. The runtime changes are limited to the benchmark harness. The production-facing change clarifies existing memory commands in the base prompt; it does not change memory storage, relay behavior, or authorization. ## References - Before the system-prompt changes, 0/3 attempts passed because agents never invoked the `buzz mem` CLI and instead searched a non existent filesystem - After the changes, 3/3 attempts passed. ACP wire logs confirmed that every agent ran `buzz mem ls` followed by `buzz mem get` and returned `net_gpv`. --------- Signed-off-by: Philip Azar <pazar@squareup.com>
34 lines
1.5 KiB
Markdown
34 lines
1.5 KiB
Markdown
# Endpoint launch configs
|
|
|
|
Deployment-time mapping from manifest endpoint names to
|
|
`EndpointLaunchConfig` (`provider` / `api_key_env` / `env`), passed to
|
|
`harbor run` as `--agent-kwarg endpoint_config=<path>`. This is deployment
|
|
config, deliberately OUTSIDE the immutable condition manifest — the manifest
|
|
endpoint string remains the join key.
|
|
|
|
Every key in these files must be a manifest endpoint name; the loader treats
|
|
all entries as endpoint configs (no comment keys).
|
|
|
|
## openai-live-wire-debug.json
|
|
|
|
Diagnostic variant of `openai-live.json` for local runs. It enables
|
|
`acp::wire=debug`, so retained agent stdout logs include full ACP messages,
|
|
including tool-call arguments and results. These logs may contain prompt or
|
|
command content; keep them local. The verifier and reward do not read them.
|
|
|
|
## m1-local.json
|
|
|
|
M1 wiring proof: both placeholder endpoints resolve to one local llama-server
|
|
(OpenAI-compatible, `http://127.0.0.1:8091/v1`, no cloud keys).
|
|
|
|
buzz-agent env contract (crates/buzz-agent/src/config.rs, pinned at the M1
|
|
binary SHA): `provider=openai` reads `OPENAI_COMPAT_API_KEY` +
|
|
`OPENAI_COMPAT_BASE_URL`; the runtime sets `BUZZ_AGENT_MODEL` from the
|
|
manifest endpoint name, which overrides `OPENAI_COMPAT_MODEL` — llama-server
|
|
ignores the model name, so the placeholder value is harmless there.
|
|
llama-server needs no real key; the provisioner's per-endpoint
|
|
`llm_api_keys` map supplies a dummy value.
|
|
|
|
The Databricks pilot config is the same file shape with real serving
|
|
endpoint hosts/keys.
|