mirror of
https://github.com/block/buzz.git
synced 2026-11-16 06:11:34 +00:00
## What this does Introduces the model-capability manifest infrastructure (Phase 1 of the Model-Capability Manifest plan v4, Thufir-approved 9/9/9). No consumer cutover — `config.rs`, `llm.rs`, `catalog.rs`, and `buzzAgentConfig.ts` are unchanged. Phase 2 wires them. **Single source of truth** replaces hand-mirrored metadata across four files: ``` scripts/model-capabilities.json → hand-curated manifest scripts/generate-model-capabilities.mjs → emits Rust + TS artifacts crates/buzz-agent/src/generated_model_capabilities.rs desktop/src/features/agents/ui/modelCapabilities.ts ``` ## Resolver contract (plan v4 §Resolver contract) Total function `resolve(provider, raw_model_id) → CapabilityResult`. Three ordered steps: 1. Provider-qualified raw exact lookup — key is `(provider, raw_model_id)`, matched before any prefix stripping. A prefixed alias never inherits an exact record. 2. Provider-scoped ordered family rules — on normalized (prefix-stripped) alias, by `match_priority` desc. 3. Per-axis provider fallback — `blank` vs `concrete_unknown`, per provider. Result is complete — every axis populated, runtime consumers never compose fields. ## Boundaries the manifest does NOT own - Transport for pure OpenAI, legacy Databricks, OpenRouter: `OpenAiApi`/`openai_request()` remain authoritative. `databricks_v2_wire_route` is DBv2-only (all other providers emit `not-applicable`). - Final display labels: `resolveModelLabel()` three-tier precedence unchanged. `registry_label` feeds only the static registry tier. - `llm.rs` scope: only `databricks_v2_route_for_model` (Phase 2). ## Test oracle (three independent layers) 1. Generated full-table coverage — `scripts/generated-model-capabilities-coverage.json`: every manifest entry + provider fallbacks. 2. Hand-authored normative corpus — `scripts/normative-corpus.json` (44 vectors): Anthropic manual-budget/adaptive families, OpenAI gpt-5 adversarial boundary cases, DBv2 segment-routing collision tests, P2-A resolver-contract vectors, P2-B blank/concrete-unknown per provider. Runs against JS resolver (`run-corpus.mjs`) and mirrored in Rust (`generated_model_capabilities_tests.rs`). 3. Schema-negative tests — `scripts/test-manifest-validator.mjs` (17 tests): every validator rule has a failing-input test. ## Reconciliation table `scripts/MODELS_DEV_RECONCILIATION.md` — all models.dev divergences dispositioned. `databricks-gpt-5-4-mini` and `databricks-gpt-5-4-nano` adopt models.dev `[low,medium,high]` (family rule adds `none+xhigh` the endpoint doesn't advertise). ## CI `.github/workflows/model-capability-regen-diff.yml`: triggers on manifest/generator/artifact changes; regenerates and fails if stale; runs JS corpus + schema-negative tests. ## Acceptance criteria (plan v4 Phase 1) - Byte-clean regen: `node scripts/generate-model-capabilities.mjs --check` passes - Rust compiles: `cargo check -p buzz-agent` - TS typechecks: `pnpm tsc --noEmit --strict` - 44/44 normative corpus vectors pass (JS interpreter) - 41/41 Rust corpus tests pass - 17/17 schema-negative tests pass (every validator rule) - Reconciliation table complete with doc citations - No consumer changes (config.rs, llm.rs, catalog.rs, buzzAgentConfig.ts untouched) --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
104 lines
5.2 KiB
Markdown
104 lines
5.2 KiB
Markdown
# Model Capabilities Manifest — Schema Reference
|
|
|
|
**Source of truth**: `scripts/model-capabilities.json`
|
|
**Generator**: `scripts/generate-model-capabilities.mjs`
|
|
**Emitted artifacts**:
|
|
- `crates/buzz-agent/src/generated_model_capabilities.rs`
|
|
- `desktop/src/features/agents/ui/modelCapabilities.ts`
|
|
- `scripts/generated-model-capabilities-coverage.json` (test fixture)
|
|
|
|
## Resolver contract (plan v4)
|
|
|
|
Resolution is a total function `resolve(provider, raw_model_id) → CapabilityResult`.
|
|
Three ordered steps:
|
|
|
|
1. **Provider-qualified raw exact lookup** — key is `(provider, raw_model_id)`, matched on the
|
|
RAW ID **before any prefix stripping**. A prefixed alias never inherits an exact record.
|
|
2. **Provider-scoped ordered family rules** — on the normalized (prefix-stripped) alias.
|
|
Rules ordered by `match_priority` descending (higher wins). Each rule is tagged with the
|
|
providers it applies to.
|
|
3. **Per-axis provider fallback** — `blank` (empty model string) vs `concrete_unknown`
|
|
(nonblank but unmatched), per provider.
|
|
|
|
`CapabilityResult` is complete — every axis is populated. Runtime consumers never compose fields
|
|
from multiple tiers.
|
|
|
|
## Axes (schema fields)
|
|
|
|
| Axis | Type | Notes |
|
|
|------|------|-------|
|
|
| `registry_label` | `string \| null` | Optional static display label. Feeds `resolveModelLabel()` registry tier only. |
|
|
| `thinking_mode` | enum | `manual-budget \| adaptive \| omit-fields \| none \| not-applicable` |
|
|
| `supported_efforts` | `ThinkingEffort[]` | Non-empty. UI effort dropdown options. |
|
|
| `default_effort` | `ThinkingEffort \| null` | null = "Inherit" (Anthropic manual-budget models). |
|
|
| `databricks_v2_wire_route` | enum | `openai-responses \| anthropic-messages \| mlflow-chat \| route-unknown \| not-applicable` |
|
|
| `normalization_policy` | enum | `none \| openai-standard \| openai-clamp-max-to-xhigh` |
|
|
|
|
### `thinking_mode` values
|
|
|
|
| Value | Meaning |
|
|
|-------|---------|
|
|
| `manual-budget` | `thinking:{type:"enabled", budget_tokens}` — claude-3*, claude-opus-4-5 |
|
|
| `adaptive` | `thinking:{type:"adaptive"}` + `output_config:{effort}` — opus-4-6+, sonnet-4-6+, etc. |
|
|
| `omit-fields` | Unknown Anthropic model — omit thinking fields rather than guess request shape |
|
|
| `none` | Non-Anthropic-routed model — thinking fields not applicable |
|
|
| `not-applicable` | Provider does not use Anthropic thinking API |
|
|
|
|
### `databricks_v2_wire_route` values
|
|
|
|
Scoped to DBv2 only. All non-DBv2 providers emit `not-applicable`.
|
|
Transport for pure OpenAI, legacy Databricks, and OpenRouter is selected by `OpenAiApi` /
|
|
`openai_request()` at runtime.
|
|
|
|
| Value | Meaning |
|
|
|-------|---------|
|
|
| `openai-responses` | `/ai-gateway/openai/v1/responses` |
|
|
| `anthropic-messages` | `/ai-gateway/anthropic/v1/messages` |
|
|
| `mlflow-chat` | `/ai-gateway/mlflow/v1/chat/completions` |
|
|
| `route-unknown` | DBv2 blank model — route not yet determinable |
|
|
| `not-applicable` | Not a DBv2 provider |
|
|
|
|
## Family rule match kinds
|
|
|
|
| Kind | Semantics |
|
|
|------|-----------|
|
|
| `exact` | Case-insensitive exact string equality on normalized alias |
|
|
| `prefix` | Normalized alias starts with match_value |
|
|
| `gpt5-token` | Boundary-aware token: present at end-of-string or followed by `-` (not digit/letter) |
|
|
| `gpt5-base` | Like gpt5-token but also rejects `-<1-3 digit>` suffixes (version-number rejection) |
|
|
| `segment` | Normalized alias contains match_value as a full alphanumeric segment (split on non-alnum) |
|
|
| `segment-prefix` | Any segment of the normalized alias starts with match_value |
|
|
|
|
## Boundaries the manifest does NOT own
|
|
|
|
- **Transport/endpoint selection for pure OpenAI, legacy Databricks, OpenRouter**: `OpenAiApi` and
|
|
`openai_request()` remain authoritative. The `databricks_v2_wire_route` axis is DBv2-only.
|
|
- **Final display labels**: `resolveModelLabel(discovered_name, registry_label, raw_id)` three-tier
|
|
precedence is authoritative. The manifest's `registry_label` feeds only the static registry tier.
|
|
- **`llm.rs` replacement scope**: only `databricks_v2_route_for_model`. Other dispatch paths remain.
|
|
|
|
## Reconciliation policy (plan v4 §Behavior policy)
|
|
|
|
Not purely behavior-preserving. `models.dev` `reasoning_options` become exact overrides. Each
|
|
divergence from family rule results is reconciled against provider docs and either:
|
|
- (a) **adopted** as an intentional correction with its own test + exact record, or
|
|
- (b) **rejected** with a curation note in the exact record.
|
|
|
|
See reconciliation table: `scripts/MODELS_DEV_RECONCILIATION.md`.
|
|
|
|
## Adding a new model family
|
|
|
|
1. Add a `family_rules` entry with a new unique `id`, appropriate `match_kind`, `providers`,
|
|
`match_priority`, and all capability axes.
|
|
2. Run `node scripts/generate-model-capabilities.mjs` to regenerate artifacts.
|
|
3. CI `model-capability-regen-diff` job verifies byte-clean regeneration.
|
|
4. The normative corpus (`scripts/normative-corpus.json`) may need new vectors.
|
|
|
|
## Adding an exact model override
|
|
|
|
1. Add an `exact_records` entry with `provider` + `raw_model_id` (the full raw ID, no prefix
|
|
stripping). Include a `_reconciliation` note and doc citation.
|
|
2. Run `node scripts/generate-model-capabilities.mjs` — completeness validator will fail if any
|
|
axis cannot be resolved.
|
|
3. Regenerate and commit.
|