mirror of
https://github.com/block/buzz.git
synced 2026-11-09 02:41:57 +00:00
d3a1520215c09d3ebe5dce65c9552ea100ac88df
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7
|
60fc24b0c6 |
fix(models): round-3 corrective pass — boundary strip, gpt-version-segment, 6-axis corpus, validation symmetry, prototype-leak
- stripCatalogPrefix boundary-aligned: customgpt/sgpt/mygpt no longer false-strip to gpt-* forms; boundary check requires position 0 or non-alphanumeric precursor - New match_kind gpt-version-segment: gpt-neox-20b routes mlflow-chat (was residual collision); gpt segment match requires next segment to start with digit or be dashless gpt5 form - PROVIDER_FALLBACKS Record→Map: constructor/__proto__ prototype-key leak closed; resolveModelCapabilities/getProviderEffortConfig return complete records for all provider strings - Corpus 69→77 vectors, all 6-axis mandatory; JS and Rust runners hard-fail on missing axes; prototype-key vectors added; boundary-negative vectors added (sgpt-5-5, mygpt-5, customgpt-5-5-endpoint all route mlflow-chat); gpt-neox-20b pinned as mlflow-chat (corrected behavior, not residual collision) - Manifest validator 33→42 tests: family match_priority integer guard, lowercase duplicate-key detection, post-inheritance materialized default∈supported check, assertEfforts shared helper covering family/fallback/exact - Clippy: map_or(false)→is_some_and in emitter template; fmt clean - Desktop regression tests: constructor/__proto__ prototype-key stability Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> |
||
|
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7
|
8ba3814854 |
fix(models): address Kalvin-review: gpt-segment/left-boundary/luna-terra/validator
CRITICAL: Fix Rust gpt5-base short-version guard to mirror TS regex semantics.
The old 4-char window check diverged from TS /^\d{1,3}(?:[^a-z\d]|$)/i for
inputs like gpt-5-10-preview (10 digits). Replaced with find(non-digit) approach
that exactly matches the TS digit-run semantics. Add left-boundary guards (start
or preceded by '-'/'.') to both gpt5_token_matches_rs and gpt5_base_matches_rs
to prevent sgpt-model-class false-positives.
IMPORTANT: Add exact_records validation to generator — enum checks for all
optional override axes, non-empty override check, canonical-order enforcement,
default_effort-in-override check, match_priority non-negative-int check
(injected into source comments, treated as injection surface). Add 9 schema-
negative tests in test-manifest-validator.mjs.
IMPORTANT: Restrict DBv2 gpt segment rule from starts_with("gpt") to exact
segment match ("gpt" or "gpt5"). Raise priority 5->6 to restore old dual-marker
OpenAI-before-Claude contract. Add left-boundary guards to token helpers. Add
collision-negative corpus vectors: gptoss-model, gptj-6b, gpt-neox,
customgpt-5-5-endpoint.
MINOR: Restore .trim() on model string before resolveModelCapabilities call in
buzzAgentConfig.ts. Make exact-record lookup case-insensitive (lowercase keys at
build + lookup). Extend run-corpus.mjs to compare all 6 axes including
normalization_policy and registry_label. Add positive corpus vectors: opus-5
rule, dbv2 gpt-segment, sol/luna/terra, gpt-5-4-nano exact record, openrouter/
unknown/legacy-databricks fallbacks, case-insensitive lookup vectors. Fix Rust
corpus test harness provider lowercasing to handle vectors like provider=OpenAI.
HYGIENE: Fix "Generated 3 files" -> "Generated 2 files" message. Remove dead
$schema pointer from manifest. Add module doc to generated_model_capabilities_tests.rs
noting hand-maintained status. Switch PROVIDER_ALIASES from plain object to Map
in formatAgentModelLabel.ts and run-corpus.mjs. Align CI node-version to 24.
Tighten python SAFE_NAME_RE from * to + to reject empty display names.
luna/terra: models.dev confirms databricks-gpt-5-6-luna and -terra advertise
[low,medium,high] (differs from sol's [low,medium,high,max]). Added two exact
records with supported_efforts=[low,medium,high], default_effort=medium.
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
|
||
|
Will Pfleger
|
899f55cc8d |
refactor(models): Phase 3 — retire old hand tables and shrink manifest apparatus (#4589)
## Summary Retires the old hand-table authorities and transitional verification scaffolding from the model-capability manifest arc. All production routes now run exclusively through the generated interpreters introduced in Phase 1 ([#3821](https://github.com/block/buzz/pull/3821)) and wired in Phase 2 ([#3958](https://github.com/block/buzz/pull/3958)). Stack: [#3821](https://github.com/block/buzz/pull/3821) → [#3958](https://github.com/block/buzz/pull/3958) → this PR Base: [#3603](https://github.com/block/buzz/pull/3603) ## What the manifest system is now **Source of truth:** `scripts/model-capabilities.json` **Generator:** `scripts/generate-model-capabilities.mjs` — emits Rust and TS interpreters only (coverage JSON output removed) **Generated interpreters:** `crates/buzz-agent/src/generated_model_capabilities.rs` (+ normative tests), `desktop/src/features/agents/ui/modelCapabilities.ts` **Label registry:** `generate-databricks-model-names.py` → `databricks_model_names.rs` / `databricksModelNames.ts` **Permanent gates:** `scripts/normative-corpus.json` + `scripts/run-corpus.mjs` (both-interpreter equivalence, 51 vectors), `scripts/test-manifest-validator.mjs` (schema, 24 cases), regen-diff job inside `ci.yml` **One doc:** `scripts/MODEL_CAPABILITIES.md` ## Deleted **Old hand-table authorities (production):** - `getProviderEffortConfig_oldHandTable()` and all supporting helpers from `desktop/src/features/agents/ui/buzzAgentConfig.ts` - `normalize_effort_for_openai_route()`, `_old_anthropic_thinking_config_for_databricks_v2()`, test-only re-export wrappers from `crates/buzz-agent/src/config.rs` - `strip_catalog_prefix()`, `anthropic_thinking_config()`, `anthropic_model_supports_xhigh()`, `clamp_adaptive_effort()`, `anthropic_efforts_for_model()`, `is_manual_budget_model()`, `is_adaptive_thinking_model()`, `gpt5_token_matches()`, `gpt5_base_matches()`, `openai_efforts_for_model()` from `crates/buzz-agent/src/config.rs` — all superseded by generated interpreter - Old DBv2 body-level tests, `_OLD_DATABRICKS_V2_*` constants, `model_name_segments()`, `_old_databricks_v2_route_for_model()`, all Phase-2 behavioral differential test functions from `crates/buzz-agent/src/llm.rs` **Transitional scaffolding:** - `scripts/run-differential.mjs` — old-vs-new JS differential harness - `scripts/run-mutation-evidence.mjs` — one-time mutation evidence runner - `desktop/src/features/agents/ui/effortTable.fixture.json` — Phase-2 TS/Rust sync fixture - `desktop/src/features/agents/ui/effortTable.fixture.test.mjs` — fixture sync guard - `.github/workflows/model-capability-regen-diff.yml` — standalone workflow (steps folded into `ci.yml`) **One-time evidence and generated snapshots:** - `scripts/MUTATION_EVIDENCE.md`, `scripts/MODEL_CAPABILITIES_SCHEMA.md`, `scripts/MODELS_DEV_RECONCILIATION.md` — consolidated into `scripts/MODEL_CAPABILITIES.md` - `scripts/generated-model-capabilities-coverage.json` — full-table snapshot (generator no longer emits it) - `scripts/catalog-sample-fixture.json` — models.dev snapshot used only by the deleted differential harness ## Verification - `cargo test -p buzz-agent --lib` with `RUSTFLAGS="-D warnings"`: **337/337** (clean — no dead_code warnings) - `node --experimental-strip-types scripts/run-corpus.mjs`: **51/51** - `node scripts/generate-model-capabilities.mjs` + regen diff: **clean (exit 0)** - `node --test scripts/test-manifest-validator.mjs`: **24/24** - `just clippy`: zero warnings, zero errors --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
Will Pfleger
|
4d47f48143 |
feat(agent): Phase 1 — model-capability manifest, generator, and test oracle (#3821)
## What this does Introduces the model-capability manifest infrastructure (Phase 1 of the Model-Capability Manifest plan v4, Thufir-approved 9/9/9). No consumer cutover — `config.rs`, `llm.rs`, `catalog.rs`, and `buzzAgentConfig.ts` are unchanged. Phase 2 wires them. **Single source of truth** replaces hand-mirrored metadata across four files: ``` scripts/model-capabilities.json → hand-curated manifest scripts/generate-model-capabilities.mjs → emits Rust + TS artifacts crates/buzz-agent/src/generated_model_capabilities.rs desktop/src/features/agents/ui/modelCapabilities.ts ``` ## Resolver contract (plan v4 §Resolver contract) Total function `resolve(provider, raw_model_id) → CapabilityResult`. Three ordered steps: 1. Provider-qualified raw exact lookup — key is `(provider, raw_model_id)`, matched before any prefix stripping. A prefixed alias never inherits an exact record. 2. Provider-scoped ordered family rules — on normalized (prefix-stripped) alias, by `match_priority` desc. 3. Per-axis provider fallback — `blank` vs `concrete_unknown`, per provider. Result is complete — every axis populated, runtime consumers never compose fields. ## Boundaries the manifest does NOT own - Transport for pure OpenAI, legacy Databricks, OpenRouter: `OpenAiApi`/`openai_request()` remain authoritative. `databricks_v2_wire_route` is DBv2-only (all other providers emit `not-applicable`). - Final display labels: `resolveModelLabel()` three-tier precedence unchanged. `registry_label` feeds only the static registry tier. - `llm.rs` scope: only `databricks_v2_route_for_model` (Phase 2). ## Test oracle (three independent layers) 1. Generated full-table coverage — `scripts/generated-model-capabilities-coverage.json`: every manifest entry + provider fallbacks. 2. Hand-authored normative corpus — `scripts/normative-corpus.json` (44 vectors): Anthropic manual-budget/adaptive families, OpenAI gpt-5 adversarial boundary cases, DBv2 segment-routing collision tests, P2-A resolver-contract vectors, P2-B blank/concrete-unknown per provider. Runs against JS resolver (`run-corpus.mjs`) and mirrored in Rust (`generated_model_capabilities_tests.rs`). 3. Schema-negative tests — `scripts/test-manifest-validator.mjs` (17 tests): every validator rule has a failing-input test. ## Reconciliation table `scripts/MODELS_DEV_RECONCILIATION.md` — all models.dev divergences dispositioned. `databricks-gpt-5-4-mini` and `databricks-gpt-5-4-nano` adopt models.dev `[low,medium,high]` (family rule adds `none+xhigh` the endpoint doesn't advertise). ## CI `.github/workflows/model-capability-regen-diff.yml`: triggers on manifest/generator/artifact changes; regenerates and fails if stale; runs JS corpus + schema-negative tests. ## Acceptance criteria (plan v4 Phase 1) - Byte-clean regen: `node scripts/generate-model-capabilities.mjs --check` passes - Rust compiles: `cargo check -p buzz-agent` - TS typechecks: `pnpm tsc --noEmit --strict` - 44/44 normative corpus vectors pass (JS interpreter) - 41/41 Rust corpus tests pass - 17/17 schema-negative tests pass (every validator rule) - Reconciliation table complete with doc citations - No consumer changes (config.rs, llm.rs, catalog.rs, buzzAgentConfig.ts untouched) --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |