mirror of
https://github.com/block/buzz.git
synced 2026-11-01 15:01:12 +00:00
refactor(models): Phase 3 — retire old hand tables and shrink manifest apparatus (#4589)
## Summary Retires the old hand-table authorities and transitional verification scaffolding from the model-capability manifest arc. All production routes now run exclusively through the generated interpreters introduced in Phase 1 ([#3821](https://github.com/block/buzz/pull/3821)) and wired in Phase 2 ([#3958](https://github.com/block/buzz/pull/3958)). Stack: [#3821](https://github.com/block/buzz/pull/3821) → [#3958](https://github.com/block/buzz/pull/3958) → this PR Base: [#3603](https://github.com/block/buzz/pull/3603) ## What the manifest system is now **Source of truth:** `scripts/model-capabilities.json` **Generator:** `scripts/generate-model-capabilities.mjs` — emits Rust and TS interpreters only (coverage JSON output removed) **Generated interpreters:** `crates/buzz-agent/src/generated_model_capabilities.rs` (+ normative tests), `desktop/src/features/agents/ui/modelCapabilities.ts` **Label registry:** `generate-databricks-model-names.py` → `databricks_model_names.rs` / `databricksModelNames.ts` **Permanent gates:** `scripts/normative-corpus.json` + `scripts/run-corpus.mjs` (both-interpreter equivalence, 51 vectors), `scripts/test-manifest-validator.mjs` (schema, 24 cases), regen-diff job inside `ci.yml` **One doc:** `scripts/MODEL_CAPABILITIES.md` ## Deleted **Old hand-table authorities (production):** - `getProviderEffortConfig_oldHandTable()` and all supporting helpers from `desktop/src/features/agents/ui/buzzAgentConfig.ts` - `normalize_effort_for_openai_route()`, `_old_anthropic_thinking_config_for_databricks_v2()`, test-only re-export wrappers from `crates/buzz-agent/src/config.rs` - `strip_catalog_prefix()`, `anthropic_thinking_config()`, `anthropic_model_supports_xhigh()`, `clamp_adaptive_effort()`, `anthropic_efforts_for_model()`, `is_manual_budget_model()`, `is_adaptive_thinking_model()`, `gpt5_token_matches()`, `gpt5_base_matches()`, `openai_efforts_for_model()` from `crates/buzz-agent/src/config.rs` — all superseded by generated interpreter - Old DBv2 body-level tests, `_OLD_DATABRICKS_V2_*` constants, `model_name_segments()`, `_old_databricks_v2_route_for_model()`, all Phase-2 behavioral differential test functions from `crates/buzz-agent/src/llm.rs` **Transitional scaffolding:** - `scripts/run-differential.mjs` — old-vs-new JS differential harness - `scripts/run-mutation-evidence.mjs` — one-time mutation evidence runner - `desktop/src/features/agents/ui/effortTable.fixture.json` — Phase-2 TS/Rust sync fixture - `desktop/src/features/agents/ui/effortTable.fixture.test.mjs` — fixture sync guard - `.github/workflows/model-capability-regen-diff.yml` — standalone workflow (steps folded into `ci.yml`) **One-time evidence and generated snapshots:** - `scripts/MUTATION_EVIDENCE.md`, `scripts/MODEL_CAPABILITIES_SCHEMA.md`, `scripts/MODELS_DEV_RECONCILIATION.md` — consolidated into `scripts/MODEL_CAPABILITIES.md` - `scripts/generated-model-capabilities-coverage.json` — full-table snapshot (generator no longer emits it) - `scripts/catalog-sample-fixture.json` — models.dev snapshot used only by the deleted differential harness ## Verification - `cargo test -p buzz-agent --lib` with `RUSTFLAGS="-D warnings"`: **337/337** (clean — no dead_code warnings) - `node --experimental-strip-types scripts/run-corpus.mjs`: **51/51** - `node scripts/generate-model-capabilities.mjs` + regen diff: **clean (exit 0)** - `node --test scripts/test-manifest-validator.mjs`: **24/24** - `just clippy`: zero warnings, zero errors --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
This commit is contained in:
Will Pfleger
44
.github/workflows/ci.yml
vendored
44
.github/workflows/ci.yml
vendored
@@ -130,6 +130,50 @@ jobs:
|
||||
- name: Unit tests
|
||||
run: just test-unit
|
||||
|
||||
model-capabilities:
|
||||
name: Model Capabilities (regen + corpus + schema + buzz-agent tests)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
|
||||
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
|
||||
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
save-if: ${{ github.event_name != 'pull_request' }}
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
||||
with:
|
||||
node-version: '22'
|
||||
package-manager-cache: false
|
||||
|
||||
- name: Regenerate artifacts
|
||||
run: node scripts/generate-model-capabilities.mjs
|
||||
|
||||
- name: Diff check — fail if generated files are stale
|
||||
run: |
|
||||
if ! git diff --exit-code \
|
||||
crates/buzz-agent/src/generated_model_capabilities.rs \
|
||||
desktop/src/features/agents/ui/modelCapabilities.ts; then
|
||||
echo ""
|
||||
echo "ERROR: Generated model-capability files are stale."
|
||||
echo "Run: node scripts/generate-model-capabilities.mjs"
|
||||
echo "Then commit the regenerated files."
|
||||
exit 1
|
||||
fi
|
||||
echo "✓ All generated files are up to date."
|
||||
|
||||
- name: Run corpus (TS interpreter via --experimental-strip-types)
|
||||
run: node --experimental-strip-types scripts/run-corpus.mjs
|
||||
|
||||
- name: Validate manifest (schema-negative tests)
|
||||
run: node --test scripts/test-manifest-validator.mjs
|
||||
|
||||
- name: Run buzz-agent unit tests (normative corpus + generated interpreter)
|
||||
run: cargo test -p buzz-agent --lib
|
||||
|
||||
desktop-core:
|
||||
name: Desktop Core
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
@@ -1,77 +0,0 @@
|
||||
name: Model Capability Regenerate-Then-Diff
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'scripts/model-capabilities.json'
|
||||
- 'scripts/generate-model-capabilities.mjs'
|
||||
- 'crates/buzz-agent/src/generated_model_capabilities.rs'
|
||||
- 'desktop/src/features/agents/ui/modelCapabilities.ts'
|
||||
- 'scripts/generated-model-capabilities-coverage.json'
|
||||
- '.github/workflows/model-capability-regen-diff.yml'
|
||||
# Differential harness and fixtures — any change to old/new side or inputs re-runs.
|
||||
- 'scripts/run-differential.mjs'
|
||||
- 'scripts/normative-corpus.json'
|
||||
- 'scripts/catalog-sample-fixture.json'
|
||||
- 'desktop/src/features/agents/ui/effortTable.fixture.json'
|
||||
- 'desktop/src/features/agents/ui/buzzAgentConfig.ts'
|
||||
- 'crates/buzz-agent/src/config.rs'
|
||||
- 'crates/buzz-agent/src/llm.rs'
|
||||
push:
|
||||
branches: [main, release, 'duncan/databricks-model-label-registry']
|
||||
|
||||
jobs:
|
||||
regen-diff:
|
||||
name: Regenerate and diff model capability artifacts
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
||||
with:
|
||||
node-version: '22'
|
||||
package-manager-cache: false
|
||||
|
||||
- name: Regenerate artifacts
|
||||
run: node scripts/generate-model-capabilities.mjs
|
||||
|
||||
- name: Diff check — fail if generated files are stale
|
||||
run: |
|
||||
if ! git diff --exit-code \
|
||||
crates/buzz-agent/src/generated_model_capabilities.rs \
|
||||
desktop/src/features/agents/ui/modelCapabilities.ts \
|
||||
scripts/generated-model-capabilities-coverage.json; then
|
||||
echo ""
|
||||
echo "ERROR: Generated model-capability files are stale."
|
||||
echo "Run: node scripts/generate-model-capabilities.mjs"
|
||||
echo "Then commit the regenerated files."
|
||||
exit 1
|
||||
fi
|
||||
echo "✓ All generated files are up to date."
|
||||
|
||||
- name: Run corpus (TS interpreter via --experimental-strip-types)
|
||||
run: node --experimental-strip-types scripts/run-corpus.mjs
|
||||
|
||||
- name: Validate manifest (schema-negative tests)
|
||||
run: node --test scripts/test-manifest-validator.mjs
|
||||
|
||||
- name: Run differential harness (old vs new, all input sets)
|
||||
run: node --experimental-strip-types scripts/run-differential.mjs
|
||||
|
||||
rust-unit-tests:
|
||||
name: buzz-agent unit tests (normative corpus + behavioral differential)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
|
||||
|
||||
- name: Run buzz-agent unit tests
|
||||
run: cargo test -p buzz-agent --lib
|
||||
Reference in New Issue
Block a user