mirror of
https://github.com/block/buzz.git
synced 2026-10-31 22:41:12 +00:00
cc00060ea3
## What Phase 2 consumer cutover targeting the `duncan/databricks-model-label-registry` umbrella branch. Wires `crates/**` and `desktop/**` consumers to the generated capability module introduced in Phase 1 (#3821), while keeping old and new paths both live for differential testing. Phase 3 removes the old paths. ## Commits (boundary-separated) ### feat(agent): Phase 2a — wire Rust consumers to generated capability module (`crates/**`, `scripts/**`) - `catalog.rs`: `DATABRICKS_V2_KNOWN_MODELS` re-exported from the generated module — single source of truth. - `llm.rs`: `databricks_v2_route_for_model` delegates to `resolve_model_capabilities("databricks_v2", model)`. Old segment-based classifier preserved as `#[cfg(test)] _old_*` for the differential harness. New `databricks_v2_route_differential_old_vs_new` test confirms 100% agreement on all 20 route vectors. - `config.rs`: new `effort_table_fixture_differential_old_vs_new` test runs `resolve_model_capabilities` over the 36-entry `effortTable.fixture.json` and asserts old/new agree modulo a doc-cited allowlist (4 F1 corrections). - `scripts/run-differential.mjs`: JS differential harness over effortTable fixture + normative corpus + catalog-sample fixture. 85 checks, 0 unexpected divergences (5 allowlisted: 4 F1 corrections + goose-opus-5 anthropic route correction). - `scripts/MODELS_DEV_RECONCILIATION.md`: deferred MINOR from Phase 1 — 8 trailing-double-space line breaks replaced with `<br>`. ### feat(desktop): Phase 2b — cut TS consumers to generated model-capabilities module (`desktop/**`) - `buzzAgentConfig.ts`: adds `getProviderEffortConfigFromManifest(provider, model?)` — thin wrapper over `resolveModelCapabilities()` from `modelCapabilities.ts`. Maps `supportedEfforts → validValues` and `defaultEffort → defaultValue` (null preserved for manual-budget/Inherit). Old `getProviderEffortConfig()` and all hand-tables stay live for the differential harness; Phase 3 retires them. - `formatAgentModelLabel.ts`: registry-label lookup re-pointed from hand-maintained `databricksModelNames.ts` import to generated `DATABRICKS_MODEL_NAMES` exported from `modelCapabilities.ts`. Same Map shape, identical contents, behavior unchanged. ### fix(scripts): add ts-esm-loader and fix allowlist coverage in run-differential (`scripts/**`) - `scripts/ts-esm-loader.mjs`: minimal ESM custom loader that resolves extensionless relative TS imports. Required because Phase 2b's `buzzAgentConfig.ts` imports `modelCapabilities` without `.ts` extension — which Node's `--experimental-strip-types` runner cannot resolve without a hook. - `scripts/run-differential.mjs`: shebang updated to self-bootstrap with the loader; fixes the `totalAllowlisted` counter (was declared but never incremented — always printed `0 allowlisted`). Replaced with per-axis hit tracking: reports exercised slot count (`N/total`) in summary; fails with `STALE_ALLOWLIST` if any declared entry fires zero divergences, preventing stale entries from silently masking future regressions. ## Verification - `cargo test -p buzz-agent --lib`: 426/426 - Corpus: 45/45 · schema-negative: 24/24 · `--check` byte-clean - Differential: 85 checks, 0 unexpected divergences, 6/6 allowlist slots exercised - Desktop: 3847/3847 · typecheck clean · biome clean - Mobile: 1019 pass, 1 skipped — same 5 flaky tests in `mobile/test/features/channels/` that reproduce at the umbrella base; zero mobile files in this branch range - `git diff --check`: clean ## What Remains (Phase 3) Remove old hand-maintained paths: `_old_*` functions in `llm.rs`/`config.rs`, old `getProviderEffortConfig` tables in `buzzAgentConfig.ts`, old `databricksModelNames.ts` import in `formatAgentModelLabel.ts`, old `databricks_model_names.rs` module. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw <41ea58f1e64c243627e8acde7c89be667052ee6e17d8f021c1195be4324ebf04@buzz.block.builderlab.xyz> Signed-off-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw <41ea58f1e64c243627e8acde7c89be667052ee6e17d8f021c1195be4324ebf04@buzz.block.builderlab.xyz>
78 lines
2.8 KiB
YAML
78 lines
2.8 KiB
YAML
name: Model Capability Regenerate-Then-Diff
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- 'scripts/model-capabilities.json'
|
|
- 'scripts/generate-model-capabilities.mjs'
|
|
- 'crates/buzz-agent/src/generated_model_capabilities.rs'
|
|
- 'desktop/src/features/agents/ui/modelCapabilities.ts'
|
|
- 'scripts/generated-model-capabilities-coverage.json'
|
|
- '.github/workflows/model-capability-regen-diff.yml'
|
|
# Differential harness and fixtures — any change to old/new side or inputs re-runs.
|
|
- 'scripts/run-differential.mjs'
|
|
- 'scripts/normative-corpus.json'
|
|
- 'scripts/catalog-sample-fixture.json'
|
|
- 'desktop/src/features/agents/ui/effortTable.fixture.json'
|
|
- 'desktop/src/features/agents/ui/buzzAgentConfig.ts'
|
|
- 'crates/buzz-agent/src/config.rs'
|
|
- 'crates/buzz-agent/src/llm.rs'
|
|
push:
|
|
branches: [main, release, 'duncan/databricks-model-label-registry']
|
|
|
|
jobs:
|
|
regen-diff:
|
|
name: Regenerate and diff model capability artifacts
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 5
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
|
|
|
|
- name: Set up Node.js
|
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
|
with:
|
|
node-version: '22'
|
|
package-manager-cache: false
|
|
|
|
- name: Regenerate artifacts
|
|
run: node scripts/generate-model-capabilities.mjs
|
|
|
|
- name: Diff check — fail if generated files are stale
|
|
run: |
|
|
if ! git diff --exit-code \
|
|
crates/buzz-agent/src/generated_model_capabilities.rs \
|
|
desktop/src/features/agents/ui/modelCapabilities.ts \
|
|
scripts/generated-model-capabilities-coverage.json; then
|
|
echo ""
|
|
echo "ERROR: Generated model-capability files are stale."
|
|
echo "Run: node scripts/generate-model-capabilities.mjs"
|
|
echo "Then commit the regenerated files."
|
|
exit 1
|
|
fi
|
|
echo "✓ All generated files are up to date."
|
|
|
|
- name: Run corpus (TS interpreter via --experimental-strip-types)
|
|
run: node --experimental-strip-types scripts/run-corpus.mjs
|
|
|
|
- name: Validate manifest (schema-negative tests)
|
|
run: node --test scripts/test-manifest-validator.mjs
|
|
|
|
- name: Run differential harness (old vs new, all input sets)
|
|
run: node --experimental-strip-types scripts/run-differential.mjs
|
|
|
|
rust-unit-tests:
|
|
name: buzz-agent unit tests (normative corpus + behavioral differential)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
|
|
|
|
- name: Run buzz-agent unit tests
|
|
run: cargo test -p buzz-agent --lib
|