mirror of
https://github.com/block/buzz.git
synced 2026-11-01 15:01:12 +00:00
c2bed46ee4
Production cutover (Thufir finding 1, Rust half): - Add normalize_effort_for_databricks_v2() and anthropic_thinking_config_for_databricks_v2() as generated-backed production helpers for DatabricksV2 requests. - llm.rs DatabricksV2 dispatch now routes through these helpers: OpenAiResponses and MlflowChatCompletions paths call normalize_effort_for_databricks_v2(); anthropic_body calls anthropic_thinking_config_for_databricks_v2(). The adopted F1 corrections (e.g. databricks-gpt-5-5: [low,medium,high]) are now enforced in real requests. - Pure OpenAi/Databricks path retains normalize_effort_for_openai_route (unchanged). Comprehensive Rust differential (Thufir finding 4): - Replace the simple route-only differential test (20-vector literal list, no allowlist scoping) with a comprehensive multi-axis test covering all 3 input sets: effortTable.fixture.json (36), normative-corpus.json (45), catalog-sample-fixture.json (14). - Axes: supported_efforts, default_effort, databricks_v2_wire_route, thinking_mode. - Allowlist is axis-scoped; 7 entries, all doc-cited. Stale entries FAIL (mirrors JS harness semantics — no whole-record skip). - Exposes strip_catalog_prefix as pub(crate); adds #[cfg(test)] pub(crate) helpers is_manual_budget_model_for_test, is_adaptive_thinking_model_for_test, and valid_effort_values_for_provider_model_for_test so llm.rs tests can call config.rs classifiers without duplicating them. - Result: 95 input entries, 7/7 allowlist slots exercised, 0 unexpected divergences. CI gate (Thufir finding 3): - Extend model-capability-regen-diff workflow path filter to cover the differential harness, all three fixtures, and both old/new consumer files (buzzAgentConfig.ts, config.rs, llm.rs) so any change to either side triggers the gate. - Add a final step: node --experimental-strip-types scripts/run-differential.mjs. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
65 lines
2.4 KiB
YAML
65 lines
2.4 KiB
YAML
name: Model Capability Regenerate-Then-Diff
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- 'scripts/model-capabilities.json'
|
|
- 'scripts/generate-model-capabilities.mjs'
|
|
- 'crates/buzz-agent/src/generated_model_capabilities.rs'
|
|
- 'desktop/src/features/agents/ui/modelCapabilities.ts'
|
|
- 'scripts/generated-model-capabilities-coverage.json'
|
|
- '.github/workflows/model-capability-regen-diff.yml'
|
|
# Differential harness and fixtures — any change to old/new side or inputs re-runs.
|
|
- 'scripts/run-differential.mjs'
|
|
- 'scripts/normative-corpus.json'
|
|
- 'scripts/catalog-sample-fixture.json'
|
|
- 'desktop/src/features/agents/ui/effortTable.fixture.json'
|
|
- 'desktop/src/features/agents/ui/buzzAgentConfig.ts'
|
|
- 'crates/buzz-agent/src/config.rs'
|
|
- 'crates/buzz-agent/src/llm.rs'
|
|
push:
|
|
branches: [main, release, 'duncan/databricks-model-label-registry']
|
|
|
|
jobs:
|
|
regen-diff:
|
|
name: Regenerate and diff model capability artifacts
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 5
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
|
|
|
|
- name: Set up Node.js
|
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
|
with:
|
|
node-version: '22'
|
|
package-manager-cache: false
|
|
|
|
- name: Regenerate artifacts
|
|
run: node scripts/generate-model-capabilities.mjs
|
|
|
|
- name: Diff check — fail if generated files are stale
|
|
run: |
|
|
if ! git diff --exit-code \
|
|
crates/buzz-agent/src/generated_model_capabilities.rs \
|
|
desktop/src/features/agents/ui/modelCapabilities.ts \
|
|
scripts/generated-model-capabilities-coverage.json; then
|
|
echo ""
|
|
echo "ERROR: Generated model-capability files are stale."
|
|
echo "Run: node scripts/generate-model-capabilities.mjs"
|
|
echo "Then commit the regenerated files."
|
|
exit 1
|
|
fi
|
|
echo "✓ All generated files are up to date."
|
|
|
|
- name: Run corpus (TS interpreter via --experimental-strip-types)
|
|
run: node --experimental-strip-types scripts/run-corpus.mjs
|
|
|
|
- name: Validate manifest (schema-negative tests)
|
|
run: node --test scripts/test-manifest-validator.mjs
|
|
|
|
- name: Run differential harness (old vs new, all input sets)
|
|
run: node --experimental-strip-types scripts/run-differential.mjs
|