mirror of
https://github.com/block/buzz.git
synced 2026-11-01 23:11:58 +00:00
## What this does Introduces the model-capability manifest infrastructure (Phase 1 of the Model-Capability Manifest plan v4, Thufir-approved 9/9/9). No consumer cutover — `config.rs`, `llm.rs`, `catalog.rs`, and `buzzAgentConfig.ts` are unchanged. Phase 2 wires them. **Single source of truth** replaces hand-mirrored metadata across four files: ``` scripts/model-capabilities.json → hand-curated manifest scripts/generate-model-capabilities.mjs → emits Rust + TS artifacts crates/buzz-agent/src/generated_model_capabilities.rs desktop/src/features/agents/ui/modelCapabilities.ts ``` ## Resolver contract (plan v4 §Resolver contract) Total function `resolve(provider, raw_model_id) → CapabilityResult`. Three ordered steps: 1. Provider-qualified raw exact lookup — key is `(provider, raw_model_id)`, matched before any prefix stripping. A prefixed alias never inherits an exact record. 2. Provider-scoped ordered family rules — on normalized (prefix-stripped) alias, by `match_priority` desc. 3. Per-axis provider fallback — `blank` vs `concrete_unknown`, per provider. Result is complete — every axis populated, runtime consumers never compose fields. ## Boundaries the manifest does NOT own - Transport for pure OpenAI, legacy Databricks, OpenRouter: `OpenAiApi`/`openai_request()` remain authoritative. `databricks_v2_wire_route` is DBv2-only (all other providers emit `not-applicable`). - Final display labels: `resolveModelLabel()` three-tier precedence unchanged. `registry_label` feeds only the static registry tier. - `llm.rs` scope: only `databricks_v2_route_for_model` (Phase 2). ## Test oracle (three independent layers) 1. Generated full-table coverage — `scripts/generated-model-capabilities-coverage.json`: every manifest entry + provider fallbacks. 2. Hand-authored normative corpus — `scripts/normative-corpus.json` (44 vectors): Anthropic manual-budget/adaptive families, OpenAI gpt-5 adversarial boundary cases, DBv2 segment-routing collision tests, P2-A resolver-contract vectors, P2-B blank/concrete-unknown per provider. Runs against JS resolver (`run-corpus.mjs`) and mirrored in Rust (`generated_model_capabilities_tests.rs`). 3. Schema-negative tests — `scripts/test-manifest-validator.mjs` (17 tests): every validator rule has a failing-input test. ## Reconciliation table `scripts/MODELS_DEV_RECONCILIATION.md` — all models.dev divergences dispositioned. `databricks-gpt-5-4-mini` and `databricks-gpt-5-4-nano` adopt models.dev `[low,medium,high]` (family rule adds `none+xhigh` the endpoint doesn't advertise). ## CI `.github/workflows/model-capability-regen-diff.yml`: triggers on manifest/generator/artifact changes; regenerates and fails if stale; runs JS corpus + schema-negative tests. ## Acceptance criteria (plan v4 Phase 1) - Byte-clean regen: `node scripts/generate-model-capabilities.mjs --check` passes - Rust compiles: `cargo check -p buzz-agent` - TS typechecks: `pnpm tsc --noEmit --strict` - 44/44 normative corpus vectors pass (JS interpreter) - 41/41 Rust corpus tests pass - 17/17 schema-negative tests pass (every validator rule) - Reconciliation table complete with doc citations - No consumer changes (config.rs, llm.rs, catalog.rs, buzzAgentConfig.ts untouched) --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
740 lines
28 KiB
Rust
740 lines
28 KiB
Rust
//! Tests for generated model capabilities — normative corpus + handwritten supplements.
|
|
//!
|
|
//! This module is conditionally compiled as #[cfg(test)] from generated_model_capabilities.rs.
|
|
//!
|
|
//! Three layers:
|
|
//! 1. Shared normative corpus executed against the Rust interpreter — every vector in
|
|
//! `scripts/normative-corpus.json` is deserialized and asserted here. This is the
|
|
//! cross-interpreter conformance gate; the JS runner executes the same file.
|
|
//! 2. Handwritten supplement tests — adversarial cases and completeness checks that
|
|
//! benefit from Rust-specific assertion ergonomics.
|
|
//! 3. Per-interpreter mutation evidence — see scripts/MUTATION_EVIDENCE.md.
|
|
|
|
#[cfg(test)]
|
|
mod shared_corpus_tests {
|
|
//! Executes every test vector in `scripts/normative-corpus.json` against the
|
|
//! Rust `resolve_model_capabilities` interpreter.
|
|
//!
|
|
//! The corpus JSON is located at `../../scripts/normative-corpus.json` relative
|
|
//! to the crate root (i.e., `<repo>/scripts/normative-corpus.json`). In the test
|
|
//! binary, `CARGO_MANIFEST_DIR` points at the crate directory.
|
|
|
|
use crate::config::ThinkingEffort;
|
|
use crate::generated_model_capabilities::{
|
|
resolve_model_capabilities, DatabricksV2Route, ThinkingMode,
|
|
};
|
|
use serde::Deserialize;
|
|
use std::path::Path;
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Corpus schema (mirrors normative-corpus.json structure)
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[derive(Deserialize)]
|
|
struct CorpusEntry {
|
|
id: Option<String>,
|
|
provider: Option<String>,
|
|
raw_model_id: Option<String>,
|
|
expect: Option<CorpusExpect>,
|
|
}
|
|
|
|
#[derive(Deserialize)]
|
|
struct CorpusExpect {
|
|
thinking_mode: Option<String>,
|
|
supported_efforts: Option<Vec<String>>,
|
|
default_effort: Option<serde_json::Value>, // string or null
|
|
databricks_v2_wire_route: Option<String>,
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Conversion helpers
|
|
// ---------------------------------------------------------------------------
|
|
|
|
fn parse_thinking_mode(s: &str) -> ThinkingMode {
|
|
match s {
|
|
"manual-budget" => ThinkingMode::ManualBudget,
|
|
"adaptive" => ThinkingMode::Adaptive,
|
|
"omit-fields" => ThinkingMode::OmitFields,
|
|
"none" => ThinkingMode::None,
|
|
"not-applicable" => ThinkingMode::NotApplicable,
|
|
other => panic!("unknown thinking_mode in corpus: {other}"),
|
|
}
|
|
}
|
|
|
|
fn parse_effort(s: &str) -> ThinkingEffort {
|
|
match s {
|
|
"minimal" => ThinkingEffort::Minimal,
|
|
"none" => ThinkingEffort::None,
|
|
"low" => ThinkingEffort::Low,
|
|
"medium" => ThinkingEffort::Medium,
|
|
"high" => ThinkingEffort::High,
|
|
"xhigh" => ThinkingEffort::XHigh,
|
|
"max" => ThinkingEffort::Max,
|
|
other => panic!("unknown effort in corpus: {other}"),
|
|
}
|
|
}
|
|
|
|
fn parse_route(s: &str) -> DatabricksV2Route {
|
|
match s {
|
|
"openai-responses" => DatabricksV2Route::OpenAiResponses,
|
|
"anthropic-messages" => DatabricksV2Route::AnthropicMessages,
|
|
"mlflow-chat" => DatabricksV2Route::MlflowChatCompletions,
|
|
"route-unknown" => DatabricksV2Route::RouteUnknown,
|
|
"not-applicable" => DatabricksV2Route::NotApplicable,
|
|
other => panic!("unknown databricks_v2_wire_route in corpus: {other}"),
|
|
}
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Corpus loader
|
|
// ---------------------------------------------------------------------------
|
|
|
|
fn load_corpus() -> Vec<CorpusEntry> {
|
|
let manifest_dir = env!("CARGO_MANIFEST_DIR");
|
|
let corpus_path = Path::new(manifest_dir)
|
|
.join("..") // crates/buzz-agent → crates
|
|
.join("..") // crates → repo root
|
|
.join("scripts")
|
|
.join("normative-corpus.json");
|
|
let corpus_str = std::fs::read_to_string(&corpus_path).unwrap_or_else(|e| {
|
|
panic!(
|
|
"failed to read normative-corpus.json at {}: {e}",
|
|
corpus_path.display()
|
|
)
|
|
});
|
|
serde_json::from_str(&corpus_str)
|
|
.unwrap_or_else(|e| panic!("failed to parse normative-corpus.json: {e}"))
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Corpus runner
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_shared_corpus_all_vectors() {
|
|
let entries = load_corpus();
|
|
let mut ran = 0;
|
|
let mut failures: Vec<String> = vec![];
|
|
|
|
for entry in &entries {
|
|
// Skip group-marker objects (no id field)
|
|
let id = match &entry.id {
|
|
Some(id) => id.clone(),
|
|
None => continue,
|
|
};
|
|
let provider = entry
|
|
.provider
|
|
.as_deref()
|
|
.unwrap_or_else(|| panic!("corpus vector {id} missing provider"));
|
|
let raw_model_id = entry
|
|
.raw_model_id
|
|
.as_deref()
|
|
.unwrap_or_else(|| panic!("corpus vector {id} missing raw_model_id"));
|
|
let expect = entry
|
|
.expect
|
|
.as_ref()
|
|
.unwrap_or_else(|| panic!("corpus vector {id} missing expect"));
|
|
|
|
let result = resolve_model_capabilities(provider, raw_model_id);
|
|
ran += 1;
|
|
|
|
// Check thinking_mode if present in expect
|
|
if let Some(expected_mode) = &expect.thinking_mode {
|
|
let expected = parse_thinking_mode(expected_mode);
|
|
if result.thinking_mode != expected {
|
|
failures.push(format!(
|
|
"[{id}] thinking_mode: got {:?}, expected {:?}",
|
|
result.thinking_mode, expected
|
|
));
|
|
}
|
|
}
|
|
|
|
// Check supported_efforts if present
|
|
if let Some(expected_efforts) = &expect.supported_efforts {
|
|
let expected: Vec<ThinkingEffort> =
|
|
expected_efforts.iter().map(|s| parse_effort(s)).collect();
|
|
let actual: Vec<ThinkingEffort> =
|
|
result.supported_efforts.iter().cloned().collect();
|
|
if actual != expected {
|
|
failures.push(format!(
|
|
"[{id}] supported_efforts: got {actual:?}, expected {expected:?}"
|
|
));
|
|
}
|
|
}
|
|
|
|
// Check default_effort if present
|
|
if let Some(expected_de) = &expect.default_effort {
|
|
let expected_parsed = match expected_de {
|
|
serde_json::Value::Null => None,
|
|
serde_json::Value::String(s) => Some(parse_effort(s)),
|
|
other => {
|
|
panic!("unexpected default_effort value in corpus vector {id}: {other:?}")
|
|
}
|
|
};
|
|
if result.default_effort != expected_parsed {
|
|
failures.push(format!(
|
|
"[{id}] default_effort: got {:?}, expected {:?}",
|
|
result.default_effort, expected_parsed
|
|
));
|
|
}
|
|
}
|
|
|
|
// Check databricks_v2_wire_route if present
|
|
if let Some(expected_route) = &expect.databricks_v2_wire_route {
|
|
let expected = parse_route(expected_route);
|
|
if result.databricks_v2_wire_route != expected {
|
|
failures.push(format!(
|
|
"[{id}] databricks_v2_wire_route: got {:?}, expected {:?}",
|
|
result.databricks_v2_wire_route, expected
|
|
));
|
|
}
|
|
}
|
|
}
|
|
|
|
if !failures.is_empty() {
|
|
panic!(
|
|
"{}/{} corpus vectors FAILED:\n{}",
|
|
failures.len(),
|
|
ran,
|
|
failures.join("\n")
|
|
);
|
|
}
|
|
assert!(
|
|
ran > 0,
|
|
"corpus was empty — check normative-corpus.json path"
|
|
);
|
|
println!("shared corpus: {ran} vectors all passed");
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod corpus_tests {
|
|
use crate::config::ThinkingEffort;
|
|
use crate::generated_model_capabilities::{
|
|
resolve_model_capabilities, DatabricksV2Route, ThinkingMode,
|
|
};
|
|
|
|
fn resolve(
|
|
provider: &str,
|
|
model: &str,
|
|
) -> crate::generated_model_capabilities::CapabilityResult {
|
|
resolve_model_capabilities(provider, model)
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Anthropic manual-budget family
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_anthropic_claude3_family_manual_budget() {
|
|
let r = resolve("anthropic", "claude-3-7-sonnet-20250219");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::ManualBudget);
|
|
assert_eq!(
|
|
r.supported_efforts.as_ref(),
|
|
&[
|
|
ThinkingEffort::Low,
|
|
ThinkingEffort::Medium,
|
|
ThinkingEffort::High
|
|
]
|
|
);
|
|
assert_eq!(r.default_effort, None);
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_anthropic_claude_opus_4_5_manual_budget() {
|
|
let r = resolve("anthropic", "claude-opus-4-5");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::ManualBudget);
|
|
assert_eq!(
|
|
r.supported_efforts.as_ref(),
|
|
&[
|
|
ThinkingEffort::Low,
|
|
ThinkingEffort::Medium,
|
|
ThinkingEffort::High
|
|
]
|
|
);
|
|
assert_eq!(r.default_effort, None);
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Anthropic adaptive xhigh-capable
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_anthropic_claude_opus_4_7_adaptive_xhigh() {
|
|
let r = resolve("anthropic", "claude-opus-4-7");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
assert_eq!(
|
|
r.supported_efforts.as_ref(),
|
|
&[
|
|
ThinkingEffort::Low,
|
|
ThinkingEffort::Medium,
|
|
ThinkingEffort::High,
|
|
ThinkingEffort::XHigh,
|
|
ThinkingEffort::Max
|
|
]
|
|
);
|
|
assert_eq!(r.default_effort, Some(ThinkingEffort::High));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_anthropic_claude_sonnet_5_adaptive_xhigh() {
|
|
let r = resolve("anthropic", "claude-sonnet-5-20260101");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::Max));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_anthropic_claude_fable_5_adaptive_xhigh() {
|
|
let r = resolve("anthropic", "claude-fable-5");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_anthropic_claude_mythos_5_adaptive_xhigh() {
|
|
let r = resolve("anthropic", "claude-mythos-5");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Anthropic adaptive no-xhigh
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_anthropic_claude_opus_4_6_adaptive_no_xhigh() {
|
|
let r = resolve("anthropic", "claude-opus-4-6");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
assert!(!r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::Max));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_anthropic_claude_sonnet_4_6_adaptive_no_xhigh() {
|
|
let r = resolve("anthropic", "claude-sonnet-4-6");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
assert!(!r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_anthropic_claude_mythos_preview_adaptive_no_xhigh() {
|
|
let r = resolve("anthropic", "claude-mythos-preview");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
assert!(!r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Anthropic fallbacks
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_anthropic_blank_adaptive_fallback() {
|
|
let r = resolve("anthropic", "");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert_eq!(r.default_effort, Some(ThinkingEffort::High));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_anthropic_concrete_unknown_omit_fields() {
|
|
let r = resolve("anthropic", "claude-ultra-9000");
|
|
assert_eq!(r.thinking_mode, ThinkingMode::OmitFields);
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// OpenAI gpt-5 family
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_openai_gpt5_pro_high_only() {
|
|
let r = resolve("openai", "gpt-5-pro");
|
|
assert_eq!(r.supported_efforts.as_ref(), &[ThinkingEffort::High]);
|
|
assert_eq!(r.default_effort, Some(ThinkingEffort::High));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_openai_gpt5_6_max_capable() {
|
|
let r = resolve("openai", "gpt-5.6");
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::Max));
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::None));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_openai_gpt5_5_no_max() {
|
|
let r = resolve("openai", "gpt-5.5");
|
|
assert!(!r.supported_efforts.contains(&ThinkingEffort::Max));
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_openai_gpt5_1_no_xhigh() {
|
|
let r = resolve("openai", "gpt-5.1");
|
|
assert!(!r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::None));
|
|
assert_eq!(r.default_effort, Some(ThinkingEffort::None));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
#[test]
|
|
fn test_openai_gpt5_base_minimal() {
|
|
let r = resolve("openai", "gpt-5");
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::Minimal));
|
|
assert!(!r.supported_efforts.contains(&ThinkingEffort::None));
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// OpenAI adversarial boundary cases
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_openai_gpt5_1106_matches_base_not_version() {
|
|
// 4-digit suffix is a date, not a version number — must match base table
|
|
let r = resolve("openai", "gpt-5-1106");
|
|
let base = resolve("openai", "gpt-5");
|
|
assert_eq!(
|
|
r.supported_efforts, base.supported_efforts,
|
|
"gpt-5-1106 must match base table (4-digit date segment)"
|
|
);
|
|
assert!(
|
|
!r.supported_efforts.contains(&ThinkingEffort::None),
|
|
"gpt-5-1106 base must NOT have none"
|
|
);
|
|
assert!(
|
|
r.supported_efforts.contains(&ThinkingEffort::Minimal),
|
|
"gpt-5-1106 base must have minimal"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn test_openai_gpt5_4o_matches_base_not_gpt5_4() {
|
|
// gpt-5-4o: '4o' after '-' is not a short version suffix (letter present)
|
|
// Must NOT match gpt-5.4 (would add xhigh), must match gpt-5 base
|
|
let r = resolve("openai", "gpt-5-4o");
|
|
let base = resolve("openai", "gpt-5");
|
|
assert_eq!(
|
|
r.supported_efforts, base.supported_efforts,
|
|
"gpt-5-4o must match base table"
|
|
);
|
|
assert!(
|
|
!r.supported_efforts.contains(&ThinkingEffort::XHigh),
|
|
"gpt-5-4o must NOT have xhigh (that's gpt-5.4 territory)"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn test_openai_gpt5_pro_before_base() {
|
|
let pro = resolve("openai", "gpt-5-pro");
|
|
let base = resolve("openai", "gpt-5");
|
|
assert_ne!(
|
|
pro.supported_efforts, base.supported_efforts,
|
|
"gpt-5-pro must hit -pro table, not base"
|
|
);
|
|
assert_eq!(pro.supported_efforts.as_ref(), &[ThinkingEffort::High]);
|
|
}
|
|
|
|
#[test]
|
|
fn test_openai_gpt5_date_suffix_still_matches_base() {
|
|
// gpt-5-20260101: 8-digit date suffix, must match base
|
|
let r = resolve("openai", "gpt-5-20260101");
|
|
let base = resolve("openai", "gpt-5");
|
|
assert_eq!(
|
|
r.supported_efforts, base.supported_efforts,
|
|
"gpt-5-20260101 must match base (4+ digit date)"
|
|
);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// DatabricksV2 routing
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_dbv2_gpt5_routes_openai_responses() {
|
|
let r = resolve("databricks_v2", "gpt-5.5");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::OpenAiResponses
|
|
);
|
|
assert!(!r.supported_efforts.contains(&ThinkingEffort::Max));
|
|
}
|
|
|
|
#[test]
|
|
fn test_dbv2_claude_routes_anthropic_messages() {
|
|
let r = resolve("databricks_v2", "claude-opus-4-7");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::AnthropicMessages
|
|
);
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
}
|
|
|
|
#[test]
|
|
fn test_dbv2_databricks_prefix_stripped() {
|
|
let r = resolve("databricks_v2", "databricks-claude-opus-4-7");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::AnthropicMessages
|
|
);
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
}
|
|
|
|
#[test]
|
|
fn test_dbv2_goose_claude_prefix_stripped() {
|
|
let r = resolve("databricks_v2", "goose-claude-fable-5");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::AnthropicMessages
|
|
);
|
|
assert_eq!(r.thinking_mode, ThinkingMode::Adaptive);
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
}
|
|
|
|
#[test]
|
|
fn test_dbv2_team_prefix_stripped() {
|
|
let r = resolve("databricks_v2", "team-x-claude-opus-4-7");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::AnthropicMessages
|
|
);
|
|
}
|
|
|
|
// Segment collision tests — these must NOT match named code names as substrings
|
|
#[test]
|
|
fn test_dbv2_consolidated_llama_not_sol_segment() {
|
|
// 'sol' is a substring of 'consolidated' — segment match must prevent this
|
|
let r = resolve("databricks_v2", "consolidated-llama");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::MlflowChatCompletions,
|
|
"consolidated-llama must NOT route OpenAI via 'sol' substring"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn test_dbv2_terraform_coder_not_terra_segment() {
|
|
// 'terra' is a prefix of 'terraform' — segment match must prevent this
|
|
let r = resolve("databricks_v2", "terraform-coder");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::MlflowChatCompletions,
|
|
"terraform-coder must NOT route OpenAI via 'terra' prefix-of-'terraform'"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn test_dbv2_corpus_reranker_not_opus_segment() {
|
|
// 'opus' must be a SEGMENT (full token), not a substring of 'corpus'
|
|
let r = resolve("databricks_v2", "corpus-reranker");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::MlflowChatCompletions,
|
|
"corpus-reranker must NOT route Anthropic via 'opus' substring of 'corpus'"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn test_dbv2_octopus_model_not_opus_segment() {
|
|
let r = resolve("databricks_v2", "octopus-model");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::MlflowChatCompletions,
|
|
"octopus-model must NOT route Anthropic (no 'opus' segment)"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn test_dbv2_goose_opus_5_is_anthropic() {
|
|
// goose-opus-5: segments are [goose, opus, 5]; 'opus' is a segment → Anthropic route
|
|
let r = resolve("databricks_v2", "goose-opus-5");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::AnthropicMessages,
|
|
"goose-opus-5 must route Anthropic ('opus' is a segment)"
|
|
);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// P2-A resolver contract vectors
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_resolver_exact_raw_id_hit() {
|
|
// databricks-gpt-5-4-mini has an exact record with low|medium|high override
|
|
let r = resolve("databricks_v2", "databricks-gpt-5-4-mini");
|
|
assert_eq!(
|
|
r.supported_efforts.as_ref(),
|
|
&[
|
|
ThinkingEffort::Low,
|
|
ThinkingEffort::Medium,
|
|
ThinkingEffort::High
|
|
],
|
|
"exact record must override family rule (models.dev: low|medium|high, not none+xhigh)"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn test_resolver_prefixed_alias_misses_exact() {
|
|
// team-x-databricks-gpt-5-4-mini: raw exact lookup MUST miss (different raw ID)
|
|
// Falls to family rules (gpt5-4 → none+xhigh)
|
|
let r = resolve("databricks_v2", "team-x-databricks-gpt-5-4-mini");
|
|
assert!(
|
|
r.supported_efforts.contains(&ThinkingEffort::XHigh),
|
|
"prefixed alias must miss exact record and fall to family (which has xhigh)"
|
|
);
|
|
assert!(
|
|
r.supported_efforts.contains(&ThinkingEffort::None),
|
|
"prefixed alias must fall to gpt5-4 family (which has none)"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn test_resolver_cross_provider_misses_exact() {
|
|
// Same raw ID but different provider — exact record is databricks_v2-scoped
|
|
let r = resolve("openai", "databricks-gpt-5-4-mini");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::NotApplicable,
|
|
"openai provider must get not-applicable (record is databricks_v2-scoped)"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn test_resolver_exact_record_complete_both_axes() {
|
|
// databricks-gpt-5-6-sol: exact record with efforts from models.dev +
|
|
// route materialized from gpt5-6 family rule
|
|
let r = resolve("databricks_v2", "databricks-gpt-5-6-sol");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::OpenAiResponses,
|
|
"exact record must have route from family rule materialization"
|
|
);
|
|
assert!(
|
|
r.supported_efforts.contains(&ThinkingEffort::Max),
|
|
"exact record must have max (from models.dev override)"
|
|
);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// P2-B blank vs concrete-unknown fallback vectors
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_dbv2_blank_route_unknown_all7() {
|
|
let r = resolve("databricks_v2", "");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::RouteUnknown,
|
|
"DBv2 blank must return route-unknown"
|
|
);
|
|
assert_eq!(
|
|
r.supported_efforts.len(),
|
|
7,
|
|
"DBv2 blank must expose all 7 efforts"
|
|
);
|
|
assert_eq!(r.default_effort, Some(ThinkingEffort::Medium));
|
|
}
|
|
|
|
#[test]
|
|
fn test_dbv2_concrete_unknown_mlflow_no_max() {
|
|
let r = resolve("databricks_v2", "some-unknown-model-xyz");
|
|
assert_eq!(
|
|
r.databricks_v2_wire_route,
|
|
DatabricksV2Route::MlflowChatCompletions,
|
|
"DBv2 concrete unknown must return mlflow-chat"
|
|
);
|
|
assert!(
|
|
!r.supported_efforts.contains(&ThinkingEffort::Max),
|
|
"DBv2 concrete unknown must NOT expose max"
|
|
);
|
|
assert_eq!(r.supported_efforts.len(), 6);
|
|
}
|
|
|
|
#[test]
|
|
fn test_openai_blank_all_except_max() {
|
|
let r = resolve("openai", "");
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
assert!(!r.supported_efforts.contains(&ThinkingEffort::Max));
|
|
assert_eq!(r.default_effort, Some(ThinkingEffort::Medium));
|
|
}
|
|
|
|
#[test]
|
|
fn test_openai_concrete_unknown_all_except_max() {
|
|
let r = resolve("openai", "gpt-4o");
|
|
assert_eq!(r.databricks_v2_wire_route, DatabricksV2Route::NotApplicable);
|
|
assert!(!r.supported_efforts.contains(&ThinkingEffort::Max));
|
|
assert_eq!(r.default_effort, Some(ThinkingEffort::Medium));
|
|
}
|
|
|
|
#[test]
|
|
fn test_anthropic_blank_adaptive_full_fallback() {
|
|
let r = resolve("anthropic", "");
|
|
assert_eq!(
|
|
r.thinking_mode,
|
|
ThinkingMode::Adaptive,
|
|
"Anthropic blank must assume adaptive"
|
|
);
|
|
assert!(r.supported_efforts.contains(&ThinkingEffort::XHigh));
|
|
assert_eq!(r.default_effort, Some(ThinkingEffort::High));
|
|
}
|
|
|
|
#[test]
|
|
fn test_anthropic_concrete_unknown_omit_fields_fallback() {
|
|
let r = resolve("anthropic", "claude-ultra-9000");
|
|
assert_eq!(
|
|
r.thinking_mode,
|
|
ThinkingMode::OmitFields,
|
|
"Anthropic concrete unknown must omit fields (never guess request shape)"
|
|
);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Completeness: every result has every axis populated
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[test]
|
|
fn test_every_result_is_complete() {
|
|
let test_cases = vec![
|
|
("anthropic", ""),
|
|
("anthropic", "claude-3-7-sonnet-20250219"),
|
|
("anthropic", "claude-opus-4-7"),
|
|
("anthropic", "claude-ultra-9000"),
|
|
("openai", ""),
|
|
("openai", "gpt-5"),
|
|
("openai", "gpt-5.6"),
|
|
("openai", "unknown-model"),
|
|
("databricks_v2", ""),
|
|
("databricks_v2", "databricks-gpt-5-4-mini"),
|
|
("databricks_v2", "goose-claude-fable-5"),
|
|
("databricks_v2", "unknown-model"),
|
|
("databricks", ""),
|
|
("openrouter", ""),
|
|
("openai-compat", ""),
|
|
("", ""),
|
|
];
|
|
for (provider, model) in test_cases {
|
|
let r = resolve_model_capabilities(provider, model);
|
|
assert!(
|
|
!r.supported_efforts.is_empty(),
|
|
"supported_efforts must be non-empty for ({provider}, {model})"
|
|
);
|
|
// default_effort is allowed to be None (manual-budget "Inherit")
|
|
// registry_label is allowed to be None
|
|
}
|
|
}
|
|
}
|