groundhog-1v1.7 · 10 s rule
Data: JevBench v1.8.19, published 2026-11-21 · method v1.7 (10 s rule), measured under the rule
Capability 80.9 · API offering, listed but not ranked on the API leaderboard
Added after the v1.8.10 ranking review; it is ranked at the next review.
Last measured: 2026-11-21 · measurement release: v1.8.19
MIT (modifications) and Apache-2.0 (base model)
intelligence
68.5
calibration
93.3
speed
90.5
cost
53.7
Composite (secondary)
72.8 · Not ranked on this board
Cost and measurement conditions
$0.035 per 1,000 decisions (estimated)
Baseten Model APIs launch list price, USD 0.035/M input with free output (https://www.baseten.co/library/groundhog-1/, read 21 Nov 2026). Counted under the frozen I-1 launch-list-price rule, excluding promotional credits. This is below the google/gemma-4-26b-a4b-it base-model reference, USD 0.09/M input and USD 0.3/M output from the frozen OpenRouter snapshot, so the M2 §2 floor binds (price floor: base-model reference price applied). The API reported one output token per item, priced at the reference output rate. The common 1,489-item basis uses 571,160 measured input tokens and yields USD 0.0348227669576897 per 1,000 decisions. The full 1,500-item run used 972,512 input tokens and yields USD 0.05865072 per 1,000 decisions.
p50 latency: 0.3 s
none (API)
Baseten Model APIs /v1/systemone (inference.baseten.co); native probabilities; served model decision-models/baseten-groundhog-26b-latest.
Measured latency over 1,500 decisions: p50 0.250 s · p95 0.353 s
0 decisions over 10 s.
A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.
Breakdowns (not part of the score)
Chance-corrected competence (0 = chance, 100 = perfect) by category on open+sealed. Raw values, not equated; the same cells feed the compare view.
Capability by subject topic
subject topics: not labelled for this draw yet (top-up pending)
Use cases (TypeSafe categories)
| Model routing | 79.4 | n=148 |
|---|---|---|
| Legal & compliance | 81.1 | n=56 |
| Customer support | 80.3 | n=66 |
| E-commerce | 80.7 | n=62 |
| Insurance claims | 68.5 | n=62 |
| Risk assessment | 70.9 | n=48 |
| Financial crime | 72.4 | n=63 |
| Feature extraction | 78.4 | n=72 |
| Lead generation | 92.7 | n=75 |
| Recruiting | 87.1 | n=65 |
| Knowledge graphs | 74.9 | n=67 |
| LLM guardrails | 90.0 | n=71 |
| Moderation | 88.6 | n=59 |
| Code linting | 76.6 | n=69 |
| Search & retrieval | 89.5 | n=67 |
| Science | 45.8 | n=42 |
| Advertising | 60.5 | n=48 |
| Gaming | 63.1 | n=58 |
| Demand forecasting | 17.7 | n=66 |
Cells need at least 30 answered items; † fewer than 30 (indicative only; drawn as hollow, lighter points on the compare radars).
Model and serving disclosure
Hosted API via Baseten Model APIs, served as decision-models/baseten-groundhog-26b-latest (a -latest alias), measured 21 Nov 2026; not tied to a Hugging Face revision. Reference Hugging Face HEAD at run time: baselabs/groundhog-1@f4602c43. Baseten documentation lists 131,072 context tokens, while its management API reported 16,384. Five items of 77–82k input tokens received answers with full reported usage; reported usage is not proof that every token was read. Failed or unanswered requests count as wrong answers. Baseten retention statements are vendor claims and were not audited.