groundhog-1v1.7 · 10 s rule

Data: JevBench v1.8.19, published 2026-11-21 · method v1.7 (10 s rule), measured under the rule

Capability 80.9 · API offering, listed but not ranked on the API leaderboard

Added after the v1.8.10 ranking review; it is ranked at the next review.

Last measured: 2026-11-21 · measurement release: v1.8.19

MIT (modifications) and Apache-2.0 (base model)

Base model: google/gemma-4-26b-a4b-itmodel card

intelligence

68.5

calibration

93.3

speed

90.5

cost

53.7

Composite (secondary)

72.8 · Not ranked on this board

Cost and measurement conditions

$0.035 per 1,000 decisions (estimated)

Baseten Model APIs launch list price, USD 0.035/M input with free output (https://www.baseten.co/library/groundhog-1/, read 21 Nov 2026). Counted under the frozen I-1 launch-list-price rule, excluding promotional credits. This is below the google/gemma-4-26b-a4b-it base-model reference, USD 0.09/M input and USD 0.3/M output from the frozen OpenRouter snapshot, so the M2 §2 floor binds (price floor: base-model reference price applied). The API reported one output token per item, priced at the reference output rate. The common 1,489-item basis uses 571,160 measured input tokens and yields USD 0.0348227669576897 per 1,000 decisions. The full 1,500-item run used 972,512 input tokens and yields USD 0.05865072 per 1,000 decisions.

p50 latency: 0.3 s

none (API)

Baseten Model APIs /v1/systemone (inference.baseten.co); native probabilities; served model decision-models/baseten-groundhog-26b-latest.

Measured latency over 1,500 decisions: p50 0.250 s · p95 0.353 s

0 decisions over 10 s.

A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.

Breakdowns (not part of the score)

Chance-corrected competence (0 = chance, 100 = perfect) by category on open+sealed. Raw values, not equated; the same cells feed the compare view.

Capability by subject topic

subject topics: not labelled for this draw yet (top-up pending)

Use cases (TypeSafe categories)

Model routing79.4n=148
Legal & compliance81.1n=56
Customer support80.3n=66
E-commerce80.7n=62
Insurance claims68.5n=62
Risk assessment70.9n=48
Financial crime72.4n=63
Feature extraction78.4n=72
Lead generation92.7n=75
Recruiting87.1n=65
Knowledge graphs74.9n=67
LLM guardrails90.0n=71
Moderation88.6n=59
Code linting76.6n=69
Search & retrieval89.5n=67
Science45.8n=42
Advertising60.5n=48
Gaming63.1n=58
Demand forecasting17.7n=66

Cells need at least 30 answered items; † fewer than 30 (indicative only; drawn as hollow, lighter points on the compare radars).

Model and serving disclosure

Hosted API via Baseten Model APIs, served as decision-models/baseten-groundhog-26b-latest (a -latest alias), measured 21 Nov 2026; not tied to a Hugging Face revision. Reference Hugging Face HEAD at run time: baselabs/groundhog-1@f4602c43. Baseten documentation lists 131,072 context tokens, while its management API reported 16,384. Five items of 77–82k input tokens received answers with full reported usage; reported usage is not proof that every token was read. Failed or unanswered requests count as wrong answers. Baseten retention statements are vendor claims and were not audited.

Published source