JevBench by Benchmark Heaven · Decision model evaluation
Decision model benchmark methodology — JevBench v1.8.7
JevBench measures typed decisions: application state and a bounded rubric go in; a structured answer and, where supported, probabilities come out. Compare accuracy and calibration alongside measured latency and modeled cost. The open-weights board leads with Capability; the hosted API board leads with Composite.
Reviewed · Maintained by Benchmark Heaven, independently of TypeSafe AI.
Release and measurement dates
Published release v1.8.7.
Artifact anchor (original scoring release) v1.6.1, initially published . The versioned leaderboard retains its release context. Additions stored in the results file: a6 (2026-10-07), a7 (2026-10-07), a8 (2026-10-07), a11 (2026-10-08), a12 (2026-10-08); amended through 2026-10-08; latest row measurement 2026-10-07. Cite source_sha256 5cd8c1332226ea9c179883d2285c0d8b8cc593ae38f2645197364a7ddd9c83c1 for these exact result bytes. The live board may also include later dated API reruns and presentation revisions.
The release draw contains 1,200 sealed and 300 public decisions. Self-hosted systems answer 1,500 decisions. The v1.6.1 full-set API rows answer the same set. Older subsets and later reruns are explicitly labeled and may be equated. Missing or refused answers are counted by the scorer, rather than silently removed.
Capability and Composite are different scores
Capability: Arithmetic mean of Intelligence and Calibration; ranked models within both official caps. Headline on the open-weights board. The cost cap is USD 0.0646 per 1,000 decisions, and the adjusted median-latency cap is 1.23 seconds. Eligibility is separate from the raw score.
Composite: Weighted harmonic mean of Intelligence, Calibration, Speed and Cost, multiplied by squared penalties when Intelligence is below its configured floor, or Speed or Cost is below 50. Equal 25% weights do not mean an arithmetic average.
Official option A uses weights Calibration 25%, Cost 25%, Intelligence 25%, Speed 25%; Intelligence floor 50. A zero axis gives zero Composite; a missing axis gives no score.
H = Σw / Σ(w / axis). Composite = H × min(1, I / 50)² × min(1, Speed / 50)² × min(1, Cost / 50)².
Cost basis: On every draw, very long items that a ranked row refuses because they exceed its context length are left out of cost for all rows: cost per 1,000 decisions is computed on the same remaining items for every row. A refused item still counts as a wrong answer for Intelligence. First published as the v1.6.1 cost rule (5 Oct 2026; 23 items on that draw). From v1.6.30 it applies to draw v1.6-regular-20261012: the 15 items the Jev reference refused with max_tokens_exceeded are left out and 1,485 remain. Rows of draw v1.6-regular-20261010-a2 still show the all-items cost they were published with until a following release re-costs them under this rule.
Intelligence is chance-corrected decision accuracy, with tier weighting and a public/sealed gap adjustment. Calibration evaluates probability distributions; it is not simply accuracy among confident answers. Noul handling, per-type normalization and equating are versioned in the scorer. The four normalized axes are on a 0–100 scale; actual seconds and USD remain separate values.
Latency: what is measured and what is estimated
We time every request and use the median (p50) and the 95th percentile (p95). API models are timed end to end from our measuring host against the vendor endpoint, so their latency includes the network round trip. Self-hosted (open-weights) models run on hardware we rent and operate (a GPU, or a CPU host for the smallest models) and are timed over loopback, where there is no network at all. We tag that value “on-GPU, no network” (or “on-CPU, no network”) wherever it appears, and it is not directly comparable with API latency.
Those two numbers are not comparable as they stand, so for every self-hosted model the ranking and the latency cap use the same estimate that adds the network: 2 × measured p50 + 150 ms (the same adjustment is applied to p95). This is an assumption, not a measurement: it stands in for the extra time an API call to the same model would spend on the network and the serving layer. The Speed axis, the Jev-class latency cap and every rank use the estimate; the measured loopback value is shown next to it on the board, the model pages and in the JSON (p50_s_measured, latency_basis).
Hosted endpoint timing on request. A model author may provide a hosted endpoint for their model. We then time it exactly like an API, network included, with no assumption, and that measured latency replaces the estimate for that row, noted in the revision history. Contact info@productivity-boost.com or note it in your submission.
Calibrating the assumption (planned, not yet applied). The 2 × + 150 ms rule is a fixed assumption. We intend to check it against real end-to-end timings of self-hosted models behind public endpoints, for example endpoints that model authors provide. Any change to the rule will be announced in the revision history and applied to all self-hosted rows at once, never switched silently for single rows. If we use timings from decisionmodels.io, we will say so, because decisionmodels.io hosts some of these models.
10-second limit per decision (method v1.7, from JevBench v1.7.0). Every decision is timed on the harness the system is measured with, from the moment the request is sent until the answer is received. A decision that takes longer than 10 seconds counts as failed: it is scored as a wrong answer and as zero calibration for that item (each type's calibration is multiplied by the share of its decisions answered within 10 seconds). Exactly 10 seconds passes. The limit uses the measured time, not the displayed estimate for self-hosted rows. The rule is the same for every system. The Speed axis uses the measured times of all decisions, including the slow tail, and every row shows its median (p50) and 95th-percentile (p95) time and how many of its decisions took longer than 10 seconds. All rows were recomputed from their recorded per-decision times. Systems whose times were recorded on a shared, loaded machine are listed as “re-measure pending” with their v1.6 values until they are measured again. Earlier results stay available with their method label; the previous method page is method v1.6.
Reproducibility and model versions
Scorer and public benchmark repository · Board source · Release manifest and scorer hashes · Published aggregate artifact.
Artifact SHA-256: 5cd8c1332226ea9c179883d2285c0d8b8cc593ae38f2645197364a7ddd9c83c1. The data card documents model pins, measurement dates, price bases and latency adjustments. Unknown pins and dates remain null; hosted service names do not guarantee immutable weights.
Sealed questions, identifiers and predictions are not downloadable. Public-split experiments can be reproduced, and aggregate Composite scores can be recalculated from the published axes. Complete independent reruns of the sealed evaluation require the official evaluation process.
Limitations and historical methods
Coverage is finite and uneven across languages and domains. Cost is modeled from a stated hardware or list-price basis; latency depends on the measured runtime and location. Confidence, coherence, robustness and real-world error costs need additional workload-specific checks. Training on the public split must be disclosed, and a sealed split is not a guarantee of no prior exposure.
Historical releases retain their original scoring definitions and measurement sets. Do not compare scores across method versions as if they were the same experiment. ImageJevBench v0.3.0 has a separate image pool and frozen eligibility envelope; core and computer-use tracks remain separate. Its reserve size is not the number of decisions answered by each model.
Another project named JevBench tests metamorphic coherence. It is independent of Benchmark Heaven and measures consistency rather than this board's accuracy/cost/latency score.