Hosting = where inference runs; company = where the provider or lab is registered.
EU hosting means the route's inference runs inside the EU: an EU region, AWS Bedrock's EU cross-region (geo) profiles, Azure's Europe Data Zone, or a provider whose entire public fleet is documented as EU-hosted, each checked per model against the provider's documentation. Global deployments do not count, and neither does an EU billing region, an EU company or an EU control plane on its own. One disclosed company-policy exception stays in, marked “EU equivalent”.
Evidence requirements (benchmark evidence, priced provider, measured task tokens) sit above the table they apply to.
Applies to price views & model offers; benchmark evidence stays unfiltered.
JevBench is Benchmark Heaven's own benchmark for Jev-class decision models: state and a bounded rubric in, a typed answer out.
Making decisions from images? Explore Image JevBench v0.1.2 and compare its systems.
Scored 27 Sept 2026 · protocol jevbench::v1.4 · 534 public + 308 sealed aggregate decisions · one request at a time from a server in Germany · harness, public tasks & scoring rules (MIT) · results JSON sha256 7f39b2f742a6… · v1.0 results
JevBench v1.4.2.2 · headline ranking
Capability averages Intelligence and Calibration. Imajev-4B leads the Jev-class systems with 66.3.
Jev-class means at most 2× Jev's cost and median latency. How we choose ↘
Wide coloured bar = Capability (0–100). Thin red line = cost per 1,000 decisions; log scale, each gridline = 10×, shorter is cheaper. * = est. (estimated cost). # counts ranked Jev-class systems; “–” marks unranked or outside systems. Tap ⓘ for the full values, median latency and cost relative to Jev.
Sorted by Capability, not numbered. Each row says which limit it misses, measured against Jev 1.13.0 ($0.040 per 1,000 decisions, median 0.65 s).
Jev-class = cost per decision at most 2× Jev 1.13.0's (≤ $0.080 per 1,000 decisions) and median latency at most 2× Jev 1.13.0's (≤ 1.30 s, the adjusted p50 — the same median the speed chart plots, not the four-axis Speed score). 52 of 93 systems qualify; the other 41, including the general-purpose LLMs, are listed below the divider in the ranking. JevK5 v0.2.0, OpenSourceJev have no recorded median latency (carried from v1.3); for them the Speed axis decides, at the 2× latency equivalent (Speed ≥ 77.2). The charts below show speed and cost beside Capability; the official JevBench Score weighs all four axes.
Jev-class systems are shown by default. Bubble size follows the official JevBench Score. The five most capable Jev-class systems are labelled.
Upper right is best: more capable and cheaper. Cost is USD per 1,000 decisions on a log scale. The dashed line is 2× Jev’s cost.
Upper right is best: more capable and faster. Speed here is the median-latency speed — the same adjusted median (p50) latency the Jev-class limit uses — a log scale, so each 20 points is 10× faster (median under the numbers). The dashed line is 2× Jev’s median latency.
JevBench v1.4.2.2
Official· four axes 0–100, equal-weight harmonic mean · What changed in v1.4 ↓
Imajev-4B leads the JevBench Score at 67.37, ahead of Plumb-4B (65.84). The v1.4.2 scoring code and earlier measurement rows are unchanged.
Greener = stronger within its column.
95 of 95 systems, sorted by official rank, #1 first.
Weights are relative: each axis counts in proportion to its slider. The score stays a weighted harmonic mean with the low-axis gates; an axis at 0 drops out together with its gate. Only equal weights give the official JevBench Score and rank.
Score = 4 / (1/I + 1/C + 1/S + 1/K) (each 0–100; × (axis / 50)² for Intelligence, Speed or Cost below 50)
system-one-openPick any two. Four radars: the score axes, accuracy per tier including the sealed set, accuracy by family on the current v1.4 question set (hard tier and sealed set together), and the sealed set alone. Further out is better on every spoke; the link keeps the pair.
Imajev-4B has no published hard-tier family breakdown; families that need it are left out (—).
| Spoke | A: Jev 1.13.0 | B: Imajev-4B |
|---|---|---|
| The four score axes | ||
| Intelligence | 53.1 | 52.2 |
| Calibration | 76.3 | 80.4 |
| Speed | 83.3 | 90.6 |
| Cost | 52.0 | 59.7 |
| Accuracy per tier, incl. sealed | ||
| Easy | 100% | 100% |
| Standard | 99% | 99% |
| Judge | 95% | 89% |
| Hard | 74% | — |
| Sealed | 37% | 37% |
| Current question set by family (hard + sealed) | ||
| Ambiguous / abstain | 43% | — |
| Judge | 54% | — |
| Long policy | 44% | — |
| Multi-hop | 64% | — |
| Probability | 63% | — |
| Temporal / numeric | 28% | — |
| Trade-off | 55% | — |
| Routing | 100% | — |
| Trap / adversarial | 83% | — |
| Paraphrase | 64% | 50% |
| Safety judge | 38% | 44% |
| Sealed set by family | ||
| Ambiguous / abstain | 30% | 41% |
| Judge | 34% | 44% |
| Long policy | 28% | 33% |
| Multi-hop | 45% | 34% |
| Paraphrase | 64% | 50% |
| Probability | 50% | 25% |
| Safety judge | 38% | 44% |
| Temporal / numeric | 29% | 30% |
| Trade-off | 38% | 38% |
| Trap / adversarial | 42% | 58% |
Every system with its four axes, public and sealed accuracy and the gap between them. Click a column heading to sort; filter by name, type, openness, API flag or release. On a phone the name column stays put while the table scrolls sideways. † = a note on that system — tap it to read.
Greener = stronger within its column; faster counts as stronger. Badges: API sealed text went to the operator's endpoint · est. estimated price · ann. announced price not yet bookable · not ranked partial run or honorable mention · new first listed in v1.4.2.2.
| Endpoint | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Imajev-4B †Imajev-4B: requested adapter c9e5f132465da85d31735ec502d5557982671a7d and server a0134749e0900189c129cd6bb5000969f3b64bb5; one rotation with calibration.json. The optimized image included flash-linear-attention/fla-core 0.5.2 and causal-conv1d 1.7.0; CUDA profiling confirmed the pinned kernels ran. Cost is an estimate using the public DeepInfra Qwen/Qwen3.5-4B base-model reference price ($0.03/M input, $0.15/M output), with no generated output tokens; it is not a GPU bill. | 67.4 | 52.2 | 80.4 | 90.6 | 59.7 | 86.1% | 37.0% | +49.1 pp | est.$0.022 | 0.04 s | RunPod GPU |
| 2 | Plumb-4B †Crh225/plumb-4b @ 55de037801a8a9b9de3db5c0e16cef86210c2186: merged bf16 weights, LoRA r16 on the attention projections of JevK5 v0.2 (itself Qwen3.5-4B), served by the author's documented command, jevk5 v0.2.0's own jevk5-serve (github.com/allebee/jevk5, Apache-2.0), unmodified. One forward pass per decision: softmax over the declared options' answer-letter logits at the last position divided by the package's own T = 2.07 (jevk5_config.json), up to 16 options, 0 generated tokens; inputs over 16,384 tokens are refused rather than truncated. Disclosed by the author: no JevBench item was trained or tuned on and every checkpoint and temperature choice came from his own held-out sets, but the 231 public items were scored after each of five training rounds and that aggregate feedback shaped the recipe (hard mining, long documents, document length) — development against the public distribution, stated plainly. His own overlap audit dropped 9 training items sharing more than two 8-word spans with a public item. Independent top-five gate (25 Sep 2026): LEGIT — cost basis (same as its JevK5 parent), public-versus-sealed gap against like-for-like rows, an 8-gram screen of its released training data (no row shares more than two 8-grams with a public item) and an exact composite recomputation. Offline self-hosted inference of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) in a network-disabled, read-only container on an evaluator-owned Lium H100 80 GB (Hopper) pod, through JevBench's unchanged typesafe adapter against the author's own server on loopback; no operator endpoint; no golds were exposed. Latency is the serial request wall time on the standard+judge items with the self-hosted adjustment. Cost is a labelled estimate at the EmpirioLabs Qwen3.5-4B public pay-as-you-go list price ($0.04/M input, no output generated), applied to this system's measured tokens; it is not a GPU bill. 25 Sep 2026 pre-release correction: retired DeepInfra $0.03/M input reference replaced by bookable EmpirioLabs $0.04/M exact-base input reference; draft score 67.09 to 65.84. | 65.8 | 53.0 | 75.5 | 93.5 | 55.8 | 89.6% | 38.0% | +51.6 pp | est.$0.030 | 0.02 s | RunPod GPU |
| 3 | decider-4b v2 †Decider-ai 1.2.2 (PyPI wheel identical to tag v1.2.2 abadc94), weights Mapika/decider-4b rev 7ab294cbdf6be6ac17fc818c10cdead744393d92 (decider_config version 4b-v2, T=1.935), uvicorn decider.serve:app. Disclosed by the author: 8,000 of the v2 LoRA rows come from generators written from the published names of the ten sealed families (no item read). Independent #1 gate (24 Sep): LEGIT. The author's private stage-2 training rows could not be audited for overlap with public items. Offline self-hosted inference of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) in a network-disabled, read-only container on an evaluator-owned Lium RTX PRO 6000 pod, through JevBench's unchanged typesafe adapter against the author's own server on loopback; no operator endpoint; no golds were exposed. Latency is the serial request wall time on the standard+judge items with the self-hosted adjustment. Cost is a labelled estimate: DeepInfra Qwen/Qwen3.5-4B list price $0.03/M input (4B dense size class, as decider-2b), read 2026-09-24, over the server's ownusage.input_tokens; it is not a GPU bill. | 64.1 | 49.4 | 75.0 | 92.9 | 60.9 | 83.5% | 34.7% | +48.8 pp | est.$0.020 | 0.02 s | RunPod GPU |
| 4 | 63.3 | 53.1 | 76.3 | 83.3 | 52.0 | 86.6% | 36.7% | +49.9 pp | $0.040 | 0.65 s | API | |
| 5 | 62.0 | 48.9 | 74.5 | 91.1 | 59.5 | 85.3% | 33.1% | +52.2 pp | est.$0.022 | — | unknown | |
| 6 | Cygnet †Blockbrain-ai/cygnet-recipe 81974de: frozen google/gemma-4-12B-it rev 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7 on unmodified vLLM 0.30.0 (image vllm/vllm-openai:v0.30.0@sha256:8a69ffad…), the author's shim: options as letters in the benchmark's label order, logits masked to the option letters, one calibration temperature T=3.4 fitted on the author's own items. The shim sends its own system prompt and renders structured state with json indent=1. Over-context/over-26-option inputs are HTTP 422 (a wrong answer). Offline self-hosted inference of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) in a network-disabled, read-only container on an evaluator-owned Lium RTX PRO 6000 Blackwell 96 GB pod, through JevBench's unchanged typesafe adapter against the author's own server on loopback; no operator endpoint; no golds were exposed. Latency is the serial request wall time on the standard+judge items with the self-hosted adjustment. Cost is a labelled estimate: OpenRouter google/gemma-3-12b-it list price $0.05/M input (the nearest hosted 12B Gemma; gemma-4-12b-it is not listed; the Winnow-12B / Jev-Omni precedent), read 2026-09-24, over the server's ownusage.input_tokens; it is not a GPU bill. | 61.8 | 49.5 | 74.9 | 90.7 | 52.8 | 87.9% | 33.8% | +54.1 pp | est.$0.037 | 0.04 s | RunPod GPU |
| 7 | Hopper †Author discloses heavy public-benchmark-directed development (26 model/prompt configs, 20+ calibration-map variants observed against the public half); scan of 17 released files vs 231 public tasks found 0 state/instruction matches; training corpus not released so overlap not independently verifiable. | 59.4 | 48.0 | 79.1 | 86.8 | 58.7 | 82.3% | 34.1% | +48.2 pp | est.$0.024 | 0.13 s | RunPod GPU |
| 8 | Winnow-12B Q8 †The submitted Q8_0 GGUF ran through the pinned author's TypeSafe-compatible /v1/systemone server with 8,192 context, four resident decision branches, Q8 KV, and full GPU offload. The private training corpus was not released. The author's checksum-based audit reports zero exact public-item overlap, but that claim cannot be independently reproduced; our scan found no exact public state or instruction text in the released artifacts. Cost uses the $0.05/M-input hosted Gemma 3 12B reference, not free/100. | 55.6 | 48.3 | 64.8 | 82.3 | 52.9 | 85.7% | 33.1% | +52.6 pp | est.$0.037 | 0.23 s | RunPod GPU |
| 9 | reflex 4B †The author's reflex-serve: Qwen3.5-4B with the published LoRA and its per-primitive calibration file; the state is encoded once and each question read from the label logits. Run serially on our GPU; the author discloses that the 231 public items were used four times as a development gate. | 54.0 | 47.5 | 70.4 | 68.0 | 59.7 | 79.2% | 28.2% | +51.0 pp | est.$0.022 | 1.80 s | RunPod GPU |
| 10 | djev †The measured endpoint was Maisa's hosted API in free preview; the cost uses its announced price ($0.035 per million input tokens, output free), and nothing was charged. The self-hostable djev-dev runtime is Apache-2.0 and applies a structured one-step inference method to Google's Apache-2.0 diffusiongemma-26B-A4B-it checkpoint; it adds no separately trained djev weights. Probabilities are djev's own (its docs call them experimental and uncalibrated). v1.4: hosted api.djev.dev was paused by its operator ("Serving is paused by the administrator"); sealed tier measured on the public djev runtime (Davipar/djev-dev 3ce907e, same weights, default mode) self-hosted on an H100; Speed/Cost kept from v1.3. | 52.2 | 47.0 | 55.4 | 91.4 | 57.6 | 84.0% | 29.9% | +54.1 pp | ann.$0.026 | 0.24 s | API |
| 11 | Jev-Omni †Akhilaaa3/Jev-Omni revision c050d51354147985d13286cf4acf90f562f2c631, the author's own load_model.py (merged text decision model + 256-way head) and his own predict(), transformers 5.17.0 / torch 2.8.0 from the pod image; built on the CPU and moved to CUDA with every nn.Linear weight cast to bfloat16 first - the same cast his reference loader jev_omni.py applies - because our 46 GB GPU cannot hold his fp32 copy; on our RunPod L40 in Czechia Offline local open-weight inference on the frozen 308-item v1.4 set in a network-disabled, read-only container; no operator endpoint received sealed text. | 51.3 | 46.8 | 64.1 | 81.5 | 53.0 | 88.7% | 32.1% | +56.6 pp | est.$0.037 | 0.22 s | RunPod GPU |
| 12 | metask-jev-4b †Model card discloses 44.8k+16.1k+390 training rows incl. synthetic families intentionally mirroring JevBench hard families, plus repeated evaluation on all 231 public items (benchmark-directed development disclosed). Scan of 61 files: 0 exact matches. | 47.8 | 44.7 | 66.9 | 89.1 | 54.5 | 79.7% | 27.6% | +52.1 pp | est.$0.033 | 0.07 s | RunPod GPU |
| 13 | 47.7 | 44.4 | 66.8 | 83.7 | 59.5 | 81.0% | 26.3% | +54.7 pp | est.$0.022 | 0.20 s | RunPod GPU | |
| 14 | Jobe Qwen3.5-4B †No trained weights/LoRA/calibration fit; release explicitly rejects fitted temperature/order averaging. Scan of 37 files vs 231 public tasks: 0 matches. | 46.9 | 44.1 | 66.1 | 85.6 | 59.5 | 81.0% | 25.6% | +55.3 pp | est.$0.022 | 0.13 s | RunPod GPU |
| 15 | local-jev Qwen3.5-4B †Scan of 72 files: 0 matches. Author discloses choosing JSON layout after observing results on the 231 public items (benchmark-directed choice, disclosed). | 46.8 | 44.4 | 73.3 | 75.0 | 55.8 | 80.5% | 26.0% | +54.5 pp | est.$0.030 | 0.71 s | RunPod GPU |
| 16 | 45.1 | 44.2 | 54.9 | 77.0 | 64.8 | 73.2% | 27.6% | +45.6 pp | est.$0.015 | 0.65 s | author demo | |
| 17 | spark-s1-4b-v6 †Abhishek085/spark-s1-4b-v6 revision 93d49ddbfb29212e3296635a75a3e80cf69da027, code github.com/abhishek085/open-spark-jev 30ac6d89b7fa36c644cf86aac68f35c1d276a919, the author's own MenuScorer.decide with his fitted calibration.json temperature, bf16, base Qwen/Qwen3.5-4B, transformers 5.17.0 / torch 2.8.0 from the pod image, flash-linear-attention 0.5.2 installed, causal_conv1d not installable here (no wheel builds against this toolchain), on our RunPod L40 in Czechia Offline local open-weight inference on the frozen 308-item v1.4 set in a network-disabled, read-only container; no operator endpoint received sealed text. | 44.6 | 45.1 | 47.6 | 81.0 | 57.9 | 79.2% | 26.6% | +52.6 pp | est.$0.025 | 0.31 s | RunPod GPU |
| 18 | Malkuth-4B †Dhtocks/malkuth-4b rev 11dc416995dab324803cb6c533f1d5c69e19d630 (rank-16 LoRA + pointer head over Qwen/Qwen3.5-4B-Base rev 1001bb4d826a52d1f399e183466143f4da7b741b), served by kev.serve from jaredpalmer/kev 557598fced1dada75dfbf36ed144dce309ac6ceb (the author's evaluation revision). The released head carries temperature 1.0 (no fitted temperature), although the card says calibrated. Offline self-hosted inference of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) in a network-disabled, read-only container on an evaluator-owned Lium RTX PRO 6000 pod, through JevBench's unchanged typesafe adapter against the author's own server on loopback; no operator endpoint; no golds were exposed. Latency is the serial request wall time on the standard+judge items with the self-hosted adjustment. Cost is a labelled estimate: DeepInfra Qwen/Qwen3.5-4B list price $0.03/M input (4B one-pass size class, as kev-4b), read 2026-09-24, over the server's ownusage.input_tokens; it is not a GPU bill. | 44.5 | 43.1 | 61.5 | 88.3 | 61.5 | 74.9% | 23.4% | +51.5 pp | est.$0.019 | 0.09 s | RunPod GPU |
| 19 | jqv †A stock Qwen3-32B with no decision training: the state is prefilled once, each question is an isolated branch and the answer is read from the option-letter logits, with one fitted temperature (3.02, 400 MMLU validation items). Re-run in v1.2.8 on our own GPU from the now-public serving code (Octalab-Inc/jqv 0189b67), so all 534 decisions including the held-out hard items were asked; this full run replaces the v1.2.7 partial row, which had been measured on the submitter's machine. Cost is the base model's public per-token tariff, not free. | 44.4 | 46.4 | 71.6 | 74.6 | 47.5 | 80.1% | 28.2% | +51.8 pp | est.$0.056 | 0.75 s | RunPod GPU |
| 20 | Qwen3-Reranker-4B †Neutral documented reranker adapter; instruction and no-instruction public calibration were run, then frozen before one held-out pass. | 43.5 | 44.6 | 65.2 | 78.7 | 49.2 | 68.0% | 29.9% | +38.1 pp | $0.050 | 0.13 s | RunPod GPU |
| 21 | decider-35b-a3b †The author's TypeSafe-compatible server and published FP8 weights, run serially on our H100 NVL. The exhaustive startup batch warmup was skipped; each required serial shape captured lazily before its measured request. Self-host latency receives the standard ×2 + 0.15 s adjustment. Cost uses the closest hosted 35B-A3B input tariff and is not the temporary rental charge. | 41.2 | 47.2 | 65.3 | 80.8 | 45.3 | 83.1% | 31.5% | +51.6 pp | est.$0.067 | 0.29 s | RunPod GPU |
| 22 | Raw Qwen3 4B Instruct 2507 direct logits †Neutral raw-logit control. | 41.0 | 46.4 | 29.1 | 87.6 | 59.7 | 69.7% | 27.3% | +42.4 pp | est.$0.022 | 0.08 s | RunPod GPU |
| 23 | OpenSourceJev †DM submission, measure-only (JevBench publishing HOLD in force). Same author as the existing simplejev-qwen3.5-0.8b row (sabeel111/Featherless AI). Round-4 audit of an earlier commit could not be measured (no public GGUF, Windows-only DLL loader, unpinned llama.cpp build); this round the author published the exact unsloth Q4_K_M GGUF (hash/size independently verified) and we built llama.cpp CUDA from current upstream master on Linux ourselves -- its ABI matched the ctypes bindings exactly, so only a loader file-naming fix was needed (documented diff), no code/scoring/calibration change. Our public-231 subset exactly reproduced the author-reported table: easy 48/48, standard 67/72, hard 66/111, schema 231/231. Calibration (noul temperature) fit only on Google BoolQ, not JevBench. Repo docs name 3 public task IDs while describing benchmark-directed algorithm fixes on the public half (disclosed); 0 exact state/instruction text matches in a released-file scan. | 40.9 | 41.8 | 60.3 | 82.0 | 64.0 | 78.4% | 26.3% | +52.1 pp | est.$0.016 | — | unknown |
| 24 | ZeroEntropy zerank-2 †Neutral documented reranker adapter; instruction and no-instruction public calibration were run, then frozen before one held-out pass. | 40.2 | 42.1 | 75.8 | 79.0 | 49.8 | 70.1% | 28.6% | +41.6 pp | $0.047 | 0.13 s | RunPod GPU |
| 25 | decision-machine-1 †A closed-weights decision model behind a production API that serves TypeSafe's wire format, so the unchanged typesafe adapter ran it. Run on a free test key (30 requests a minute, 2.2 s between requests); the provider states the inference infrastructure is the same as for paid keys. Cost is the public paid tariff, $0.04 per million input tokens (output free), times the input tokens the API reported. | 39.9 | 41.3 | 68.3 | 92.9 | 53.7 | 67.5% | 25.6% | +41.9 pp | $0.035 | 0.17 s | API |
| 26 | Malkuth-2B †Dhtocks/malkuth-2b rev 401304b989451876070d83c488271b4f927d03ab (rank-16 LoRA + pointer head over empero-ai/Qwen3.8-2B-Distill rev e37a2dc4acc68ad75a91e07e63168cb04cc06345, fitted T=1.1755), served by kev.serve from jaredpalmer/kev 557598fced1dada75dfbf36ed144dce309ac6ceb. Offline self-hosted inference of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) in a network-disabled, read-only container on an evaluator-owned Lium RTX PRO 6000 pod, through JevBench's unchanged typesafe adapter against the author's own server on loopback; no operator endpoint; no golds were exposed. Latency is the serial request wall time on the standard+judge items with the self-hosted adjustment. Cost is a labelled estimate: DeepInfra Qwen/Qwen3.5-4B list price $0.03/M input (no hosted ~2B listed; the 4B price errs high, the decider-2b precedent), read 2026-09-24, over the server's ownusage.input_tokens; it is not a GPU bill. | 38.9 | 41.3 | 53.5 | 91.4 | 61.5 | 69.7% | 24.4% | +45.3 pp | est.$0.019 | 0.04 s | RunPod GPU |
| 27 | Raw Phi-4 mini direct logits †Neutral raw-logit control, not JevBench-directed. | 38.0 | 41.8 | 58.8 | 88.8 | 49.6 | 65.8% | 29.2% | +36.6 pp | est.$0.048 | 0.06 s | RunPod GPU |
| 28 | JEV Qwen3.5-9B Base NVFP4 †Byte-identical to upstream March-2026 NVFP4 checkpoint, predates JevBench v1.2, no task-specific training added. Had 3 preliminary scope/policy scoring errors corrected during review (pooled-ECE, full-run latency, cost estimate); score above is final corrected value. | 37.7 | 46.8 | 67.7 | 93.3 | 43.3 | 75.3% | 29.5% | +45.8 pp | est.$0.077 | 0.02 s | RunPod GPU |
| 29 | 36.9 | 45.4 | 55.0 | 83.2 | 45.5 | 81.8% | 28.6% | +53.2 pp | est.$0.066 | 0.24 s | RunPod GPU | |
| 30 | kev 4B †Self-hosted from the author's repository at commit 20fa626 through its native TypeSafe-compatible `/v1/systemone` server, BF16 on an RTX 3090; measured serially from Sandy over the internet. The author labels this checkpoint a research preview. 306/308 sealed items answered validly (failures count as wrong) | 36.1 | 42.1 | 39.6 | 75.7 | 61.8 | 66.2% | 22.4% | +43.8 pp | est.$0.019 | 0.55 s | RunPod GPU |
| 31 | Decision 2B †Flymy-ai/decision-2b-preview revision df57b75db927acc9ad91ec8115508c1e487086eb (checkpoint minicpm5_reduced_v16_4k_v59), base openbmb/MiniCPM5-2B revision 12a3808a956f869c767195e9266b59c4d21d92e2, the submitter's own FlyMyJevPackageAdapter and frozen calibrator, bf16, unmerged adapter, 4096-token packing, transformers 4.57.6 / peft 0.15.2 as pinned, torch 2.8.0 from the pod image, on our RunPod L40 in Czechia Offline local open-weight inference on the frozen 308-item v1.4 set in a network-disabled, read-only container; no operator endpoint received sealed text. | 35.8 | 38.8 | 74.1 | 84.3 | 62.5 | 75.3% | 26.0% | +49.4 pp | est.$0.018 | 0.19 s | RunPod GPU |
| 32 | Qwen3.5-9B Jev-like data-mix v2 †The author disclosed development on the public JevBench set and public-result comparisons; released-data overlap scan found no matches, but 764 gap and 382 replay training rows are unreleased. | 35.2 | 47.4 | 61.3 | 82.0 | 42.4 | 78.4% | 29.2% | +49.1 pp | est.$0.083 | — | unknown |
| 33 | 35.1 | 95.8 | 92.0 | 73.7 | 36.9 | 99.1% | 92.9% | +6.3 pp | $0.127 | 1.44 s | API | |
| 34 | SimpleJev Qwen3.8-27B †Author's no-login shared demo, model id recorded verbatim, one request at a time at or below its 2 RPS limit. SimpleJev reads answer-token logits and returns the complete distribution; it does not generate an answer. Speed uses the public-demo x2 load adjustment; cost uses a hosted size-class input price and is not free/100. | 34.6 | 51.6 | 74.5 | 71.2 | 39.5 | 86.6% | 35.7% | +50.9 pp | est.$0.104 | 1.01 s | author demo |
| 35 | NInfer Qwen3.8-Flash-Next mixed †Engine scan (1,758 files) vs 231 public tasks: 0 exact matches. Submitter discloses no training for Flash-Next but repeated consultation of public items and a public-hard temperature sweep. | 34.0 | 49.5 | 78.6 | 88.2 | 38.9 | 89.6% | 34.1% | +55.5 pp | est.$0.109 | 0.08 s | RunPod GPU |
| 36 | swanOne †Draft vocabulary derived via AGPL-3.0 generator (provenance recorded separately). Submitter consulted all 231 public tasks and swept temperature over 111 public hard tasks. Score corrected during review from pooled-534 ECE/latency to hard-tier/242-item block. blockbrain-ai/swanone-recipe abcca2e789316472e58d4e824b61c04f419c3ba5: Mia-AiLab/Qwen3.8-Flash-Next-NVFP4 rev 925d7be6c14c6c9442ef83e8f05b5a3c39304f69 on vllm/vllm-openai:qwen38-flash-next@sha256:0aea3024… with MiaAI Lab's nine patched vLLM files (hash-verified) and the author's shim (option letters in the benchmark's order, the model's own renormalised letter mass, no temperature), H100.md serve command, here on an RTX PRO 6000 Blackwell. The shim sends its own system prompt; structured state is rendered with json indent=1. Offline self-hosted inference of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) in a network-disabled, read-only container on an evaluator-owned Lium RTX PRO 6000 Blackwell 96 GB pod, through JevBench's unchanged typesafe adapter against the author's own server on loopback; no operator endpoint; no golds were exposed. Latency is the serial request wall time on the standard+judge items with the self-hosted adjustment. Cost is a labelled estimate: OpenRouter qwen/qwen3.8-flash list price $0.15/M input (same underlying Flash-Next weights; the NInfer Flash-Next precedent), read 2026-09-24, over the server's ownusage.input_tokens; it is not a GPU bill. | 33.6 | 52.6 | 71.0 | 82.5 | 38.6 | 88.7% | 38.6% | +50.1 pp | est.$0.111 | 0.28 s | RunPod GPU |
| 37 | 33.3 | 97.4 | 93.5 | 72.6 | 36.0 | 99.6% | 95.5% | +4.1 pp | $0.135 | 1.48 s | API | |
| 38 | open-alternative-jev †With the options in reverse order (A. no, B. yes) the same model scored 21 % instead of 72 % on yes/no answer-judging items — small models are very sensitive to option order. | 33.2 | 38.6 | 58.7 | 83.5 | 59.6 | 74.0% | 24.4% | +49.7 pp | est.$0.022 | 0.21 s | RunPod GPU |
| 39 | jev-local †The author's local Jev-compatible server in its default full configuration: a frozen Qwen3.5-9B scores each option by its mean log-probability (one forward pass per option, no generation, no decision training). Run serially on our GPU. It re-reads the state once per option; if its reported token count covers one pass only, a per-token hosted price would be higher than this estimate. | 32.5 | 45.2 | 64.2 | 69.2 | 43.3 | 74.9% | 29.5% | +45.3 pp | est.$0.077 | 1.05 s | RunPod GPU |
| 40 | Decision Fast †Flymy-ai/decision-fast-preview revision 4225d41c66119fe28e95a2631bb0103decae6d56 (checkpoint qwen3_06b_headfirst_ep2a_v53), base Qwen/Qwen3-0.6B-Base revision da87bfb608c14b7cf20ba1ce41287e8de496c0cd, the submitter's own FlyMyJevPackageAdapter and frozen calibrator, bf16, unmerged adapter, 4096-token packing, transformers 4.57.6 / peft 0.15.2 as pinned, torch 2.8.0 from the pod image, on our RunPod L40 in Czechia Offline local open-weight inference on the frozen 308-item v1.4 set in a network-disabled, read-only container; no operator endpoint received sealed text. | 32.5 | 37.1 | 65.3 | 81.6 | 76.1 | 63.2% | 25.6% | +37.6 pp | est.$0.0063 | 0.24 s | RunPod GPU |
| 41 | decider-2b †The author's TypeSafe-compatible server and published weights (Qwen3.5-2B-Base with a trained one-pass decision readout), run serially on our GPU. Self-host latency gets the standard ×2 + 0.15 s adjustment. | 30.7 | 38.5 | 43.5 | 83.2 | 61.0 | 71.0% | 24.7% | +46.3 pp | est.$0.020 | 0.26 s | RunPod GPU |
| 42 | 30.6 | 36.8 | 67.9 | 63.5 | 76.6 | 62.8% | 33.1% | +29.7 pp | est.$0.0060 | 0.94 s | CPU | |
| 43 | 30.3 | 36.1 | 63.7 | 71.1 | 86.2 | 58.4% | 30.8% | +27.6 pp | est.$0.0029 | 0.79 s | CPU | |
| 44 | Autoloops – Gemma 4 31B IT †Gemma 4 31B IT served through the Autoloops systemone API. Cost = the API's own token usage x Autoloops' published rates ($0.20/M input, $0.45/M output; no output tokens were used). API measurement: public and sealed item content reached the operator endpoint; no gold labels or answers were sent. | 30.0 | 59.8 | 79.2 | 84.0 | 36.0 | 92.8% | 45.8% | +47.1 pp | $0.136 | 0.60 s | API |
| 45 | Standard One 8B †StandardThinking/StandardOne-8B rev 0f14d009a9400e55ea5a00a89b4d859882db704e (Ministral-3-8B-Instruct-2512 + LoRA, merged) on stock SGLang 0.5.20 behind the author's jev-adapter from server/ at the same revision, nominated configuration: --prompt-wording native --native-system-prompt none --default-temperature 1.35, context 8192. Disclosed by the author: benchmark-directed development; the native wording was selected by an ablation on the public hard tier; 181 of 359,497 training rows reuse one generic 58-character public-hard instruction. Offline self-hosted inference of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) in a network-disabled, read-only container on an evaluator-owned Lium RTX PRO 6000 Blackwell 96 GB pod, through JevBench's unchanged typesafe adapter against the author's own server on loopback; no operator endpoint; no golds were exposed. Latency is the serial request wall time on the standard+judge items with the self-hosted adjustment. Cost is a labelled estimate: OpenRouter mistralai/ministral-8b-2512 list price $0.15/M input (= the author's proposed Mistral API price for the exact base), read 2026-09-24, over the server's ownusage.input_tokens; it is not a GPU bill. | 29.1 | 46.2 | 65.9 | 92.0 | 39.4 | 76.6% | 26.6% | +50.0 pp | est.$0.104 | 0.02 s | RunPod GPU |
| 46 | lev-350m †Weights franckverrot/lev-350m revision ab08ad8b8f346994d983152917e114224f6adac7, code github.com/franckverrot/lev c48a945dbf629998d7458dcc5c16f58df964db94, the author's own lev.serve /v1/systemone endpoint with its shipped calibration temperature, base LiquidAI/LFM2.5-350M, on our RunPod L40 in Czechia Offline local open-weight inference on the frozen 308-item v1.4 set in a network-disabled, read-only container; no operator endpoint received sealed text. | 28.5 | 34.8 | 70.6 | 85.3 | 76.1 | 58.4% | 25.0% | +33.4 pp | est.$0.0063 | 0.17 s | RunPod GPU |
| 47 | 27.7 | 49.4 | 69.2 | 77.1 | 36.5 | 85.3% | 33.1% | +52.2 pp | est.$0.131 | 0.68 s | author demo | |
| 48 | 27.5 | 34.5 | 75.7 | 70.5 | 77.8 | 57.1% | 27.9% | +29.2 pp | est.$0.0055 | — | unknown | |
| 49 | NInfer Qwen3.8-27B NVFP4 †T=1.5 is an offline recomputation from raw logits of the same run, not a second execution. T=1.5 was chosen by sweeping public hard items (development-set calibrated, disclosed). | 26.9 | 51.5 | 76.0 | 80.1 | 35.2 | 83.1% | 33.1% | +50.0 pp | est.$0.145 | 0.37 s | RunPod GPU |
| 50 | NInfer Qwen3.8-27B NVFP4 †Raw T=1.0 row. Author discloses repeated public-item consultation and public-hard tuning (applies to both NInfer 27B rows). | 26.3 | 51.5 | 67.2 | 80.1 | 35.2 | 83.1% | 33.1% | +50.0 pp | est.$0.145 | 0.37 s | RunPod GPU |
| 51 | 25.6 | 41.8 | 40.2 | 74.9 | 44.0 | 71.4% | 21.8% | +49.7 pp | est.$0.073 | 0.59 s | RunPod GPU | |
| 52 | 25.5 | 47.6 | 77.6 | 88.5 | 35.9 | 89.6% | 33.8% | +55.8 pp | est.$0.137 | 0.09 s | RunPod GPU | |
| 53 | typecastlm †Typecastlm[server] 1.1.2 (wheel identical to git e95911e), checkpoint mihailgribov/typecastlm-qwen3.5-3.8b rev dcfecfdc28e44ef82631901628535b71574e27af: frozen Qwen/Qwen3.5-4B blocks 0-27 plus a computed 39-row head, one forward pass, four calibration temperatures by question type; transformers 5.3.0, flash-linear-attention 0.5.2. States over 32,768 tokens are folded in the middle, not refused. The server reads a noul question's meaning from the order of its two criteria (first = true), not from their keys. Offline self-hosted inference of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) in a network-disabled, read-only container on an evaluator-owned Lium RTX PRO 6000 Blackwell 96 GB pod, through JevBench's unchanged typesafe adapter against the author's own server on loopback; no operator endpoint; no golds were exposed. Latency is the serial request wall time on the standard+judge items with the self-hosted adjustment. Cost is a labelled estimate: DeepInfra Qwen/Qwen3.5-4B list price $0.03/M input (the exact base weights; one pass, no output), read 2026-09-24, over the server's ownusage.input_tokens; it is not a GPU bill. | 25.3 | 34.0 | 62.1 | 92.6 | 60.2 | 77.9% | 29.9% | +48.1 pp | est.$0.021 | 0.03 s | RunPod GPU |
| 54 | SimpleJev Qwen3.6-35B-A3B †Author's no-login shared demo, model id recorded verbatim, one request at a time at or below its 2 RPS limit. SimpleJev reads answer-token logits and returns the complete distribution; it does not generate an answer. Speed uses the public-demo x2 load adjustment; cost uses a hosted size-class input price and is not free/100. | 24.9 | 45.7 | 59.8 | 75.0 | 38.1 | 81.4% | 28.2% | +53.1 pp | est.$0.116 | 0.85 s | author demo |
| 55 | kev 0.6B †Self-hosted from the author's repository at commit 20fa626 through its native TypeSafe-compatible `/v1/systemone` server, BF16 on an RTX 3090; measured serially from Sandy over the internet. The author labels this checkpoint a research preview. 307/308 sealed items answered validly (failures count as wrong) | 24.8 | 34.2 | 50.0 | 75.6 | 76.1 | 66.7% | 24.0% | +42.6 pp | est.$0.0063 | 0.59 s | RunPod GPU |
| 56 | Raw Qwen3 8B direct logits †Neutral raw-logit control. | 23.7 | 45.7 | 24.1 | 86.3 | 41.9 | 68.4% | 26.3% | +42.1 pp | est.$0.087 | 0.08 s | RunPod GPU |
| 57 | system-oneby Sean Goedecke · Qwen3-8B, Sean Goedecke · details | 23.4 | 43.6 | 32.8 | 84.4 | 41.5 | 71.9% | 24.4% | +47.5 pp | est.$0.089 | 0.17 s | RunPod GPU |
| 58 | OpenDecision †A zero-shot NLI classifier behind a TypeSafe-compatible server, not a trained decision model: it scores each option as an entailment hypothesis with ModernBERT-large-zeroshot-v2.0. Its choice path runs several NLI passes over the same state, which the reported token count does not include, so a per-token hosted price would be higher than the estimate here. Pre-registered for our CPU in v1.2.7, run on our GPU because the CPU was far too slow. | 21.6 | 31.8 | 57.1 | 79.9 | 75.3 | 53.2% | 25.6% | +27.6 pp | est.$0.0066 | 0.34 s | RunPod GPU |
| 59 | LitJev †The author's reproduction of Jev's decision layer on an off-the-shelf model, in its default configuration: Qwen3.8-27B, scores read from the output head, no training and no calibration file (its README says probabilities are not calibrated by default). Run serially on our GPU through an SSH tunnel, because its server binds to localhost; the request still crosses the internet and gets the ×2 + 0.15 s adjustment. | 19.5 | 46.3 | 76.6 | 66.7 | 33.6 | 86.1% | 30.8% | +55.3 pp | est.$0.163 | 2.03 s | RunPod GPU |
| 60 | openJev Verdict 1.4 †Same public weights as the earlier Verdict row, run through the author's fixed v1.4 engine. That engine auto-loads the calibrator for every option count, frames candidate labels as NLI sentences and uses a 512-token context budget. Run locally on our CPU, serially. | 19.0 | 29.4 | 72.0 | 78.1 | 82.4 | 57.6% | 27.9% | +29.7 pp | est.$0.0039 | 0.31 s | CPU |
| 61 | kev 0.5B †Self-hosted from the author's repository at commit 20fa626 through its native TypeSafe-compatible `/v1/systemone` server, BF16 on an RTX 3090; measured serially from Sandy over the internet. This is the v0.1 release. 307/308 sealed items answered validly (failures count as wrong) | 18.9 | 30.5 | 49.7 | 77.0 | 76.1 | 49.4% | 27.3% | +22.1 pp | est.$0.0063 | 0.43 s | RunPod GPU |
| 62 | Bespoke Nimble 9B †Re-run in v1.2.8 at Bespoke Labs' request after they raised the serving prompt limit from 2,048 to 8,192 tokens (bespokelabsai/nimble PR #4). Same recipe as the v1.1.3 run — the published LoRA merged into Qwen3.5-9B with the author's PEFT safe-merge, served with SGLang and the author's Jev-compatible API — now from current nimble main; the adapter weights are unchanged. Hard-tier accuracy rose from 43.6 % to 65.5 %, yet the score fell: the long hard items that used to fail at once are now answered and priced (so Cost fell), and this pod was in Canada while the v1.1.3 run's was in Sweden, so part of the lower Speed is network distance from our server in Germany. This complete run replaces the earlier row; its old score is kept in the artifact under superseded_rows. | 18.7 | 46.3 | 56.4 | 78.7 | 33.4 | 79.7% | 28.9% | +50.8 pp | est.$0.166 | 0.39 s | RunPod GPU |
| 63 | 18.5 | 93.1 | 87.4 | 77.5 | 28.5 | 97.4% | 89.0% | +8.4 pp | $0.242 | 0.97 s | API | |
| 64 | openJev Verdict †The openJev-verdict-2.0 Hugging Face repo ships no weights; its config is byte-identical to heman10x/rlcd-modernbert-151m, whose published weights we ran with the author's engine. The 'verdict2-base' checkpoint behind the README's numbers is not downloadable yet (Git LFS 404); we will run it once it is. | 18.1 | 30.0 | 47.0 | 76.7 | 83.1 | 55.4% | 24.7% | +30.7 pp | est.$0.0037 | 0.28 s | CPU |
| 65 | Raw Qwen3 1.7B direct logits †Neutral raw-logit control. | 18.1 | 33.2 | 24.3 | 89.7 | 64.9 | 54.1% | 26.0% | +28.1 pp | est.$0.015 | 0.07 s | RunPod GPU |
| 66 | reflex-27b †The frozen public Qwen3.8-27B checkpoint through reflex at the requested pinned commit, with two option orders averaged and temperature 1. No adapter or fitted calibration file. Run serially on our H100 NVL. Self-host latency receives the standard ×2 + 0.15 s adjustment; cost uses the exact base model's public hosted input tariff. | 17.8 | 46.4 | 77.2 | 67.5 | 32.3 | 87.0% | 29.5% | +57.5 pp | est.$0.181 | 1.89 s | RunPod GPU |
| 67 | JevAct †Requested in GitHub issue #66. The author published an endpoint and an evaluation client, not weights, and his API is not the TypeSafe wire format, so JevBench's own `jevact_api` adapter implements exactly the mapping table in his eval_code.zip: state to state, instructions to the question, the labels with their criteria text as the options in label order, noul as false/true, and results[0].options[i].probability read back as the probability of labels[i] by index. His server answers HTTP 400 with `all inference items exceeded max_context_tokens or contained reserved markers` for an input it cannot take, and his own client and test script record exactly that as one failed item and continue; the adapter therefore surfaces that one documented refusal as the harness's 422 refusal path, which scores it as a wrong answer and does not count toward the stop rule - the same treatment the swanOne and Cygnet packages get for their own 422. Any other 400 or an outage still stops the run. The endpoint reports no token accounting, so cost is the labelled 2B size-class estimate. The endpoint is one machine in China and the latency includes that distance; it is not a production API and gets the standard x2 self-host adjustment, without the +0.15 s that only our own servers carry. 237/308 sealed items answered validly (failures count as wrong) | 16.9 | 29.3 | 55.1 | 76.5 | 64.3 | 61.9% | 23.4% | +38.5 pp | est.$0.015 | 0.40 s | author demo |
| 68 | djev †Experimental full-generation path over the same DiffusionGemma checkpoint as djev-dev: thinking was enabled and the model could generate up to 8,192 tokens before returning its distribution. Current djev-dev itself hard-codes enable_thinking=false, diffusion_max_steps=1 and read_only=true, so this is not a switch in its published typed API. It is substantially slower/costlier, and 72/534 requests exhausted the output budget without a parseable distribution; those are failures. Cost uses measured tokens and a same-size hosted reference, not the H200 rental bill. 220/308 sealed items answered validly (failures count as wrong) | 15.2 | 71.6 | 87.8 | 75.2 | 26.9 | 87.4% | 60.1% | +27.4 pp | est.$0.274 | 0.43 s | RunPod GPU |
| 69 | GLiNER2 large †The large checkpoint of Fastino's earlier GLiNER2 family, same documented mapping as the GLiNER2 row: the question goes in front of the text and the probabilities are the model's own single-label softmax over the labels, read out in full. A general schema classifier, not a Jev rebuild. | 15.1 | 31.1 | 24.8 | 61.7 | 73.3 | 56.7% | 28.6% | +28.1 pp | est.$0.0077 | 1.10 s | CPU |
| 70 | OpenJev †OpenJev's real typed-API thinking switch at think=512, using its own /v1/systemone server over BF16 DiffusionGemma. The thought is generated first, then native probability reads are taken after it. All 534 requests returned valid distributions. Cost counts the server's billed input and thought output tokens. | 14.8 | 58.1 | 58.1 | 76.1 | 27.8 | 88.7% | 42.2% | +46.5 pp | est.$0.255 | 0.46 s | RunPod GPU |
| 71 | Qwen3.5-0.8B Decision Model †JevLite SystemOne on CPU; bundled per-question calibration; no operator endpoint or network access. Local CPU measurement on the existing 534-decision v1.3 set plus 308 sealed v1.4 decisions; no operator endpoint received sealed text. | 14.5 | 28.1 | 68.2 | 49.2 | 75.7 | 59.3% | 34.7% | +24.6 pp | est.$0.0065 | 7.15 s | CPU |
| 72 | Gemini 3.1 Flash-Lite †307/308 sealed items answered validly (failures count as wrong) | 14.3 | 54.5 | 59.3 | 81.8 | 27.4 | 87.0% | 38.6% | +48.4 pp | $0.264 | 0.76 s | API |
| 73 | open-jev-deberta-v3-large †297/308 sealed items answered validly (failures count as wrong) | 12.6 | 25.6 | 66.6 | 66.0 | 74.0 | 52.4% | 29.5% | +22.8 pp | est.$0.0073 | 1.77 s | CPU |
| 74 | smalljev semantic-v9 †The public semantic-v9 LoRA and native heads over MiniCPM5-2B-Base, through the mapping frozen before the run. It has a typed Python contract but no TypeSafe-compatible HTTP route. The released training recipe explicitly hill-climbed against JevBench's public shape and source families; this allowed public benchmark-directed development is disclosed. Cost is $0.04/M measured input tokens, not free/100. | 12.3 | 25.7 | 59.2 | 79.8 | 57.9 | 60.6% | 26.9% | +33.7 pp | est.$0.025 | 0.41 s | RunPod GPU |
| 75 | 11.8 | 27.4 | 25.2 | 71.8 | 83.1 | 58.0% | 29.2% | +28.8 pp | est.$0.0037 | 0.31 s | CPU | |
| 76 | Instinct †Requested in GitHub issue #69. A free evaluation/demo endpoint that speaks TypeSafe's /v1/systemone wire format, so the unchanged typesafe adapter ran it and no mapping of ours was involved; we registered the account and created the key ourselves in their console. The author states frozen Qwen3.8-27B base weights with no fine-tuning, read in one forward pass per question with no autoregressive decoding; the serving stack is not public, so nothing about it could be reviewed and the row rests on the API's own behaviour. Cost is an ESTIMATE: ZooWork publishes no bookable price, so the base model's public reference price is used (OpenRouter model-level qwen/qwen3.8-27b, $0.42/M input, zero output for a direct-logit readout); the author-announced tariff is not used. Classified as a free evaluation/demo endpoint (no SLA, no status page, no terms, pricing 'to be announced'), so the x2 latency adjustment applies. Re-scored when a bookable price is published. | 11.4 | 51.1 | 78.7 | 83.9 | 24.6 | 86.6% | 35.1% | +51.5 pp | est.$0.325 | 0.26 s | author demo |
| 77 | Open-Jev 9B †The author's pinned LoRA adapter, trained scalar decision head and calibration temperature, served by the author's Open-Jev server with prefix caching off, batch size 1 and 4,096-token limit. Serial requests were measured from Sandy over an SSH tunnel to the H100. Self-host latency receives the standing x2 + 0.15 s adjustment. Cost uses the exact Qwen3.5-9B hosted input tariff for 9B and the same conservative same-family proxy for the unlisted 2B; neither receives an automatic 100. Exact normalized comparison found no JevBench public task state or instruction in the 79,116-row public training projection. | 11.2 | 44.2 | 61.8 | 72.0 | 28.1 | 77.5% | 29.9% | +47.6 pp | est.$0.249 | 0.75 s | RunPod GPU |
| 78 | Open-Jev 2B †The author's pinned LoRA adapter, trained scalar decision head and calibration temperature, served by the author's Open-Jev server with prefix caching off, batch size 1 and 4,096-token limit. Serial requests were measured from Sandy over an SSH tunnel to the H100. Self-host latency receives the standing x2 + 0.15 s adjustment. Cost uses the exact Qwen3.5-9B hosted input tariff for 9B and the same conservative same-family proxy for the unlisted 2B; neither receives an automatic 100. Exact normalized comparison found no JevBench public task state or instruction in the 79,116-row public training projection. | 10.0 | 42.3 | 55.3 | 73.5 | 28.1 | 64.5% | 26.3% | +38.2 pp | est.$0.249 | 0.66 s | RunPod GPU |
| 79 | 9.8 | 23.1 | 57.2 | 67.8 | 82.4 | 48.9% | 32.8% | +16.1 pp | est.$0.0039 | 0.43 s | CPU | |
| 80 | CLM-8B †Contrastive-LM/CLM commit cca045ffdb07b3ebcfe6938537cdeac5e14899c9, head Contrastive-LM/CLM-v0.1-8B rev 87655cb835bd76fd66c2da78e1e3709f7fa11a94 (clm-latest), Qwen3-8B rev b968826d9c46dd6066d109eabc6255188de91218 last-token pooling via vLLM. Authors' documented system_one path: state + instructions as state text, each option description as a candidate action, softmax over contrastive scores at temperature 1.0. Offline local open-weight inference on the frozen 308-item v1.4 set in a network-disabled, read-only container on an evaluator-owned Lium RTX PRO 6000 pod; no operator endpoint; no golds were exposed. Latency is in-process Engine.answer time on the serial standard+judge items with the self-hosted adjustment. Cost is estimated at the Qwen3-Embedding-8B hosted list price ($0.01/M input, same-size 8B pooling encoder) over CLM's measured encoder tokens; it is not a GPU bill. | 8.6 | 22.4 | 39.8 | 93.6 | 78.4 | 40.7% | 24.0% | +16.7 pp | est.$0.0052 | 0.02 s | RunPod GPU |
| 81 | 7.5 | 21.5 | 49.1 | 57.5 | 68.3 | 54.5% | 34.7% | +19.8 pp | est.$0.011 | 3.99 s | CPU | |
| 82 | GLiNER2.5 small †The small GLiNER2.5 checkpoint (74M), same family and same documented mapping as the GLiNER2 row: the question goes in front of the text and the probabilities are the model's own single-label softmax over the labels, read out in full. A general schema classifier, not a Jev rebuild. | 7.2 | 20.5 | 50.7 | 77.8 | 82.4 | 45.9% | 28.6% | +17.3 pp | est.$0.0039 | 0.11 s | CPU |
| 83 | Raw Qwen3 0.6B direct logits †Neutral raw-logit control. | 7.1 | 22.8 | 20.7 | 89.9 | 73.9 | 48.1% | 25.6% | +22.4 pp | est.$0.0074 | 0.07 s | RunPod GPU |
| 84 | verdict-small †Requested in GitHub issue #73. Run through the author's own `verdict serve` on our CPU, which speaks TypeSafe's /v1/systemone wire format, so JevBench's unchanged typesafe adapter ran it and no mapping of ours was involved. A 118M multilingual bi-encoder: every option is scored against the rendered state by cosine similarity, at the model's own scale (temperature 1.0, no calibrator fitted on JevBench items, as the author states). Structured state is rendered as `key: value` lines by his own code. Code review before the run: the only network call is the Hugging Face download of his own checkpoint, no telemetry, no key, no rule written against public items. The `usage.input_tokens` his server reports is a word count and not a tokeniser count, so the cost is the labelled size-class estimate rather than a measured token price. Self-host latency gets the standard x2 + 0.15 s adjustment. The author's own public-set figures were easy 0.938, standard 0.486, hard 0.396 on an Apple M5 CPU. Offline measurement of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) on our own CPU through the author's server; no operator endpoint. | 5.7 | 18.1 | 66.9 | 85.0 | 96.6 | 53.7% | 28.9% | +24.8 pp | est.$0.0013 | 0.05 s | CPU |
| 85 | DeepSeek V4.1 Flash †298/308 sealed items answered validly (failures count as wrong) | 4.8 | 94.0 | 95.5 | 71.6 | 16.8 | 97.8% | 94.8% | +3.0 pp | $0.594 | 1.42 s | API |
| 86 | 2.1 | 13.6 | 26.0 | 70.8 | 73.3 | 41.1% | 9.1% | +32.0 pp | est.$0.0077 | 0.90 s | author demo | |
| 87 | Mixedbread mxbai-rerank-base-v2 †Neutral documented reranker adapter; instruction and no-instruction public calibration were run, then frozen before one held-out pass. | 0.4 | 6.8 | 84.1 | 87.5 | 67.9 | 37.2% | 34.4% | +2.8 pp | $0.012 | 0.07 s | RunPod GPU |
| 88 | BAAI bge-reranker-v2-m3 †Neutral documented reranker adapter; instruction and no-instruction public calibration were run, then frozen before one held-out pass. | 0.2 | 5.0 | 84.2 | 89.5 | 73.4 | 39.4% | 27.9% | +11.5 pp | $0.0077 | 0.03 s | RunPod GPU |
| 89 | Alibaba GTE Reranker ModernBERT-base †Neutral documented reranker adapter; instruction and no-instruction public calibration were run, then frozen before one held-out pass. | 0.2 | 4.8 | 78.9 | 90.6 | 69.6 | 33.8% | 33.4% | +0.3 pp | $0.010 | 0.05 s | RunPod GPU |
| 90 | Certo v1 †The public Certo v1 checkpoint through the author's DecisionModel, serially on our rented GPU. The question instruction is prepended to the state because Certo exposes state + runtime options but no separate question field; the published 64-token state and 48-token option limits are unchanged. The model card says v1 does not yet transfer to arbitrary natural-language prose. Cost is an estimate from same-size hosted encoders times the checkpoint's retained input tokens, not free/100. | 0.0 | 0.1 | 83.0 | 94.0 | 100.0 | 31.6% | 29.5% | +2.1 pp | est.$0.0010 | 0.02 s | RunPod GPU |
| 91 | Open Jev JSON Canvas †Returns only a final label, not a probability distribution — calibration counts as 0 in the composite. Scan of 39 files: 0 matches. | 0.0 | 48.2 | 0.0 | 84.1 | 45.6 | 84.4% | 31.2% | +53.2 pp | est.$0.065 | 0.22 s | RunPod GPU |
| — | classifier.dev honorable mention · not ranked†Its own benchmark page says the fast tier is Jev. Free for us; the price is its published Pro plan ($20/month for 200,000 fast classifications a day) at full use, $0.0033 per 1,000 decisions. | 70.8 | 51.6 | 72.4 | 87.6 | 84.3 | 85.3% | 34.4% | +50.9 pp | est.$0.0033 | 0.39 s | API |
| — | Qwen3.8 27B partial · not ranked†69/308 sealed items answered validly (failures count as wrong); partial: Chutes rate limit stopped the run after 81/308 items; unranked as in v1.3 | 0.0 | 40.4 | 93.6 | 61.3 | 0.0 | 71.9% | 21.8% | +50.1 pp | est.$2.669 | 5.75 s | API |
| — | Needle 3, options as tools partial · not ranked†V1.4: Not re-measured: same as needle-3. | — | — | — | — | — | 22.1% | — | — | est.$0.014 | 3.78 s | CPU |
| — | Needle 3 partial · not ranked†V1.4: Not re-measured: ~100-250 s per item on a rented CPU pod (19 s on Sandy); needs a dedicated CPU host. | — | — | — | — | — | 22.5% | — | — | est.$0.024 | 1.69 s | CPU |
95 of 95 systems, sorted by official rank, #1 first.
API = the operator's endpoint received sealed item text during evaluation; the answers and item-level results are not published. The sealed text and answers remain private; only system-level aggregates appear here. Cost is per 1,000 decisions. Hover endpoint, cost and API labels for their recorded details.
usage.input_tokens; it is not a GPU bill.usage.input_tokens; it is not a GPU bill.usage.input_tokens; it is not a GPU bill.usage.input_tokens; it is not a GPU bill.usage.input_tokens; it is not a GPU bill.usage.input_tokens; it is not a GPU bill.usage.input_tokens; it is not a GPU bill.usage.input_tokens` his server reports is a word count and not a tokeniser count, so the cost is the labelled size-class estimate rather than a measured token price. Self-host latency gets the standard x2 + 0.15 s adjustment. The author's own public-set figures were easy 0.938, standard 0.486, hard 0.396 on an Apple M5 CPU. Offline measurement of all 842 decisions (534 frozen v1.2 + 308 sealed v1.4) on our own CPU through the author's server; no operator endpoint.Rows without a † have no note beyond the shared provenance: every row was measured or re-run with its recorded recipe, and deviations are in its run manifest.
Artifact: v1.4.2.2 results JSON · SHA-256 7f39b2f742a6… · JevBench v1.4.2.2 release and method
I = 0.8 × I_v1.3 + 0.2 × I_sealed, where I_sealed = 100 × max(0, (acc_sealed − 0.293) / (1 − 0.293)). Public and sealed scores are published only as aggregates.C = C_v1.3 + (C_v1.4 − C_v1.3) × min(1, 0.2 / 0.35).k = 1 generalization penalty reduces Intelligence when public accuracy exceeds sealed accuracy by more than 25 percentage points: I × (1 − max(0, gap − 25) / 100). It rewards systems that generalize beyond the public half.p = −1). Intelligence below 50 keeps its quadratic penalty; Speed and Cost each have a separate Jev-class gate below 50. Speed and Cost axis calculations are unchanged from v1.3.0.The chart and table above compare the tested systems, not marketing claims. These are the practical answers readers most often need before choosing a Jev-class decision model.
The highest-ranked open entrants in this run are Imajev-4B (#1, 67.4), Plumb-4B (#2, 65.8), Hopper (#7, 59.4), Winnow-12B Q8 (#8, 55.6). “Open” here means the tested row publishes code or weights; check the licence and exact configuration in the board before adopting one.
Open entrants with released code or weights can run on infrastructure you choose, including EU infrastructure. That can support data residency, but neither open source nor an EU server makes a deployment GDPR-compliant by itself. Assess your data, contracts, retention, subprocessors and security for the complete setup. See Benchmark Heaven's broader EU-hosting comparison.
jev-router.com offers self-hosted open decision models. Neutrality disclosure: it is run by the authors of this benchmark; it receives no scoring advantage and is not a ranked entrant.
The official score is the equal-weight harmonic mean of Intelligence, Calibration, Speed and Cost. Version v1.4.2.2 measures 534 public and 308 sealed decisions: 20% of Intelligence comes from the sealed set, a public-minus-sealed gap above 25 points costs Intelligence, and Intelligence, Speed or Cost below 50 each pull the score down quadratically. What changed in v1.4 · method and tiers.
Open an issue in the JevBench repository with a reproducible endpoint or runnable code, the exact model and licence, and whether public JevBench items were used during development. New entrants use the same frozen harness and appear in a new version. For private data, see the custom evaluation options.
Every price here is US dollars per 1,000 decisions — not per 1,000 tokens. One decision is a whole question — state, rubric and options — about 950 input tokens for Jev 1.13.0, so at its $0.042 per million input tokens 1,000 decisions cost $0.0399.
One decision is a whole question, not a token. Jev 1.13.0 reads 950 input tokens per decision on average over the 534 v1.2 decisions. At its public tariff of $0.042 per MILLION input tokens (output tokens are free, https://docs.typesafe.ai/models), 1,000 decisions therefore cost 950 x 1,000 x $0.042 / 1,000,000 = $0.0399. That is what the Cost column shows: $0.0399 per 1,000 decisions, not per 1,000 tokens.
Systems with a public tariff (per token or per request) are priced at that tariff times the tokens we measured. Systems without one — open weights, author demos, models we ran locally — are priced as if a large inference provider hosted them: the OpenRouter list price of the same weights; if OpenRouter does not list them, the nearest larger sibling; if no model of that size class is on OpenRouter, the DeepInfra list price of the same weights or of the nearest larger model of the same class. We do not use per-minute GPU rental or our own CPU time — providers buy capacity in bulk or own the hardware, and price accordingly. Price × tokens per decision = $ per 1,000 decisions, marked “est.”.
Price rules from v1.4.3 and v1.5 on (disclosed before any v1.5 result): only public, bookable list prices that have been in effect for at least 30 days count; promotions, subsidies, credits and free tiers do not. The scoring price is never below the market reference price of the system's base model, found the same way as the estimates above. A later price change triggers a re-score with a visible note on the row.
usage.input_tokens; nothing generated; estimated, not chargedusage.input_tokens; nothing generated; estimated, not chargedusage.input_tokens; nothing generated; estimated, not chargedusage.input_tokens; nothing generated; estimated, not chargedusage.input_tokens; nothing generated; estimated, not chargedusage.input_tokens; nothing generated; estimated, not chargedusage.input_tokens; nothing generated; estimated, not chargedusage.input_tokens (encoder tokens on cache misses, documented default action cache); estimated, not chargedSources: OpenRouter https://openrouter.ai/api/v1/models and DeepInfra https://api.deepinfra.com/models/list (both read 2026-09-19).
usd_per_1000_v11_tiers = 1000 x (mean input tokens per decision x $/M in + output tokens charged x $/M out) / 1e6, over all 314 v1.1 decisions (72 easy + 242 standard+judge), each decision counted exactly once and priced exactly once. A metered row uses the provider's own tariff and its own measured token counts, including the requests whose answer could not be parsed; an estimated row uses the reference tariff for its weights or size class and, when the run reports no usage, the input tokens of the gemini-3.1-flash-lite run on the same prompts over the same 314 decisions. usd_per_1000 = (v11 x 314 + hard x 220) / 534.
No tariff, measurement, item, answer or rank changed. The prices before and after:
An exclusion is an availability fact about our run — hardware, access, terms — never a quality verdict. Partial runs are in the table above, greyed and without a rank; so are the honorable mentions, which are complete runs that simply are not ranked.
Built and run by us, not collected from someone else's leaderboard; the results describe the tested configurations, not every application.
Capability and Jev-class (headline ranking). Capability = (Intelligence + Calibration) / 2. A system is Jev-class if its cost per decision is at most 2× Jev 1.13.0's and its median latency (the adjusted p50 — the median the speed chart plots, not the Speed-axis score) is at most 2× Jev 1.13.0's; rows without a recorded median latency use the Speed axis at the 2× equivalent (77.2). The headline ranks Jev-class systems by Capability; the others, including general-purpose LLMs, are listed below a divider. This is a way of presenting the same measurements; it changes no score and no official rank.
JevBench Score. Equal-weight harmonic mean of Intelligence, Calibration, Speed and Cost (power mean p=-1). If Intelligence <50 multiply by (I/50)^2. For Speed and Cost separately, if below 50 multiply by (axis/50)^2. The weight sliders re-score the same axes with other weights for exploration; only equal weights give the official score and rank.
Intelligence. 0.8 × v1.3 chance-corrected Intelligence on the frozen v1.2 items + 0.2 × 100 × max(0, (sealed accuracy − 0.293)/(1 − 0.293)); then multiply by 1 − max(0, public-minus-sealed accuracy gap in percentage points − 25)/100. Original tier weights: easy .14, standard .28, judge .28, hard .30.
Revision v1.4.2.2. v1.4.2.2 adds the verified Imajev-4B row to v1.4.2.1 using the exact live v1.4.2 scoring code. Earlier measurements and score fields are unchanged; only ranks and preset ranks move where the new row changes the ordering. A 28 Sep text-only amendment clarifies that full-forward input tokens are counted once for the single pinned server pass; no measurement, score, axis, rank, eligibility, or numeric value changed.
Every system sees the same state, instructions, rubric and exact label set; only the transport differs. Requests go out one at a time with no retries, so latency includes the network. Estimated costs are hosted-provider prices for the same weights or size class and are marked “est.” — hover one for its basis, or see how costs are estimated. Every system has a price; none gets a free 100.
A service running another entrant's model is listed, but not ranked against the models. Ranking it would rank the same model twice, once at the model's own price and once at the service's. The row keeps every number, axis, cost basis and per-task outcome; it carries no rank number. Which rows this applies to, and why: Honorable mentions — services built on another entrant's model.
v1.4 scores are not comparable with v1.3 or earlier (sealed blend, gap penalty and harmonic mean). The v1.3.0 board stays below as history; the v1.0 page keeps its own numbers, calibration plots and per-family tables.
Harness, public tasks and every scoring rule: github.com/fstandhartinger/jevbench (MIT). Each project links its author's repository or vendor page.
Authors: if we tested the wrong configuration, tell us and we will rerun it. New entrants become a new version rather than silently changing this one.
3D view: three.js r128 (MIT).
Loading capability views…
Loading context-length views…
Open this section to load the earlier public-only board and diagnostics.