JevBench · fresh regular draw

Refusals, errors and context-bound failures stay in the full 1,500 denominator. Every row kept its own package, model, runtime and context settings; nothing was re-run for presentation.

NOT DIRECTLY COMPARABLE WITH THE LIVE v1.6.1 BOARD. This release is a different draw with its own field median: G_med = 10.073244581339713, frozen from the eight v1.6.5 members (Clef-omni, Blink and modernbert-jev v2 do not move it), against 2.573387642438244 on the live field. G_med_flag_gt10 is TRUE, which has not been true in an earlier published JevBench artifact. Concretely: the gap penalty is max(0, 1 - max(0, (gap - G_med) - 8)/100), so this field's threshold is G_med + 8 = 18.073 and no row here is penalised for its public-vs-sealed gap, whereas at the live field's median four of the eleven would be. The difference was measured, not estimated, and is small: rsi-jev-v6.1-vl-27b Intelligence 61.564 would be 59.551, standard_one_8b_sh 34.465 would be 32.460, liquid-d1-3b 22.139 would be 21.949, vegaml-4b 16.689 would be 16.673, and wald-4b-v2-1 (gap 9.478, below the live threshold 10.573), blink-v0-3-26b-a4b-nvfp4 (gap 8.161), modernbert-jev-v2 (gap 2.433) and the other four (clef-omni, gutsy-0.8b-v0.3, liquid-d1-omni-600m, vegaml-0.8b) are unchanged. The direction is one-sided: this field flatters its own members.

Cost: no item is excluded from the cost basis on this draw. Every row is priced from its own measured usage block against the frozen 25 Sep 2026 snapshot, or from a documented per-1,000 override, so the paid draw's public-counter-overflow exclusions have no analogue here. The cost normalizer C0 is USD 0.001 per 1,000 decisions with 30 cost points per decade. Every price on these rows is a clearly-labelled ESTIMATE; none is a bill any of these systems issued.

gutsy-0.8b-v0.3: 29 of 1,500 items returned HTTP 500 "llama_decode returned 1" - the longest items, which llama.cpp cannot decode into the author's 8,192-token context. They are scored wrong, as the method requires. This is a crash class rather than a declared refusal. Its cost is input-only and marked *est.: the author's server emits no output-token field, and under the 9 Oct 2026 rule a missing output_tokens on a model with no generation path at all is treated as 0 output tokens.

standard_one_8b_sh and rsi-jev-v6.1-vl-27b: the same 29 longest items exceed each system's declared 32,768-token context and were refused with HTTP 422 rather than truncated; scored wrong per method. For RSI-Jev the refusal boundary was proven before the scored run: 13 of 13 boundary checks passed, including that a 32,768-token question is accepted and seen whole and a 32,769-token one returns 422 with nothing silently cut.

liquid-d1-omni-600m truncates states on the right at 16,384 tokens by design, and its published run is the second of two: a first run on a 24 GB card hit 29 CUDA out-of-memory errors, which was our under-provisioning and not model behaviour; it is retained and not scored. vegaml-0.8b and vegaml-4b truncate 29 states on the right at 73,728 tokens by design. The Vega checkpoint repository declares no licence field; the vegaml package is Apache-2.0.

v1.6.8 ADDS ONE ROW, modernbert-jev v2, to the ten of v1.6.7 on the same draw, under the same rule: the field median stays the frozen v1.6.5 value, so all ten earlier rows keep identical point scores, axes and capability; ranks and bootstrap confidence intervals are recomputed for the eleven-row roster, and raw_on_A of standard_one_8b_sh and the field-level diagnostics (G_med_api_basis_P_vs_A, the API-basis offsets) move again for the reason given below; none of them enters a score.

modernbert-jev-v2: a 150M ModernBERT-base NLI encoder, int8 (unsigned) ONNX, served by the author's own CLI at its own defaults on ONE CPU thread of the evaluator's shared Sandy host (AMD Ryzen 5 3600, Zen 2), not a dedicated machine; the 1-minute load average was logged every 60 s and ranged about 5.5 to 27.9 (median 10.2) during the run, so Speed is a lower bound for this model on an idle core. The run was interrupted once by the evaluator's session ending after 41 answered items and resumed with the unchanged runner's --resume, which keeps answered rows and sends only the rest; the server is deterministic (byte-identical answers on repeated requests). By the author's documented design (README: "windowing retrieval is future work"), a state is cut at 20,000 characters and read through at most six 300-word windows spread across it: the 29 longest states of this draw are answered from that sample rather than refused, so words between the windows are never seen. Probabilities are native (per-head temperatures from the shipped calibration.json). With the shipped Noul temperature of 2.5, every one of its 375 Noul answers lies between P(yes) 0.23 and 0.80, so none counts as decisive under the O1S Noul method, which puts every Noul tier of this row at its floor; this is the model as shipped, not a serving fault. Output tokens are a literal 0; cost is a labelled estimate at the base-size encoder rate.

v1.6.7 ADDED ONE ROW, Blink v0.3 26B-A4B, to the nine of v1.6.6 on the same draw, under the same rule: the field median stays the frozen v1.6.5 value, so all nine earlier rows keep identical point scores, axes and capability; ranks and bootstrap confidence intervals are recomputed for the ten-row roster, and raw_on_A of standard_one_8b_sh moves again for the reason given below. Blink's gap (8.161) is below the threshold.

blink-v0-3-26b-a4b-nvfp4: measured from the model card's own plain-vLLM 0.30.0 serve command (NVFP4, fp8 KV cache, 32,768-token context, T=1.3 option-letter readout through a loopback shim reproducing the card's decide() client) with ONE serving-only deviation: VLLM_BATCH_INVARIANT=1. With the command as published, four identical repeats of 50 public items differed by up to 0.126 in probability, failing the pre-registered repeatability gate (0.005); with batch-invariant kernels the repeats were byte-identical. This makes the run repeatable, not more correct: between serving settings the same item's probabilities differ by up to 0.39 and the argmax by 2 of 49 public items, so a different stack or card can score differently. Latency is of the batch-invariant path (about twice the default per request). Run on an RTX 5090 32 GB (no RTX PRO 6000 was free). 29 of 1,500 states (77,706 to 82,401 tokens) exceed the card's 32,768-token context and were refused with HTTP 422; scored wrong per method. Cost is a labelled estimate from the base model google/gemma-4-26b-a4b-it (USD 0.09 in / 0.30 out per 1M, frozen 25 Sep 2026 OpenRouter snapshot).

v1.6.6 ADDED ONE ROW, Clef-omni, to the eight of v1.6.5 on the same draw. The field median is held at the frozen v1.6.5 value (g_med_fixed, as for earlier same-draw additions), so all eight earlier rows keep identical point scores, axes and capability. Ranks and bootstrap confidence intervals are recomputed for the nine-row roster, as the v1.6.3 and v1.6.4 addenda did. One diagnostic also moves: raw_on_A (scores on the 300-item A subset) of standard_one_8b_sh, because the A-basis median is not part of the frozen value. Clef-omni's gap (5.555) is below the threshold.

clef-omni: measured as shipped on its own in-process entry point at the pinned revision, defaults unchanged. 29 of 1,500 items reach the model's own 64,000-token default and are answered on the truncated state, as the authors' encoder does by design; none was refused. The Omni video processor needs torchvision to load, which the model card's tested stack does not list; torchvision 0.26.0+cu130, the build matching its torch 2.11.0+cu130, was added and nothing else changed. Output tokens are a literal 0, so the cost is input-only.

wald-4b-v2-1: the shipped v2.1 server applies a documented Noul decisive floor (serving.json noul_decisive_floor 0.81): every calibrated P(yes) inside (0.19, 0.81) is moved to the nearer edge, on every Noul answer and not keyed to any item, so it answers all 375 Noul items decisively. Measured as shipped, server default effort none (one pass, 0 output tokens). It is the successor of the published wald-4b-v2 row on the paid draw and carries the same base-model cost basis.

Subject topics and TypeSafe use cases were labelled for this draw's 1,500 items in their own run (Winnow-12B Q8 on our own pod, egress cut before the sealed upload, the unchanged T1-T3/U1/U2 rules; no new rule was added). On 75 held-out public items checked by hand, topic agreement after the rules is 71/75 = 94.7 %. Known systematic miss: 17 of 27 router_policy items are filed under Coding by the subject of the request rather than under Rules, policy & law by the routing task; left as labelled because adding a rule after seeing the sample would change the method.