JevBench · fresh regular draw

Refusals, errors and context-bound failures stay in the full 1,500 denominator. Every row kept its own package, model, runtime and context settings; nothing was re-run for presentation.

NOT DIRECTLY COMPARABLE WITH THE LIVE v1.6.1 BOARD. This release is a different draw with its own field median: G_med = 10.073244581339713, frozen from the eight v1.6.5 members (Clef-omni, Blink, modernbert-jev v2 and the five v1.6.9 rows do not move it), against 2.573387642438244 on the live field. G_med_flag_gt10 is TRUE, which has not been true in an earlier published JevBench artifact. Concretely: the gap penalty is max(0, 1 - max(0, (gap - G_med) - 8)/100), so this field's threshold is G_med + 8 = 18.073 and no row here is penalised for its public-vs-sealed gap, whereas at the live field's median SIX of the sixteen would be - the two 4B rows this revision adds are among them. The difference was measured, not estimated, and is small: rsi-jev-v6.1-vl-27b Intelligence 61.564 would be 59.551, standard_one_8b_sh 34.465 would be 32.460, jiwo-4b 34.784 would be 33.997, classone-qwen3.5-4b 27.338 would be 25.413, liquid-d1-3b 22.139 would be 21.949, vegaml-4b 16.689 would be 16.673, and wald-4b-v2-1 (gap 9.478, below the live threshold 10.573), blink-v0-3-26b-a4b-nvfp4 (gap 8.161), modernbert-jev-v2 (gap 2.433), classone-gemma4-e4b (gap 10.263, also just below the live threshold), jiwo-0.8b (gap 9.322), classone-qwen3.5-2b (gap 2.157) and the other four (clef-omni, gutsy-0.8b-v0.3, liquid-d1-omni-600m, vegaml-0.8b) are unchanged. The direction is one-sided: this field flatters its own members.

Cost: no item is excluded from the cost basis on this draw. Every row is priced from its own measured usage block against the frozen 25 Sep 2026 snapshot, or from a documented per-1,000 override, so the paid draw's public-counter-overflow exclusions have no analogue here. The cost normalizer C0 is USD 0.001 per 1,000 decisions with 30 cost points per decade. Every price on these rows is a clearly-labelled ESTIMATE; none is a bill any of these systems issued.

gutsy-0.8b-v0.3: 29 of 1,500 items returned HTTP 500 "llama_decode returned 1" - the longest items, which llama.cpp cannot decode into the author's 8,192-token context. They are scored wrong, as the method requires. This is a crash class rather than a declared refusal. Its cost is input-only and marked *est.: the author's server emits no output-token field, and under the 9 Oct 2026 rule a missing output_tokens on a model with no generation path at all is treated as 0 output tokens.

standard_one_8b_sh and rsi-jev-v6.1-vl-27b: the same 29 longest items exceed each system's declared 32,768-token context and were refused with HTTP 422 rather than truncated; scored wrong per method. For RSI-Jev the refusal boundary was proven before the scored run: 13 of 13 boundary checks passed, including that a 32,768-token question is accepted and seen whole and a 32,769-token one returns 422 with nothing silently cut.

liquid-d1-omni-600m truncates states on the right at 16,384 tokens by design, and its published run is the second of two: a first run on a 24 GB card hit 29 CUDA out-of-memory errors, which was our under-provisioning and not model behaviour; it is retained and not scored. vegaml-0.8b and vegaml-4b truncate 29 states on the right at 73,728 tokens by design. The Vega checkpoint repository declares no licence field; the vegaml package is Apache-2.0.

v1.6.9 ADDS FIVE ROWS - jiwo 0.8b, jiwo 4b, ClassOne Qwen 3.5 2B, ClassOne Qwen 3.5 4B and ClassOne Gemma 4 E4B - to the eleven of v1.6.8 on the same draw, under the same rule: the field median stays the frozen v1.6.5 value, so all eleven earlier rows keep identical point scores, axes and capability; ranks and bootstrap confidence intervals are recomputed for the sixteen-row roster, and raw_on_A of standard_one_8b_sh and the field-level diagnostics (G_med_api_basis_P_vs_A, the API-basis offsets) move again for the reason given below; none of them enters a score. All five were measured in one tranche on ONE RTX 5090 32 GB pod in Brasov, Romania, strictly one row at a time, under a pre-registration written before the pod existed (r90/PREREG-R90.md, sha256 758959702bf04a3b8ab6a29f87d3320b2ca487b8548398341a37e2d3bf37f35d), on the authority of board #17148.

jiwo 0.8b and jiwo 4b: the author's own package and `jiwo serve` at the pinned snapshot directory, because their server takes no revision and a bare repo id would resolve main. JIWO_MAX_LENGTH=65536 is the author's own documented value for their published runs; the 8,192 default refuses longer prompts with HTTP 400 and never truncates. 29 of 1,500 items - the longest states of this draw, 76,381 to 80,883 tokens as the author's own tokenizer counts them - exceed even 65,536 and were refused with HTTP 400; they are scored wrong, as the method requires (the pre-registration had expected 65,536 to cover every item, so this is a disclosed miss of that prediction, not a changed setting), and they are exactly the same 29 items the published gutsy, standard_one_8b_sh, RSI-Jev, Blink and ClassOne rows of this draw also lose to their own context limits. The trained context is 8,192 tokens, so every longer item is answered outside the range the model saw in training. CUDA graphs are OFF: the author's own card says graphs agree with eager on only 99.7 % of fields, so eager is the exact path - the author's 14.5 ms headline uses a mode this measurement does not use. Native probabilities, output tokens a literal 0, cost a labelled estimate from the author's own declared base. Speed: the author's install line does not install the optional causal_conv1d kernel, so transformers used its reference PyTorch fallback (stated in the serve logs; correct but slower). The same holds for the two ClassOne Qwen rows, not for ClassOne Gemma 4 E4B. The Speed axis of those four rows is that of the authors' stacks as shipped. Not comparable with the author's headline numbers, which are Decision Index 0.3, a DIFFERENT benchmark: "1st of 27 under 1B" and "3rd of 26 in 3-6B" are placements on that board. The jiwo 4b submitter's note claims "top2 for its model size", which contradicts the author's own card (3rd of 26 in the 3-6B class); and both cards' figures are v1's, while the measured revision is v2, whose placement the cards themselves mark pending. The authors publish no decontamination procedure against JevBench items; no JevBench item is involved either way, because the model is run here. English only by the author's own statement.

ClassOne Qwen 3.5 2B, Qwen 3.5 4B and Gemma 4 E4B: the author's own install and serve lines at code pin 6cca132b, with CLASSONE_BASE_MODEL pointed at each pinned snapshot directory, which their own code accepts and which makes classone_heads.pt resolve locally, so no revision can drift and no request-time hub fetch exists. Their hard-coded MAX_TOKENS_PER_REQUEST = 8192 refuses longer prompts with HTTP 400 and never truncates: 29 of 1,500 items were refused that way (the same 29 longest states: 76,324 to 80,861 tokens as the author's own tokenizer counts them for the two Qwen rows, 76,378 to 82,123 for Gemma 4 E4B), so each ClassOne row answered 1,471. The pre-registration had predicted 23 such refusals, the count on the predecessor draw; this draw has 29, which is that prediction missing, not a changed setting. Those items are scored wrong, as the method requires, exactly as the two already-published ClassOne rows carry it. ONE documented mapping of ours and the only one: their server keys score probabilities "1".."n" while JevBench's canonical keys are "0".."n-1", so run 58's byte-identical ClassOneReindexAdapter relabels score keys k -> k-1 by calling the unchanged vendored typesafe adapter's own run(), and an item whose key set is not exactly {"1".."n"} is refused rather than rescored on a guess; choice and noul are untouched. The same stricter run-58 smoke gate passed on synthetic requests before any sealed item was sent. Native probabilities, output tokens a literal 0. Not comparable with the author's public scorecard, which is self-reported on a different 231-item public set - and in the scorecard of the author's class-one code-repo README the Gemma 4 E4B row is byte-identical to the E2B row in all four of its JevBench columns, so at least one of those two is not a separate measurement. Cost for the two Qwen rows resolves through the unmodified frozen path at the author's own declared spelling; for Gemma 4 E4B pricing_v15.reference() raises on every spelling of the base, so that row carries a labelled estimate from the reference the frozen 25 Sep 2026 snapshot itself contains for exactly this base and size (deepinfra:google/gemma-4-E4B-it, USD 0.02 in / 0.10 out per 1M) - the same rate the published classone-gemma4-e2b row uses as a one-size-UP estimate. A release-lane line is recorded with that row: the base needs a listing decision or an UNLISTED_BASE_MODELS entry in the frozen table.

v1.6.8 ADDED ONE ROW, modernbert-jev v2, to the ten of v1.6.7 on the same draw, under the same rule: the field median stays the frozen v1.6.5 value, so all ten earlier rows keep identical point scores, axes and capability; ranks and bootstrap confidence intervals were recomputed for the eleven-row roster, and raw_on_A of standard_one_8b_sh and the field-level diagnostics (G_med_api_basis_P_vs_A, the API-basis offsets) moved for the reason given below; none of them enters a score.

modernbert-jev-v2: a 150M ModernBERT-base NLI encoder, int8 (unsigned) ONNX, served by the author's own CLI at its own defaults on ONE CPU thread of the evaluator's shared Sandy host (AMD Ryzen 5 3600, Zen 2), not a dedicated machine; the 1-minute load average was logged every 60 s and ranged about 5.5 to 27.9 (median 10.2) during the run, so Speed is a lower bound for this model on an idle core. The run was interrupted once by the evaluator's session ending after 41 answered items and resumed with the unchanged runner's --resume, which keeps answered rows and sends only the rest; the server is deterministic (byte-identical answers on repeated requests). By the author's documented design (README: "windowing retrieval is future work"), a state is cut at 20,000 characters and read through at most six 300-word windows spread across it: the 29 longest states of this draw are answered from that sample rather than refused, so words between the windows are never seen. Probabilities are native (per-head temperatures from the shipped calibration.json). With the shipped Noul temperature of 2.5, every one of its 375 Noul answers lies between P(yes) 0.23 and 0.80, so none counts as decisive under the O1S Noul method, which puts every Noul tier of this row at its floor; this is the model as shipped, not a serving fault. Output tokens are a literal 0; cost is a labelled estimate at the base-size encoder rate.

v1.6.7 ADDED ONE ROW, Blink v0.3 26B-A4B, to the nine of v1.6.6 on the same draw, under the same rule: the field median stays the frozen v1.6.5 value, so all nine earlier rows keep identical point scores, axes and capability; ranks and bootstrap confidence intervals are recomputed for the ten-row roster, and raw_on_A of standard_one_8b_sh moves again for the reason given below. Blink's gap (8.161) is below the threshold.

blink-v0-3-26b-a4b-nvfp4: measured from the model card's own plain-vLLM 0.30.0 serve command (NVFP4, fp8 KV cache, 32,768-token context, T=1.3 option-letter readout through a loopback shim reproducing the card's decide() client) with ONE serving-only deviation: VLLM_BATCH_INVARIANT=1. With the command as published, four identical repeats of 50 public items differed by up to 0.126 in probability, failing the pre-registered repeatability gate (0.005); with batch-invariant kernels the repeats were byte-identical. This makes the run repeatable, not more correct: between serving settings the same item's probabilities differ by up to 0.39 and the argmax by 2 of 49 public items, so a different stack or card can score differently. Latency is of the batch-invariant path (about twice the default per request). Run on an RTX 5090 32 GB (no RTX PRO 6000 was free). 29 of 1,500 states (77,706 to 82,401 tokens) exceed the card's 32,768-token context and were refused with HTTP 422; scored wrong per method. Cost is a labelled estimate from the base model google/gemma-4-26b-a4b-it (USD 0.09 in / 0.30 out per 1M, frozen 25 Sep 2026 OpenRouter snapshot).

v1.6.6 ADDED ONE ROW, Clef-omni, to the eight of v1.6.5 on the same draw. The field median is held at the frozen v1.6.5 value (g_med_fixed, as for earlier same-draw additions), so all eight earlier rows keep identical point scores, axes and capability. Ranks and bootstrap confidence intervals are recomputed for the nine-row roster, as the v1.6.3 and v1.6.4 addenda did. One diagnostic also moves: raw_on_A (scores on the 300-item A subset) of standard_one_8b_sh, because the A-basis median is not part of the frozen value. Clef-omni's gap (5.555) is below the threshold.

clef-omni: measured as shipped on its own in-process entry point at the pinned revision, defaults unchanged. 29 of 1,500 items reach the model's own 64,000-token default and are answered on the truncated state, as the authors' encoder does by design; none was refused. The Omni video processor needs torchvision to load, which the model card's tested stack does not list; torchvision 0.26.0+cu130, the build matching its torch 2.11.0+cu130, was added and nothing else changed. Output tokens are a literal 0, so the cost is input-only.

wald-4b-v2-1: the shipped v2.1 server applies a documented Noul decisive floor (serving.json noul_decisive_floor 0.81): every calibrated P(yes) inside (0.19, 0.81) is moved to the nearer edge, on every Noul answer and not keyed to any item, so it answers all 375 Noul items decisively. Measured as shipped, server default effort none (one pass, 0 output tokens). It is the successor of the published wald-4b-v2 row on the paid draw and carries the same base-model cost basis.

Subject topics and TypeSafe use cases were labelled for this draw's 1,500 items in their own run (Winnow-12B Q8 on our own pod, egress cut before the sealed upload, the unchanged T1-T3/U1/U2 rules; no new rule was added). On 75 held-out public items checked by hand, topic agreement after the rules is 71/75 = 94.7 %. Known systematic miss: 17 of 27 router_policy items are filed under Coding by the subject of the request rather than under Rules, policy & law by the routing task; left as labelled because adding a rule after seeing the sample would change the method.