{ "G_med": 10.073244581339713, "G_med_api_basis_P_vs_A": 9.100119455663567, "G_med_flag_gt10": true, "benchmark": "JevBench", "board": { "A": { "markers": [], "order": [ "wald-4b-v2-1", "rsi-jev-v6.1-vl-27b", "standard_one_8b_sh", "liquid-d1-3b", "clef-omni", "vegaml-4b", "gutsy-0.8b-v0.3", "liquid-d1-omni-600m", "vegaml-0.8b" ] }, "B": { "markers": [], "order": [ "wald-4b-v2-1", "rsi-jev-v6.1-vl-27b", "standard_one_8b_sh", "liquid-d1-3b", "clef-omni", "vegaml-4b", "gutsy-0.8b-v0.3", "liquid-d1-omni-600m", "vegaml-0.8b" ] }, "C": { "markers": [], "order": [ "wald-4b-v2-1", "rsi-jev-v6.1-vl-27b", "standard_one_8b_sh", "liquid-d1-3b", "clef-omni", "vegaml-4b", "gutsy-0.8b-v0.3", "liquid-d1-omni-600m", "vegaml-0.8b" ] }, "capability": { "order": [ "rsi-jev-v6.1-vl-27b", "wald-4b-v2-1", "standard_one_8b_sh", "clef-omni", "liquid-d1-3b", "gutsy-0.8b-v0.3", "vegaml-4b", "vegaml-0.8b", "liquid-d1-omni-600m" ] } }, "bootstrap": { "B": 1000, "bootstrap_seed": 16, "g_med_fixed": true }, "headline": "A", "label": "JevBench v1.6.6 \u00b7 fresh regular draw, Clef-omni joins the v1.6.5 field", "measurement_notes": [ "Refusals, errors and context-bound failures stay in the full 1,500 denominator. Every row kept its own package, model, runtime and context settings; nothing was re-run for presentation.", "NOT DIRECTLY COMPARABLE WITH THE LIVE v1.6.1 BOARD. This release is a different draw with its own field median: G_med = 10.073244581339713, frozen from the eight v1.6.5 members (Clef-omni does not move it), against 2.573387642438244 on the live field. G_med_flag_gt10 is TRUE, which has not been true in an earlier published JevBench artifact. Concretely: the gap penalty is max(0, 1 - max(0, (gap - G_med) - 8)/100), so this field's threshold is G_med + 8 = 18.073 and no row here is penalised for its public-vs-sealed gap, whereas at the live field's median four of the nine would be. The difference was measured, not estimated, and is small: rsi-jev-v6.1-vl-27b Intelligence 61.564 would be 59.551, standard_one_8b_sh 34.465 would be 32.460, liquid-d1-3b 22.139 would be 21.949, vegaml-4b 16.689 would be 16.673, and wald-4b-v2-1 (gap 9.478, below the live threshold 10.573) and the other three are unchanged. The direction is one-sided: this field flatters its own members.", "Cost: no item is excluded from the cost basis on this draw. Every row is priced from its own measured usage block against the frozen 25 Sep 2026 snapshot, or from a documented per-1,000 override, so the paid draw's public-counter-overflow exclusions have no analogue here. The cost normalizer C0 is USD 0.001 per 1,000 decisions with 30 cost points per decade. Every price on these rows is a clearly-labelled ESTIMATE; none is a bill any of these systems issued.", "gutsy-0.8b-v0.3: 29 of 1,500 items returned HTTP 500 \"llama_decode returned 1\" - the longest items, which llama.cpp cannot decode into the author's 8,192-token context. They are scored wrong, as the method requires. This is a crash class rather than a declared refusal. Its cost is input-only and marked *est.: the author's server emits no output-token field, and under the 9 Oct 2026 rule a missing output_tokens on a model with no generation path at all is treated as 0 output tokens.", "standard_one_8b_sh and rsi-jev-v6.1-vl-27b: the same 29 longest items exceed each system's declared 32,768-token context and were refused with HTTP 422 rather than truncated; scored wrong per method. For RSI-Jev the refusal boundary was proven before the scored run: 13 of 13 boundary checks passed, including that a 32,768-token question is accepted and seen whole and a 32,769-token one returns 422 with nothing silently cut.", "liquid-d1-omni-600m truncates states on the right at 16,384 tokens by design, and its published run is the second of two: a first run on a 24 GB card hit 29 CUDA out-of-memory errors, which was our under-provisioning and not model behaviour; it is retained and not scored. vegaml-0.8b and vegaml-4b truncate 29 states on the right at 73,728 tokens by design. The Vega checkpoint repository declares no licence field; the vegaml package is Apache-2.0.", "v1.6.6 ADDS ONE ROW, Clef-omni, to the eight of v1.6.5 on the same draw. The field median is held at the frozen v1.6.5 value (g_med_fixed, as for earlier same-draw additions), so all eight earlier rows keep identical point scores, axes and capability. Ranks and bootstrap confidence intervals are recomputed for the nine-row roster, as the v1.6.3 and v1.6.4 addenda did. One diagnostic also moves: raw_on_A (scores on the 300-item A subset) of standard_one_8b_sh, because the A-basis median is not part of the frozen value. Clef-omni's gap (5.555) is below the threshold.", "clef-omni: measured as shipped on its own in-process entry point at the pinned revision, defaults unchanged. 29 of 1,500 items reach the model's own 64,000-token default and are answered on the truncated state, as the authors' encoder does by design; none was refused. The Omni video processor needs torchvision to load, which the model card's tested stack does not list; torchvision 0.26.0+cu130, the build matching its torch 2.11.0+cu130, was added and nothing else changed. Output tokens are a literal 0, so the cost is input-only.", "wald-4b-v2-1: the shipped v2.1 server applies a documented Noul decisive floor (serving.json noul_decisive_floor 0.81): every calibrated P(yes) inside (0.19, 0.81) is moved to the nearer edge, on every Noul answer and not keyed to any item, so it answers all 375 Noul items decisively. Measured as shipped, server default effort none (one pass, 0 output tokens). It is the successor of the published wald-4b-v2 row on the paid draw and carries the same base-model cost basis.", "Subject topics and TypeSafe use cases were labelled for this draw's 1,500 items in their own run (Winnow-12B Q8 on our own pod, egress cut before the sealed upload, the unchanged T1-T3/U1/U2 rules; no new rule was added). On 75 held-out public items checked by hand, topic agreement after the rules is 71/75 = 94.7 %. Known systematic miss: 17 of 27 router_policy items are filed under Coding by the subject of the request rather than under Rules, policy & law by the routing task; left as labelled because adding a rule after seeing the sample would change the method." ], "n_ranked": 9, "not_measured": [], "noul_method": { "applied": "O1S", "native_method_source_sha256": "119e59486ab3300b972fbcbadb0fe1dceceb124287607375aa61c66063c607d0" }, "options": { "A": { "intelligence_floor": 50.0, "weights": { "calibration": 25, "cost": 25, "intelligence": 25, "speed": 25 } }, "B": { "intelligence_floor": 50.0, "weights": { "calibration": 20, "cost": 20, "intelligence": 40, "speed": 20 } }, "C": { "intelligence_floor": 60.0, "weights": { "calibration": 25, "cost": 25, "intelligence": 25, "speed": 25 } } }, "protocol": "jevbench::v1.6", "provisional": false, "revision": "v1.6.6", "roster_count": 9, "run_kind": "regular-queue", "sample": { "open": 300, "published_open": 300, "sealed": 1200, "total": 1500 }, "sealed_share_of_intelligence": 0.5, "source_note": "Nine systems: Clef-omni, added in v1.6.6, and the eight v1.6.5 systems, all measured on the fresh regular draw v1.6-regular-20261010-a2 (seeded, rotation-ledger draw of 1,200 sealed + the 300-item epoch public set; every drawn item had zero prior scored uses and the set does not overlap v1.6.0 or the paid fast-lane draw). One score_v16.py invocation, unmodified frozen path, JEV_NOUL_METHOD=O1S, B=1000, seed 16. v1.6.6 is one score_v16 invocation over all nine with --g-med-fixed 10.073244581339713, the frozen v1.6.5 median, so the eight earlier rows keep every point score. Drawn, frozen and scored by jevbench-add-requests-20260919 as v1.6.5 release owner (board #16539, #16625, #16630, #16703, #16809; v1.6.6 under CoS #16771, board #16827).", "source_sha256": "18f9665d3e4674cdf015a254c897aea27c7bae846eed90a16b58381a7b1fb8ee", "status": "published", "systems": [ { "addendum": null, "api_flag": false, "api_subset": false, "author": "Cloudflare", "axes": { "calibration": 82.15469329918429, "cost": 26.792900075221908, "intelligence": 34.47716016566848, "speed": 86.7880661703893 }, "calibration": { "parts": { "choice": { "ece_hard": 0.09821189024390234, "ece_score": 80.35762195121954, "mean_tvd": null, "n": 750, "n_ece_hard": 328, "n_tvd": 0, "score": 80.35762195121954 }, "noul": { "abstention_rate": 0.3493333333333334, "brier": 0.1572640221333334, "ece": 0.06652586666666661, "n": 375, "score": 86.69482666666669 }, "score": { "interval90_coverage": 0.8506666666666667, "interval90_width": 0.36177777777777775, "n": 375, "nrps": 0.11951350773999989, "score": 79.41163127966668, "top_ece": 0.14612693333333326 } }, "score": 82.15469329918429 }, "capability": 58.315926732426384, "composite_ci95": { "A": [ 3.7625229387967183, 8.453137879795586 ], "B": [ 3.4413449720032188, 8.185527143483494 ], "C": [ 2.612863151942165, 5.870234638746934 ] }, "cost": { "basis": "ESTIMATE (*est.), base-class reference. Self-hosted open weights; the declared base Qwen/Qwen3-Omni-30B-A3B-Instruct is not a key of any frozen pricing table. Method as the v1.6.5 Liquid rows: the nearest same-family entry of the frozen 25 Sep 2026 snapshot, deepinfra:Qwen/Qwen3-VL-30B-A3B-Instruct (the multimodal Qwen3 30B-A3B), at USD 0.15 per 1M input and USD 0.60 per 1M output. The authors' systemone() has no generation path, so output_tokens is a literal 0. This row's own usage on draw v1.6-regular-20261010-a2: 2,755,730 input and 0 output tokens over 1,500 decisions.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.275573 }, "display": "Clef-omni", "endpoint_condition": "evaluator-owned Lium RTX PRO 6000 Blackwell 96 GB pod (Beltsville US, 29 min, USD 1.19/h); the authors' own load_release_model() + systemone() from the pinned repo's joint_schema_model.py behind the run-39 loopback shim (127.0.0.1, single-threaded, no answer arithmetic), pointed at the LOCAL PINNED SNAPSHOT DIRECTORY, defaults unchanged (max_length 64,000). Stack as the model card tested: torch 2.11.0+cu130, transformers 5.10.2, plus torchvision 0.26.0+cu130, which the Omni video processor requires to load. HF/transformers offline, no credentials, pod-level iptables egress cut installed and verified before the sealed input was uploaded. All 17 LFS files verified against the Hugging Face tree digests. GPU memory 62,729 MiB after load and at most 74,735 MiB during the run (per-minute log), of 97,887 MiB.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 37.25444425506104, "I_sealed": 31.699876076275928, "base": 34.47716016566848, "excess": -4.518676402554602, "gap": 5.554568178785111, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 65.02240890008018, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 84.43243243243245, "hard": 67.9144385026738, "judge": 56.49307844429796, "standard": 62.32733361239012 } }, "open|noul": { "cc": 10.814354727398204, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 71.42857142857142, "hard": -13.043478260869568, "judge": 0.0, "standard": 44.44444444444444 } }, "open|score": { "cc": 35.92656913770473, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 51.82848484848485, "hard": 42.1251851851852, "judge": 49.07338983050846, "standard": -4.141851851851852 } }, "sealed|choice": { "cc": 53.547209374932514, "n": { "easy": 72, "hard": 260, "judge": 141, "standard": 127 }, "tiers": { "easy": 62.45698195849411, "hard": 49.543805650214345, "judge": 52.513824767896104, "standard": 58.64920744314267 } }, "sealed|noul": { "cc": 9.67795362532205, "n": { "easy": 39, "hard": 132, "judge": 72, "standard": 57 }, "tiers": { "easy": 43.58974358974359, "hard": -10.606060606060607, "judge": 16.666666666666675, "standard": 22.807017543859654 } }, "sealed|score": { "cc": 31.87446522857323, "n": { "easy": 33, "hard": 115, "judge": 75, "standard": 77 }, "tiers": { "easy": 45.32728971962617, "hard": 22.845027777777847, "judge": 36.40668141592918, "standard": 36.408603603603595 } } } }, "jevbench_score": 6.06640325953113, "key": "clef-omni", "last_measured_on": "2026-10-10", "licence": "Apache-2.0", "listing": "ranked", "measured_in": "v1.6.6", "measurement_date_status": "measured", "model_pin": "Cloudflare/clef-omni@0db1cd2607d76a7bdb2a382f659e7b313079f84b", "noul_decisive": { "acc_among_decisive": 0.8565573770491803, "decisive": 244, "decisive_rate": 0.6506666666666666, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 5, "ranked": true, "ranks": { "A": 5, "B": 5, "C": 5, "capability": 4 }, "repo": "https://huggingface.co/Cloudflare/clef-omni", "scores": { "A": 6.06640325953113, "B": 5.735167687307061, "C": 4.2127800413410625 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.43618169901892545, "p50_s_raw": 0.14309084950946271, "p95_s_adjusted": 0.4803147715982049, "p95_s_raw": 0.16515738579910247 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.896551724137931, "cc": 86.57097222222222, "cc_by_type": { "choice": 91.66666666666667, "noul": 100.0, "score": 68.04625 }, "low_n": true, "n": 29, "n_by_type": { "choice": 16, "noul": 6, "score": 7 }, "score": 86.57097222222222 }, "calibration_gen": { "accuracy": 0.5982142857142857, "cc": 32.91760471412644, "cc_by_type": { "choice": 55.745341614906835, "noul": 15.384615384615374, "score": 27.622857142857114 }, "low_n": false, "n": 112, "n_by_type": { "choice": 56, "noul": 26, "score": 30 }, "score": 32.91760471412644 }, "composite_gen": { "accuracy": 0.6632653061224489, "cc": 50.659661596119925, "cc_by_type": { "choice": 74.07407407407408, "noul": 35.71428571428572, "score": 42.19062499999998 }, "low_n": false, "n": 98, "n_by_type": { "choice": 36, "noul": 28, "score": 34 }, "score": 50.659661596119925 }, "fairness": { "accuracy": 0.7407407407407407, "cc": 51.41265359477125, "cc_by_type": { "choice": 88.23529411764707, "noul": 0.0, "score": 66.00266666666667 }, "low_n": true, "n": 27, "n_by_type": { "choice": 12, "noul": 6, "score": 9 }, "score": 51.41265359477125 }, "jaggedness": { "accuracy": 0.42162162162162165, "cc": -0.8300322403138493, "cc_by_type": { "choice": 30.523390458545624, "noul": -53.84615384615385, "score": 20.832666666666675 }, "low_n": false, "n": 185, "n_by_type": { "choice": 97, "noul": 39, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.6818181818181818, "cc": 57.872019428772546, "cc_by_type": { "choice": 50.76637250348352, "noul": 53.42465753424656, "score": 69.42502824858757 }, "low_n": false, "n": 264, "n_by_type": { "choice": 131, "noul": 73, "score": 60 }, "score": 57.872019428772546 }, "long_state": { "accuracy": 0.3103448275862069, "cc": -1.3167821067821077, "cc_by_type": { "choice": 16.66666666666667, "noul": -27.27272727272727, "score": 6.655714285714276 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 11, "score": 7 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.5483870967741935, "cc": 5.659007803790407, "cc_by_type": { "choice": 53.84615384615384, "noul": -25.0, "score": -11.869130434782615 }, "low_n": false, "n": 31, "n_by_type": { "choice": 15, "noul": 8, "score": 8 }, "score": 5.659007803790407 }, "multilingual": { "accuracy": 0.5714285714285714, "cc": 29.241308106053285, "cc_by_type": { "choice": 54.32154052843709, "noul": 5.747126436781613, "score": 27.655257352941153 }, "low_n": false, "n": 315, "n_by_type": { "choice": 146, "noul": 87, "score": 82 }, "score": 29.241308106053285 }, "negation_pair_gen": { "accuracy": 0.6607142857142857, "cc": 18.345555555555563, "cc_by_type": { "choice": 66.66666666666667, "noul": 19.999999999999996, "score": -31.62999999999998 }, "low_n": false, "n": 56, "n_by_type": { "choice": 28, "noul": 20, "score": 8 }, "score": 18.345555555555563 }, "option_order": { "accuracy": 0.8947368421052632, "cc": 86.68146503884572, "cc_by_type": { "choice": 86.68146503884572 }, "low_n": false, "n": 57, "n_by_type": { "choice": 57 }, "score": 86.68146503884572 }, "perturb_pair": { "accuracy": 0.9, "cc": 81.10219498659866, "cc_by_type": { "choice": 84.27260812581913, "noul": 71.42857142857142, "score": 87.6054054054054 }, "low_n": false, "n": 60, "n_by_type": { "choice": 34, "noul": 14, "score": 12 }, "score": 81.10219498659866 }, "usecase": { "accuracy": 0.5232067510548524, "cc": 17.11602027853846, "cc_by_type": { "choice": 53.74831374060514, "noul": -15.789473684210531, "score": 13.389220779220778 }, "low_n": false, "n": 237, "n_by_type": { "choice": 111, "noul": 57, "score": 69 }, "score": 17.11602027853846 } }, "lang": { "ar": { "accuracy": 0.45454545454545453, "cc": -23.164444444444445, "cc_by_type": { "choice": 46.666666666666664, "noul": -50.0, "score": -66.16 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 4, "score": 2 }, "score": 0.0 }, "cs": { "accuracy": 0.5, "cc": 7.90960451977401, "cc_by_type": { "choice": 49.152542372881356, "noul": -33.333333333333336 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 3 }, "score": 7.90960451977401 }, "da": { "accuracy": 0.5416666666666666, "cc": 23.599881656804737, "cc_by_type": { "choice": 52.662721893491124, "noul": -19.999999999999996, "score": 38.13692307692308 }, "low_n": true, "n": 24, "n_by_type": { "choice": 11, "noul": 5, "score": 8 }, "score": 23.599881656804737 }, "de": { "accuracy": 0.7111111111111111, "cc": 51.80671574665221, "cc_by_type": { "choice": 72.84248642124321, "noul": 55.55555555555556, "score": 27.02210526315789 }, "low_n": false, "n": 45, "n_by_type": { "choice": 24, "noul": 9, "score": 12 }, "score": 51.80671574665221 }, "el": { "accuracy": 0.3333333333333333, "cc": -55.7336546184739, "cc_by_type": { "choice": 39.7590361445783, "noul": -100.0, "score": -106.96 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 5, "score": 2 }, "score": 0.0 }, "en": { "accuracy": 0.6094727435210009, "cc": 34.60248155699925, "cc_by_type": { "choice": 56.70218732473357, "noul": 9.774436090225569, "score": 37.33082125603862 }, "low_n": false, "n": 1119, "n_by_type": { "choice": 577, "noul": 266, "score": 276 }, "score": 34.60248155699925 }, "es": { "accuracy": 0.7142857142857143, "cc": 54.17545454545455, "cc_by_type": { "choice": 73.81818181818181, "noul": 63.63636363636365, "score": 25.071818181818184 }, "low_n": true, "n": 28, "n_by_type": { "choice": 10, "noul": 11, "score": 7 }, "score": 54.17545454545455 }, "fi": { "accuracy": 0.1111111111111111, "cc": -52.816901408450704, "cc_by_type": { "choice": -5.63380281690141, "noul": -100.0 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 3 }, "score": 0.0 }, "fr": { "accuracy": 0.6, "cc": 17.07420198356729, "cc_by_type": { "choice": 74.61212976022568, "noul": -50.0, "score": 26.610476190476174 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 4, "score": 6 }, "score": 17.07420198356729 }, "hi": { "accuracy": 0.6333333333333333, "cc": 44.937223084886135, "cc_by_type": { "choice": 62.73291925465839, "noul": 60.00000000000001, "score": 12.078749999999994 }, "low_n": false, "n": 30, "n_by_type": { "choice": 11, "noul": 10, "score": 9 }, "score": 44.937223084886135 }, "id": { "accuracy": 0.16666666666666666, "cc": -48.387096774193544, "cc_by_type": { "choice": 3.225806451612906, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 0.0 }, "it": { "accuracy": 0.7058823529411765, "cc": 50.644668253408554, "cc_by_type": { "choice": 74.61212976022568, "noul": 50.0, "score": 27.321874999999995 }, "low_n": true, "n": 17, "n_by_type": { "choice": 5, "noul": 4, "score": 8 }, "score": 50.644668253408554 }, "ja": { "accuracy": 0.46875, "cc": 18.159518002322883, "cc_by_type": { "choice": 26.829268292682926, "noul": -14.28571428571429, "score": 41.93500000000001 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 18.159518002322883 }, "ko": { "accuracy": 0.6363636363636364, "cc": 66.1930382513661, "cc_by_type": { "choice": 67.21311475409836, "noul": 100.0, "score": 31.365999999999982 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 2, "score": 5 }, "score": 66.1930382513661 }, "mixed": { "accuracy": 1.0, "cc": 85.82333333333332, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 57.469999999999956 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 85.82333333333332 }, "nl": { "accuracy": 0.5384615384615384, "cc": 15.964908077994421, "cc_by_type": { "choice": 49.860724233983284, "noul": 0.0, "score": -1.9660000000000233 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 6, "score": 2 }, "score": 15.964908077994421 }, "no": { "accuracy": 0.2857142857142857, "cc": -15.846994535519126, "cc_by_type": { "choice": 1.6393442622950836, "noul": -33.333333333333336 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 3 }, "score": 0.0 }, "pl": { "accuracy": 0.575, "cc": 36.12173351608311, "cc_by_type": { "choice": 84.25885439440316, "noul": 19.999999999999996, "score": 4.106346153846174 }, "low_n": false, "n": 40, "n_by_type": { "choice": 16, "noul": 10, "score": 14 }, "score": 36.12173351608311 }, "pt": { "accuracy": 0.8333333333333334, "cc": 74.19423813512005, "cc_by_type": { "choice": 81.90954773869345, "noul": 66.66666666666667, "score": 74.0065 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 6, "score": 5 }, "score": 74.19423813512005 }, "sv": { "accuracy": 0.375, "cc": 4.158888888888879, "cc_by_type": { "choice": -33.333333333333336, "noul": 50.0, "score": -4.190000000000027 }, "low_n": true, "n": 8, "n_by_type": { "choice": 3, "noul": 4, "score": 1 }, "score": 4.158888888888879 }, "tr": { "accuracy": 0.7272727272727273, "cc": 22.35695906432748, "cc_by_type": { "choice": 73.6842105263158, "noul": 33.33333333333333, "score": -39.94666666666669 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 3, "score": 3 }, "score": 22.35695906432748 }, "uk": { "accuracy": 0.7272727272727273, "cc": 12.17157190635451, "cc_by_type": { "choice": 78.26086956521739, "noul": -100.0, "score": 58.25384615384615 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 1, "score": 4 }, "score": 12.17157190635451 }, "zh": { "accuracy": 0.625, "cc": 50.52282282282281, "cc_by_type": { "choice": 35.13513513513514, "noul": 33.33333333333333, "score": 83.09999999999995 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 50.52282282282281 } }, "type": { "choice": { "accuracy": 0.6653333333333333, "cc": 56.64190706512352, "cc_by_type": { "choice": 56.64190706512352 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 56.64190706512352 }, "noul": { "accuracy": 0.5573333333333333, "cc": 11.466666666666669, "cc_by_type": { "noul": 11.466666666666669 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 11.466666666666669 }, "score": { "accuracy": 0.5333333333333333, "cc": 33.64992119089322, "cc_by_type": { "score": 33.64992119089322 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 33.64992119089322 } }, "usecase": { "advertising": { "accuracy": 0.5416666666666666, "cc": 8.511753615229727, "cc_by_type": { "choice": 40.691927512355846, "noul": -60.0, "score": 44.843333333333334 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 8.511753615229727 }, "code_linting": { "accuracy": 0.5833333333333334, "cc": 9.050032733224224, "cc_by_type": { "choice": 70.54009819967267, "noul": -19.999999999999996, "score": -23.39 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 9.050032733224224 }, "customer_support": { "accuracy": 0.3, "cc": -13.059735318444998, "cc_by_type": { "choice": 35.483870967741936, "noul": -75.0, "score": 0.336923076923068 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 8, "score": 4 }, "score": 0.0 }, "demand_forecasting": { "accuracy": 0.38461538461538464, "cc": -8.969614214433495, "cc_by_type": { "choice": 39.7590361445783, "noul": -50.0, "score": -16.667878787878788 }, "low_n": true, "n": 26, "n_by_type": { "choice": 11, "noul": 4, "score": 11 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5555555555555556, "cc": 47.39004937091894, "cc_by_type": { "choice": 44.099378881987576, "noul": 60.00000000000001, "score": 38.07076923076923 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 5, "score": 6 }, "score": 47.39004937091894 }, "feature_extraction": { "accuracy": 0.4827586206896552, "cc": 12.539619269619266, "cc_by_type": { "choice": 48.71794871794871, "noul": -60.0, "score": 48.90090909090908 }, "low_n": true, "n": 29, "n_by_type": { "choice": 15, "noul": 5, "score": 9 }, "score": 12.539619269619266 }, "financial_crime": { "accuracy": 0.5384615384615384, "cc": 14.042347826086965, "cc_by_type": { "choice": 73.91304347826087, "noul": 0.0, "score": -31.78599999999998 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 14.042347826086965 }, "gaming": { "accuracy": 0.41379310344827586, "cc": 5.618274556027025, "cc_by_type": { "choice": 28.712871287128706, "noul": -14.28571428571429, "score": 2.4276666666666613 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 7, "score": 11 }, "score": 5.618274556027025 }, "insurance_claims": { "accuracy": 0.2857142857142857, "cc": -11.208359788359795, "cc_by_type": { "choice": 20.634920634920636, "noul": -100.0, "score": 45.73999999999998 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 5, "score": 1 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.5217391304347826, "cc": 9.780414954002737, "cc_by_type": { "choice": 45.03816793893129, "noul": -60.0, "score": 44.30307692307692 }, "low_n": true, "n": 23, "n_by_type": { "choice": 14, "noul": 5, "score": 4 }, "score": 9.780414954002737 }, "lead_generation": { "accuracy": 0.4, "cc": 5.368834688346889, "cc_by_type": { "choice": 83.73983739837398, "noul": -75.0, "score": 7.3666666666666885 }, "low_n": true, "n": 25, "n_by_type": { "choice": 8, "noul": 8, "score": 9 }, "score": 5.368834688346889 }, "legal_compliance": { "accuracy": 0.3076923076923077, "cc": -20.515570556091294, "cc_by_type": { "choice": 34.42622950819673, "noul": -100.0, "score": 4.027058823529394 }, "low_n": true, "n": 13, "n_by_type": { "choice": 8, "noul": 1, "score": 4 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.6206896551724138, "cc": 32.95168666196189, "cc_by_type": { "choice": 63.30275229357799, "noul": 0.0, "score": 35.55230769230767 }, "low_n": true, "n": 29, "n_by_type": { "choice": 14, "noul": 8, "score": 7 }, "score": 32.95168666196189 }, "model_routing": { "accuracy": 0.74, "cc": 63.43091944512483, "cc_by_type": { "choice": 69.44518895738408, "noul": 77.27272727272727, "score": 43.57484210526314 }, "low_n": false, "n": 150, "n_by_type": { "choice": 74, "noul": 44, "score": 32 }, "score": 63.43091944512483 }, "moderation_trust_safety": { "accuracy": 0.5, "cc": 24.84468671679197, "cc_by_type": { "choice": 21.052631578947366, "noul": -19.999999999999996, "score": 73.48142857142854 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 5, "score": 3 }, "score": 24.84468671679197 }, "recruiting": { "accuracy": 0.6190476190476191, "cc": 25.69128720525705, "cc_by_type": { "choice": 69.84924623115576, "noul": 0.0, "score": 7.2246153846153955 }, "low_n": true, "n": 21, "n_by_type": { "choice": 9, "noul": 6, "score": 6 }, "score": 25.69128720525705 }, "risk_assessment": { "accuracy": 0.5384615384615384, "cc": 14.524019916142551, "cc_by_type": { "choice": 43.39622641509434, "noul": 33.33333333333333, "score": -33.15750000000002 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 3, "score": 3 }, "score": 14.524019916142551 }, "scientific_discovery": { "accuracy": 0.3125, "cc": -9.035004163241226, "cc_by_type": { "choice": 28.286852589641434, "noul": -42.85714285714286, "score": -12.53472222222225 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.8181818181818182, "cc": 63.02598587305031, "cc_by_type": { "choice": 71.36038186157518, "noul": 66.66666666666667, "score": 51.05090909090908 }, "low_n": true, "n": 22, "n_by_type": { "choice": 9, "noul": 6, "score": 7 }, "score": 63.02598587305031 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 75.92686147764063, 84.22517335430506 ], "I": [ 28.162084593953164, 39.81062325419229 ], "capability": [ 53.04164389447117, 61.30267107541918 ], "composite": { "A": [ 3.7625229387967183, 8.453137879795586 ], "B": [ 3.4413449720032188, 8.185527143483494 ], "C": [ 2.612863151942165, 5.870234638746934 ] }, "seed": 16 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 10.073244581339713, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 82.15469329918429, "capability": 58.315926732426384, "intelligence": 34.47716016566848 }, "per_type": { "choice": { "accuracy": 0.6653333333333333, "calibration": 80.35762195121954, "cc": 56.64190706512352, "cc_by_type": { "choice": 56.64190706512352 }, "cc_tiered": 59.28480913750634, "cc_tiered_open": 65.02240890008018, "cc_tiered_sealed": 53.547209374932514, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 56.64190706512352 }, "noul": { "accuracy": 0.5573333333333333, "calibration": 86.69482666666669, "cc": 11.466666666666669, "cc_by_type": { "noul": 11.466666666666669 }, "cc_tiered": 10.246154176360127, "cc_tiered_open": 10.814354727398204, "cc_tiered_sealed": 9.67795362532205, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 11.466666666666669 }, "score": { "accuracy": 0.5333333333333333, "calibration": 79.41163127966668, "cc": 33.64992119089322, "cc_by_type": { "score": 33.64992119089322 }, "cc_tiered": 33.90051718313898, "cc_tiered_open": 35.92656913770473, "cc_tiered_sealed": 31.87446522857323, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 33.64992119089322 } }, "raw_on_A": { "calibration": 80.93316891446167, "capability": 59.00548965209646, "intelligence": 37.07781038973123 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 5.261179060414329, "Emphasis on Accuracy 60:20:20": 5.02460048275502, "Emphasis on Cost 20:20:60": 4.476423118785528, "Emphasis on Speed 20:60:20": 6.765856602997514, "Intelligence only": 4.70710504931535 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Omar / @kouhxp (hi@kouh.me)", "axes": { "calibration": 77.50040036924202, "cost": 78.3542548810428, "intelligence": 8.600978834309654, "speed": 72.8877498600171 }, "calibration": { "parts": { "choice": { "ece_hard": 0.08639168454258676, "ece_score": 82.72166309148264, "mean_tvd": null, "n": 739, "n_ece_hard": 317, "n_tvd": 0, "score": 82.72166309148264 }, "noul": { "abstention_rate": 0.706043956043956, "brier": 0.24853405620583507, "ece": 0.12309816483516484, "n": 364, "score": 75.38036703296703 }, "score": { "interval90_coverage": 0.8695652173913043, "interval90_width": 0.6397192028985506, "n": 368, "nrps": 0.16768646620403746, "score": 74.39917098327638, "top_ece": 0.17216505706521745 } }, "score": 77.50040036924202 }, "capability": 43.05068960177584, "composite_ci95": { "A": [ 0.3773769198315772, 1.786744528078155 ], "B": [ 0.2637088587675396, 1.3280900163194176 ], "C": [ 0.26206730543859524, 1.2407948111653853 ] }, "cost": { "basis": "ESTIMATE (*est.), input tokens only. The author's gutsy-inference 0.3.0 server emits no output-token field in any spelling; under the chief-of-staff decision of 9 Oct 2026 a missing output_tokens field on a native option-probability model that generates no text is treated as 0 output tokens, exactly as a server reporting a literal 0 is treated. Zero output is a verified property of this backend, not an assumption: it has no generation path at all (no generate/create_completion/n_predict/max_tokens/sampler); it prefills via llm.eval(tokens) and reads llama_get_logits_ith(ctx,-1) (r52 PREREG-AMENDMENT-4). Rate: the declared base Qwen/Qwen3.5-0.8B resolves in the frozen 25 Sep 2026 snapshot to deepinfra:Qwen/Qwen3.5-0.8B at USD 0.01 per 1M input and USD 0.05 per 1M output. This row's own measured usage on draw v1.6-regular-20261010-a2: 789,980 input tokens and 0 output over 1,500 decisions => USD 0.0052665333333333335 per 1,000 decisions.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.0052665333333333335 }, "display": "Gutsy 0.8B v0.3", "endpoint_condition": "evaluator-owned Lium pod (2xRTX3090 node, Germany), CPU-only by the author's own config (n_gpu_layers 0), 32 vCPU, egress cut with iptables and proven before the sealed input was uploaded; the author's own gutsy-inference 0.3.0 with their models.example.json settings unchanged (n_ctx 8192, reject_slot true, n_threads null)", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 9.070301583187167, "I_sealed": 8.131656085432141, "base": 8.600978834309654, "excess": -9.134599083584687, "gap": 0.9386454977550258, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 25.412389147780978, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 37.72972972972974, "hard": 20.729789241899976, "judge": 12.986156888595916, "standard": 47.25826705734617 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 14.28571428571428, "hard": -47.82608695652174, "judge": -60.0, "standard": -33.333333333333336 } }, "open|score": { "cc": 1.7985156017805268, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 48.240515151515154, "hard": -7.74469814814811, "judge": 1.9498728813559296, "standard": -2.5630925925926196 } }, "sealed|choice": { "cc": 24.394968256296423, "n": { "easy": 72, "hard": 260, "judge": 141, "standard": 127 }, "tiers": { "easy": 23.036813014912926, "hard": 20.853028470924478, "judge": 26.031150119222776, "standard": 29.70365265334253 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 39, "hard": 132, "judge": 72, "standard": 57 }, "tiers": { "easy": -43.58974358974359, "hard": -78.78787878787878, "judge": -47.22222222222222, "standard": -61.40350877192983 } }, "sealed|score": { "cc": 0.0, "n": { "easy": 33, "hard": 115, "judge": 75, "standard": 77 }, "tiers": { "easy": 12.014685981308393, "hard": -2.484919999999935, "judge": -12.305056637168143, "standard": 12.255661711711696 } } } }, "jevbench_score": 0.7604367004496922, "key": "gutsy-0.8b-v0.3", "last_measured_on": "2026-10-10", "listing": "ranked", "measured_in": "v1.6.5", "measurement_date_status": "measured", "model_pin": "kouhxp/gutsy@d325eb1742cc1c200e04b24417f192a0dbf74f91", "noul_decisive": { "acc_among_decisive": 0.7383177570093458, "decisive": 107, "decisive_rate": 0.29395604395604397, "n": 375, "supported": true, "valid": 364 }, "planned_weight_denominator": 1.0, "rank": 7, "ranked": true, "ranks": { "A": 7, "B": 7, "C": 7, "capability": 6 }, "repo": "https://huggingface.co/kouhxp/gutsy", "scores": { "A": 0.7604367004496922, "B": 0.5441131491342173, "C": 0.5280810419789529 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 1.6613200119999418, "p50_s_raw": 0.755660005999971, "p95_s_adjusted": 3.0957915995999423, "p95_s_raw": 1.4728957997999712 }, "status": { "answered_ok": 1471, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.5862068965517241, "cc": 30.238494444444445, "cc_by_type": { "choice": 58.333333333333336, "noul": 0.0, "score": 32.38215000000001 }, "low_n": true, "n": 29, "n_by_type": { "choice": 16, "noul": 6, "score": 7 }, "score": 30.238494444444445 }, "calibration_gen": { "accuracy": 0.38392857142857145, "cc": -19.57648651457238, "cc_by_type": { "choice": 41.770186335403736, "noul": -92.3076923076923, "score": -8.191953571428567 }, "low_n": false, "n": 112, "n_by_type": { "choice": 56, "noul": 26, "score": 30 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.2653061224489796, "cc": -24.789154706790118, "cc_by_type": { "choice": 29.62962962962963, "noul": -100.0, "score": -3.9970937499999915 }, "low_n": false, "n": 98, "n_by_type": { "choice": 36, "noul": 28, "score": 34 }, "score": 0.0 }, "fairness": { "accuracy": 0.5555555555555556, "cc": 21.924596601307183, "cc_by_type": { "choice": 52.94117647058823, "noul": 0.0, "score": 12.83261333333332 }, "low_n": true, "n": 27, "n_by_type": { "choice": 12, "noul": 6, "score": 9 }, "score": 21.924596601307183 }, "jaggedness": { "accuracy": 0.2756756756756757, "cc": -23.766898037764125, "cc_by_type": { "choice": 9.680407596109312, "noul": -79.48717948717949, "score": -1.4939222222221993 }, "low_n": false, "n": 185, "n_by_type": { "choice": 97, "noul": 39, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.4583333333333333, "cc": 21.153469908954868, "cc_by_type": { "choice": 26.61402693915468, "noul": 1.3698630136986356, "score": 35.47651977401129 }, "low_n": false, "n": 264, "n_by_type": { "choice": 131, "noul": 73, "score": 60 }, "score": 21.153469908954868 }, "long_state": { "accuracy": 0.0, "cc": -110.31746031746032, "cc_by_type": { "choice": -30.952380952380945, "noul": -100.0, "score": -200.0 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 11, "score": 7 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.25806451612903225, "cc": -38.646075919732446, "cc_by_type": { "choice": 26.153846153846153, "noul": -100.0, "score": -42.09207391304348 }, "low_n": false, "n": 31, "n_by_type": { "choice": 15, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.346031746031746, "cc": -10.63069831663855, "cc_by_type": { "choice": 30.138826690550832, "noul": -58.62068965517242, "score": -3.4102319852940655 }, "low_n": false, "n": 315, "n_by_type": { "choice": 146, "noul": 87, "score": 82 }, "score": 0.0 }, "negation_pair_gen": { "accuracy": 0.26785714285714285, "cc": -26.649703571428574, "cc_by_type": { "choice": 14.285714285714286, "noul": -90.0, "score": -4.234825000000009 }, "low_n": false, "n": 56, "n_by_type": { "choice": 28, "noul": 20, "score": 8 }, "score": 0.0 }, "option_order": { "accuracy": 0.17543859649122806, "cc": -4.328523862375131, "cc_by_type": { "choice": -4.328523862375131 }, "low_n": false, "n": 57, "n_by_type": { "choice": 57 }, "score": 0.0 }, "perturb_pair": { "accuracy": 0.5166666666666667, "cc": 19.74922901184277, "cc_by_type": { "choice": 41.02228047182176, "noul": -28.57142857142857, "score": 46.79683513513512 }, "low_n": false, "n": 60, "n_by_type": { "choice": 34, "noul": 14, "score": 12 }, "score": 19.74922901184277 }, "usecase": { "accuracy": 0.3206751054852321, "cc": -16.990773799131347, "cc_by_type": { "choice": 27.153594141453095, "noul": -75.43859649122807, "score": -2.6873190476190745 }, "low_n": false, "n": 237, "n_by_type": { "choice": 111, "noul": 57, "score": 69 }, "score": 0.0 } }, "lang": { "ar": { "accuracy": 0.36363636363636365, "cc": -27.022149999999996, "cc_by_type": { "choice": 20.000000000000004, "noul": -50.0, "score": -51.06644999999999 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 4, "score": 2 }, "score": 0.0 }, "cs": { "accuracy": 0.5, "cc": -12.711864406779661, "cc_by_type": { "choice": 74.57627118644068, "noul": -100.0 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 3 }, "score": 0.0 }, "da": { "accuracy": 0.20833333333333334, "cc": -32.610591814595644, "cc_by_type": { "choice": 5.325443786982243, "noul": -100.0, "score": -3.1572192307691926 }, "low_n": true, "n": 24, "n_by_type": { "choice": 11, "noul": 5, "score": 8 }, "score": 0.0 }, "de": { "accuracy": 0.4666666666666667, "cc": 2.2274148960461893, "cc_by_type": { "choice": 40.25347012673505, "noul": -33.333333333333336, "score": -0.23789210526314886 }, "low_n": false, "n": 45, "n_by_type": { "choice": 24, "noul": 9, "score": 12 }, "score": 2.2274148960461893 }, "el": { "accuracy": 0.2222222222222222, "cc": -51.29191646586346, "cc_by_type": { "choice": 15.662650602409636, "noul": -100.0, "score": -69.53840000000002 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 5, "score": 2 }, "score": 0.0 }, "en": { "accuracy": 0.34405719392314565, "cc": -11.632396312202536, "cc_by_type": { "choice": 22.602355580482268, "noul": -61.654135338345874, "score": 4.154590821255999 }, "low_n": false, "n": 1119, "n_by_type": { "choice": 577, "noul": 266, "score": 276 }, "score": 0.0 }, "es": { "accuracy": 0.5, "cc": 13.68897878787879, "cc_by_type": { "choice": 60.72727272727273, "noul": -27.27272727272727, "score": 7.612390909090916 }, "low_n": true, "n": 28, "n_by_type": { "choice": 10, "noul": 11, "score": 7 }, "score": 13.68897878787879 }, "fi": { "accuracy": 0.2222222222222222, "cc": -42.25352112676057, "cc_by_type": { "choice": 15.49295774647887, "noul": -100.0 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 3 }, "score": 0.0 }, "fr": { "accuracy": 0.3333333333333333, "cc": -15.758361508496193, "cc_by_type": { "choice": 74.61212976022568, "noul": -100.0, "score": -21.887214285714272 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 4, "score": 6 }, "score": 0.0 }, "hi": { "accuracy": 0.3, "cc": -16.665771829710142, "cc_by_type": { "choice": 13.043478260869566, "noul": -60.0, "score": -3.0407937499999926 }, "low_n": false, "n": 30, "n_by_type": { "choice": 11, "noul": 10, "score": 9 }, "score": 0.0 }, "id": { "accuracy": 0.0, "cc": -64.51612903225806, "cc_by_type": { "choice": -29.032258064516125, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 0.0 }, "it": { "accuracy": 0.6470588235294118, "cc": 27.99738442348378, "cc_by_type": { "choice": 49.22425952045134, "noul": 0.0, "score": 34.76789375 }, "low_n": true, "n": 17, "n_by_type": { "choice": 5, "noul": 4, "score": 8 }, "score": 27.99738442348378 }, "ja": { "accuracy": 0.3125, "cc": -23.373968770567163, "cc_by_type": { "choice": 18.69918699186992, "noul": -71.42857142857143, "score": -17.39252187499998 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "ko": { "accuracy": 0.2727272727272727, "cc": -31.130543497267748, "cc_by_type": { "choice": 34.42622950819673, "noul": -100.0, "score": -27.81785999999997 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.36363636363636365, "cc": -3.6486553113553115, "cc_by_type": { "choice": 34.06593406593406, "noul": -50.0, "score": 4.988100000000006 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 0.0 }, "nl": { "accuracy": 0.46153846153846156, "cc": -0.1424452553389107, "cc_by_type": { "choice": 49.860724233983284, "noul": 0.0, "score": -50.288060000000016 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 6, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.42857142857142855, "cc": 0.5464480874316955, "cc_by_type": { "choice": 34.42622950819673, "noul": -33.333333333333336 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 3 }, "score": 0.5464480874316955 }, "pl": { "accuracy": 0.4, "cc": 12.299376097900057, "cc_by_type": { "choice": 60.64713598600787, "noul": -19.999999999999996, "score": -3.7490076923077043 }, "low_n": false, "n": 40, "n_by_type": { "choice": 16, "noul": 10, "score": 14 }, "score": 12.299376097900057 }, "pt": { "accuracy": 0.5, "cc": 26.514546072026793, "cc_by_type": { "choice": 45.7286432160804, "noul": 0.0, "score": 33.81499499999999 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 6, "score": 5 }, "score": 26.514546072026793 }, "sv": { "accuracy": 0.0, "cc": -56.24097777777778, "cc_by_type": { "choice": -33.333333333333336, "noul": -100.0, "score": -35.3896 }, "low_n": true, "n": 8, "n_by_type": { "choice": 3, "noul": 4, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.45454545454545453, "cc": 0.7387181286549614, "cc_by_type": { "choice": 47.368421052631575, "noul": 33.33333333333333, "score": -78.48560000000002 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 3, "score": 3 }, "score": 0.7387181286549614 }, "uk": { "accuracy": 0.09090909090909091, "cc": -46.02425072463768, "cc_by_type": { "choice": -8.695652173913041, "noul": -100.0, "score": -29.3771 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 1, "score": 4 }, "score": 0.0 }, "zh": { "accuracy": 0.125, "cc": -50.39295435435435, "cc_by_type": { "choice": -29.729729729729726, "noul": -33.333333333333336, "score": -88.11580000000001 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 0.0 } }, "type": { "choice": { "accuracy": 0.41733333333333333, "cc": 24.512005527724998, "cc_by_type": { "choice": 24.512005527724998 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 24.512005527724998 }, "noul": { "accuracy": 0.21066666666666667, "cc": -57.86666666666667, "cc_by_type": { "noul": -57.86666666666667 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.3466666666666667, "cc": 0.9913860770578653, "cc_by_type": { "score": 0.9913860770578653 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.9913860770578653 } }, "usecase": { "advertising": { "accuracy": 0.3333333333333333, "cc": -20.792305967417175, "cc_by_type": { "choice": 30.807248764415156, "noul": -60.0, "score": -33.184166666666684 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.16666666666666666, "cc": -51.04649872817785, "cc_by_type": { "choice": -8.01963993453355, "noul": -60.0, "score": -85.11985625 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 0.0 }, "customer_support": { "accuracy": 0.4, "cc": 7.8360928866832085, "cc_by_type": { "choice": 35.483870967741936, "noul": -50.0, "score": 38.02440769230768 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 8, "score": 4 }, "score": 7.8360928866832085 }, "demand_forecasting": { "accuracy": 0.23076923076923078, "cc": -39.86741871729342, "cc_by_type": { "choice": -8.43373493975904, "noul": -100.0, "score": -11.168521212121219 }, "low_n": true, "n": 26, "n_by_type": { "choice": 11, "noul": 4, "score": 11 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.2777777777777778, "cc": -17.362241654722098, "cc_by_type": { "choice": 44.099378881987576, "noul": -100.0, "score": 3.8138961538461302 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 5, "score": 6 }, "score": 0.0 }, "feature_extraction": { "accuracy": 0.1724137931034483, "cc": -29.604258430458433, "cc_by_type": { "choice": -2.564102564102566, "noul": -60.0, "score": -26.248672727272737 }, "low_n": true, "n": 29, "n_by_type": { "choice": 15, "noul": 5, "score": 9 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.38461538461538464, "cc": -8.077263014492752, "cc_by_type": { "choice": 47.82608695652174, "noul": -100.0, "score": 27.942124 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 0.0 }, "gaming": { "accuracy": 0.27586206896551724, "cc": -33.71579672167217, "cc_by_type": { "choice": 16.831683168316832, "noul": -100.0, "score": -17.97907333333333 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 7, "score": 11 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.14285714285714285, "cc": -70.40589841269842, "cc_by_type": { "choice": 4.761904761904763, "noul": -100.0, "score": -115.9796 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 5, "score": 1 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.21739130434782608, "cc": -16.350229066353496, "cc_by_type": { "choice": -9.923664122137408, "noul": -60.0, "score": 20.872976923076926 }, "low_n": true, "n": 23, "n_by_type": { "choice": 14, "noul": 5, "score": 4 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.36, "cc": -2.498773089430904, "cc_by_type": { "choice": 83.73983739837398, "noul": -75.0, "score": -16.23615666666669 }, "low_n": true, "n": 25, "n_by_type": { "choice": 8, "noul": 8, "score": 9 }, "score": 0.0 }, "legal_compliance": { "accuracy": 0.38461538461538464, "cc": -26.543207810993252, "cc_by_type": { "choice": 34.42622950819673, "noul": -100.0, "score": -14.055852941176482 }, "low_n": true, "n": 13, "n_by_type": { "choice": 8, "noul": 1, "score": 4 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.3793103448275862, "cc": -5.703711032698185, "cc_by_type": { "choice": 44.95412844036697, "noul": -50.0, "score": -12.065261538461524 }, "low_n": true, "n": 29, "n_by_type": { "choice": 14, "noul": 8, "score": 7 }, "score": 0.0 }, "model_routing": { "accuracy": 0.5, "cc": 20.346316406896474, "cc_by_type": { "choice": 42.106673813990895, "noul": 9.090909090909083, "score": 9.841366315789447 }, "low_n": false, "n": 150, "n_by_type": { "choice": 74, "noul": 44, "score": 32 }, "score": 20.346316406896474 }, "moderation_trust_safety": { "accuracy": 0.3125, "cc": -21.78233634085213, "cc_by_type": { "choice": 36.84210526315789, "noul": -100.0, "score": -2.1891142857142887 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 5, "score": 3 }, "score": 0.0 }, "recruiting": { "accuracy": 0.38095238095238093, "cc": -7.3411204741657015, "cc_by_type": { "choice": 54.77386934673366, "noul": -100.0, "score": 23.20276923076924 }, "low_n": true, "n": 21, "n_by_type": { "choice": 9, "noul": 6, "score": 6 }, "score": 0.0 }, "risk_assessment": { "accuracy": 0.5384615384615384, "cc": 28.47671559224318, "cc_by_type": { "choice": 62.264150943396224, "noul": 33.33333333333333, "score": -10.167337500000006 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 3, "score": 3 }, "score": 28.47671559224318 }, "scientific_discovery": { "accuracy": 0.21875, "cc": -30.3334952412572, "cc_by_type": { "choice": 12.350597609561751, "noul": -100.0, "score": -3.35108333333336 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.18181818181818182, "cc": -40.61265147175815, "cc_by_type": { "choice": 14.08114558472553, "noul": -100.0, "score": -35.91909999999998 }, "low_n": true, "n": 22, "n_by_type": { "choice": 9, "noul": 6, "score": 7 }, "score": 0.0 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 71.00609920538326, 79.69915502973062 ], "I": [ 6.686984873558308, 11.785980151765898 ], "capability": [ 39.28660140967173, 45.042438645829456 ], "composite": { "A": [ 0.3773769198315772, 1.786744528078155 ], "B": [ 0.2637088587675396, 1.3280900163194176 ], "C": [ 0.26206730543859524, 1.2407948111653853 ] }, "seed": 17 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 10.073244581339713, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 77.50040036924202, "capability": 43.05068960177584, "intelligence": 8.600978834309654 }, "per_type": { "choice": { "accuracy": 0.41733333333333333, "calibration": 82.72166309148264, "cc": 24.512005527724998, "cc_by_type": { "choice": 24.512005527724998 }, "cc_tiered": 24.9036787020387, "cc_tiered_open": 25.412389147780978, "cc_tiered_sealed": 24.394968256296423, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 24.512005527724998 }, "noul": { "accuracy": 0.21066666666666667, "calibration": 75.38036703296703, "cc": -57.86666666666667, "cc_by_type": { "noul": -57.86666666666667 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.3466666666666667, "calibration": 74.39917098327638, "cc": 0.9913860770578653, "cc_by_type": { "score": 0.9913860770578653 }, "cc_tiered": 0.8992578008902634, "cc_tiered_open": 1.7985156017805268, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.9913860770578653 } }, "raw_on_A": { "calibration": 78.45530493357067, "capability": 43.54816426637622, "intelligence": 8.641023599181757 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.014666666666666717, "n": 750 }, "noul": { "invalid_rate": 0.029333333333333322, "n": 375 }, "score": { "invalid_rate": 0.01866666666666672, "n": 375 } }, "views": { "Balanced 33:33:33": 0.6218800588368827, "Emphasis on Accuracy 60:20:20": 0.39424898336691744, "Emphasis on Cost 20:20:60": 0.8792467571253542, "Emphasis on Speed 20:60:20": 0.8693565458528719, "Intelligence only": 0.25450928339078893 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Liquid AI", "axes": { "calibration": 76.87411794897184, "cost": 50.56108621702079, "intelligence": 22.13919485880221, "speed": 94.32773191369529 }, "calibration": { "parts": { "choice": { "ece_hard": 0.11313979352292604, "ece_score": 77.37204129541479, "mean_tvd": null, "n": 750, "n_ece_hard": 328, "n_tvd": 0, "score": 77.37204129541479 }, "noul": { "abstention_rate": 0.5093333333333334, "brier": 0.19849004201230716, "ece": 0.11360395591990471, "n": 375, "score": 77.27920881601906 }, "score": { "interval90_coverage": 0.9093333333333333, "interval90_width": 0.5644444444444446, "n": 375, "nrps": 0.1509830793884296, "score": 75.9711037354817, "top_ece": 0.16479742295096816 } }, "score": 76.87411794897184 }, "capability": 49.50665640388703, "composite_ci95": { "A": [ 5.391616814571421, 12.175163701432744 ], "B": [ 4.327224359437512, 10.309753350985918 ], "C": [ 3.744178343452376, 8.454974792661627 ] }, "cost": { "basis": "ESTIMATE (*est.), size-class reference; self-hosted open weights, no bookable listing for this base. The declared base LiquidAI/LFM2.5-VL-3B is not a key of any frozen pricing table, and the frozen 25 Sep 2026 snapshot's only Liquid entry is liquid/lfm-2.5-2.6b:free at (0, 0) - a free route, which may not price a row (Florian, free tiers: use the published paid price or the estimate). Method, as aplomb-1, wald-4b-v2, jiwo-* and the ClassOne trio carry: the DeepInfra Qwen3.5 size-class reference from the frozen snapshot, rounded ONE CLASS UP so the estimate can only OVERSTATE this row's cost - deepinfra:Qwen/Qwen3.5-4B at USD 0.03 per 1M input and USD 0.15 per 1M output, for a 3B model. Applied to this row's own measured usage: 2,222,971 input tokens and 0 output tokens over 1,500 decisions. The 2B rate (0.02, 0.10) would give 0.029639613333333335 and was deliberately NOT used because it understates. Precedent for a LiquidAI base priced as a labelled estimate: the live lev-350m row (LiquidAI/LFM2.5-350M, in UNLISTED_BASE_MODELS, estimate 0.00462807881773399 per 1,000).", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.044459420000000006 }, "display": "Liquid AI d1-3B", "endpoint_condition": "evaluator-owned Lium RTX 5090 32 GB pod (France), offline after setup (egress cut and proven), author's custom code at the pinned commit via trust_remote_code from the sha256-verified local copy, bfloat16, no torch.compile", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 27.85582362507926, "I_sealed": 16.42256609252516, "base": 22.13919485880221, "excess": 1.3600129512143884, "gap": 11.433257532554101, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 53.72008003683186, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 72.75675675675676, "hard": 43.378420887071414, "judge": 60.44825313117996, "standard": 54.79280033486815 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 57.14285714285714, "hard": -30.434782608695656, "judge": -19.999999999999996, "standard": -11.111111111111116 } }, "open|score": { "cc": 29.847390838405925, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 36.55757331890299, "hard": 17.69560694707587, "judge": 41.70520484649972, "standard": 33.00914636867683 } }, "sealed|choice": { "cc": 39.056033872257295, "n": { "easy": 72, "hard": 260, "judge": 141, "standard": 127 }, "tiers": { "easy": 54.948378350192925, "hard": 33.21974277234252, "judge": 38.815889604789206, "standard": 43.14266023432116 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 39, "hard": 132, "judge": 72, "standard": 57 }, "tiers": { "easy": 7.692307692307687, "hard": -45.45454545454546, "judge": -5.555555555555558, "standard": -5.263157894736848 } }, "sealed|score": { "cc": 10.211664405318189, "n": { "easy": 33, "hard": 115, "judge": 75, "standard": 77 }, "tiers": { "easy": 27.92356315993724, "hard": 9.442062390485162, "judge": 0.13958879484807785, "standard": 18.003032473379875 } } } }, "jevbench_score": 8.855712627780655, "key": "liquid-d1-3b", "last_measured_on": "2026-10-10", "licence": "LFM Open License v1.0 (commercial use only below USD 10 M annual revenue)", "listing": "ranked", "measured_in": "v1.6.5", "measurement_date_status": "measured", "model_pin": "LiquidAI/d1-3B@051bcc464b01b9f92942b364d9586b0ef5912432", "noul_decisive": { "acc_among_decisive": 0.8315217391304348, "decisive": 184, "decisive_rate": 0.49066666666666664, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 4, "ranked": true, "ranks": { "A": 4, "B": 4, "C": 4, "capability": 5 }, "repo": "https://huggingface.co/LiquidAI/d1-3B", "scores": { "A": 8.855712627780655, "B": 7.330614913163864, "C": 6.149800435958788 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.1797857349505648, "p50_s_raw": 0.0148928674752824, "p95_s_adjusted": 0.20533906437922267, "p95_s_raw": 0.027669532189611334 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.7241379310344828, "cc": 54.390853941466354, "cc_by_type": { "choice": 75.0, "noul": 66.66666666666667, "score": 21.50589515773239 }, "low_n": true, "n": 29, "n_by_type": { "choice": 16, "noul": 6, "score": 7 }, "score": 54.390853941466354 }, "calibration_gen": { "accuracy": 0.45535714285714285, "cc": 7.885435170467154, "cc_by_type": { "choice": 41.770186335403736, "noul": 0.0, "score": -18.113880824002273 }, "low_n": false, "n": 112, "n_by_type": { "choice": 56, "noul": 26, "score": 30 }, "score": 7.885435170467154 }, "composite_gen": { "accuracy": 0.5408163265306123, "cc": 25.303042500575433, "cc_by_type": { "choice": 77.77777777777779, "noul": -21.42857142857143, "score": 19.559921152519944 }, "low_n": false, "n": 98, "n_by_type": { "choice": 36, "noul": 28, "score": 34 }, "score": 25.303042500575433 }, "fairness": { "accuracy": 0.8518518518518519, "cc": 47.39465841291355, "cc_by_type": { "choice": 88.23529411764707, "noul": 0.0, "score": 53.948681121093585 }, "low_n": true, "n": 27, "n_by_type": { "choice": 12, "noul": 6, "score": 9 }, "score": 47.39465841291355 }, "jaggedness": { "accuracy": 0.31891891891891894, "cc": -22.274181763566684, "cc_by_type": { "choice": 20.79666512274201, "noul": -79.48717948717949, "score": -8.13203092626258 }, "low_n": false, "n": 185, "n_by_type": { "choice": 97, "noul": 39, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.625, "cc": 51.3034512100294, "cc_by_type": { "choice": 47.05062703204832, "noul": 50.68493150684932, "score": 56.17479509119057 }, "low_n": false, "n": 264, "n_by_type": { "choice": 131, "noul": 73, "score": 60 }, "score": 51.3034512100294 }, "long_state": { "accuracy": 0.1724137931034483, "cc": -29.57896952664178, "cc_by_type": { "choice": 16.66666666666667, "noul": -100.0, "score": -5.403575246592007 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 11, "score": 7 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.2903225806451613, "cc": -13.862612846690979, "cc_by_type": { "choice": 7.692307692307688, "noul": -50.0, "score": 0.7198537676193717 }, "low_n": false, "n": 31, "n_by_type": { "choice": 15, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.41904761904761906, "cc": 4.0623722132052436, "cc_by_type": { "choice": 34.617107030900144, "noul": -33.333333333333336, "score": 10.903342942048921 }, "low_n": false, "n": 315, "n_by_type": { "choice": 146, "noul": 87, "score": 82 }, "score": 4.0623722132052436 }, "negation_pair_gen": { "accuracy": 0.44642857142857145, "cc": -3.4722633613639875, "cc_by_type": { "choice": 38.095238095238095, "noul": -50.0, "score": 1.4879718206699422 }, "low_n": false, "n": 56, "n_by_type": { "choice": 28, "noul": 20, "score": 8 }, "score": 0.0 }, "option_order": { "accuracy": 0.7017543859649122, "cc": 62.264150943396224, "cc_by_type": { "choice": 62.264150943396224 }, "low_n": false, "n": 57, "n_by_type": { "choice": 57 }, "score": 62.264150943396224 }, "perturb_pair": { "accuracy": 0.7166666666666667, "cc": 57.31023460546714, "cc_by_type": { "choice": 52.81782437745741, "noul": 42.85714285714286, "score": 76.25573658180112 }, "low_n": false, "n": 60, "n_by_type": { "choice": 34, "noul": 14, "score": 12 }, "score": 57.31023460546714 }, "usecase": { "accuracy": 0.39662447257383965, "cc": -1.9444617361056398, "cc_by_type": { "choice": 38.716515706301806, "noul": -43.85964912280702, "score": -0.6902517918117024 }, "low_n": false, "n": 237, "n_by_type": { "choice": 111, "noul": 57, "score": 69 }, "score": 0.0 } }, "lang": { "ar": { "accuracy": 0.18181818181818182, "cc": -100.14332542233501, "cc_by_type": { "choice": 20.000000000000004, "noul": -100.0, "score": -220.429976267005 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 4, "score": 2 }, "score": 0.0 }, "cs": { "accuracy": 0.375, "cc": -4.802259887005647, "cc_by_type": { "choice": 23.72881355932204, "noul": -33.333333333333336 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 3 }, "score": 0.0 }, "da": { "accuracy": 0.375, "cc": -8.98770932887382, "cc_by_type": { "choice": 17.159763313609464, "noul": -60.0, "score": 15.877108699769082 }, "low_n": true, "n": 24, "n_by_type": { "choice": 11, "noul": 5, "score": 8 }, "score": 0.0 }, "de": { "accuracy": 0.4222222222222222, "cc": 6.322815228633009, "cc_by_type": { "choice": 34.8219674109837, "noul": -11.111111111111116, "score": -4.742410613973558 }, "low_n": false, "n": 45, "n_by_type": { "choice": 24, "noul": 9, "score": 12 }, "score": 6.322815228633009 }, "el": { "accuracy": 0.2777777777777778, "cc": -39.39660414361751, "cc_by_type": { "choice": 15.662650602409636, "noul": -100.0, "score": -33.85246303326217 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 5, "score": 2 }, "score": 0.0 }, "en": { "accuracy": 0.4924039320822163, "cc": 14.003345625582787, "cc_by_type": { "choice": 44.13909141895677, "noul": -18.045112781954884, "score": 15.916058239746478 }, "low_n": false, "n": 1119, "n_by_type": { "choice": 577, "noul": 266, "score": 276 }, "score": 14.003345625582787 }, "es": { "accuracy": 0.6428571428571429, "cc": 33.758516235110754, "cc_by_type": { "choice": 73.81818181818181, "noul": 27.27272727272727, "score": 0.18463961442317656 }, "low_n": true, "n": 28, "n_by_type": { "choice": 10, "noul": 11, "score": 7 }, "score": 33.758516235110754 }, "fi": { "accuracy": 0.0, "cc": -63.38028169014085, "cc_by_type": { "choice": -26.760563380281692, "noul": -100.0 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 3 }, "score": 0.0 }, "fr": { "accuracy": 0.6, "cc": 28.727877312438604, "cc_by_type": { "choice": 100.00000000000001, "noul": -50.0, "score": 36.183631937315795 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 4, "score": 6 }, "score": 28.727877312438604 }, "hi": { "accuracy": 0.3, "cc": 1.3860870885139314, "cc_by_type": { "choice": 13.043478260869566, "noul": -19.999999999999996, "score": 11.114783004672224 }, "low_n": false, "n": 30, "n_by_type": { "choice": 11, "noul": 10, "score": 9 }, "score": 1.3860870885139314 }, "id": { "accuracy": 0.5, "cc": 17.741935483870968, "cc_by_type": { "choice": 35.483870967741936, "noul": 0.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 17.741935483870968 }, "it": { "accuracy": 0.6470588235294118, "cc": 40.934769362003244, "cc_by_type": { "choice": 49.22425952045134, "noul": 50.0, "score": 23.5800485655584 }, "low_n": true, "n": 17, "n_by_type": { "choice": 5, "noul": 4, "score": 8 }, "score": 40.934769362003244 }, "ja": { "accuracy": 0.34375, "cc": -7.243137787326186, "cc_by_type": { "choice": 18.69918699186992, "noul": -42.85714285714286, "score": 2.428542503294384 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "ko": { "accuracy": 0.5454545454545454, "cc": 21.42666905349395, "cc_by_type": { "choice": 34.42622950819673, "noul": 0.0, "score": 29.853777652285117 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 2, "score": 5 }, "score": 21.42666905349395 }, "mixed": { "accuracy": 0.7272727272727273, "cc": 49.321740609526785, "cc_by_type": { "choice": 78.02197802197803, "noul": 0.0, "score": 69.94324380660233 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 49.321740609526785 }, "nl": { "accuracy": 0.38461538461538464, "cc": 19.828553848785166, "cc_by_type": { "choice": -0.2785515320334245, "noul": 0.0, "score": 59.76421307838892 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 6, "score": 2 }, "score": 19.828553848785166 }, "no": { "accuracy": 0.42857142857142855, "cc": 17.486338797814206, "cc_by_type": { "choice": 1.6393442622950836, "noul": 33.33333333333333 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 3 }, "score": 17.486338797814206 }, "pl": { "accuracy": 0.525, "cc": 29.992507104770652, "cc_by_type": { "choice": 84.25885439440316, "noul": 0.0, "score": 5.718666919908799 }, "low_n": false, "n": 40, "n_by_type": { "choice": 16, "noul": 10, "score": 14 }, "score": 29.992507104770652 }, "pt": { "accuracy": 0.6666666666666666, "cc": 54.789361628198456, "cc_by_type": { "choice": 63.81909547738693, "noul": 33.33333333333333, "score": 67.2156560738751 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 6, "score": 5 }, "score": 54.789361628198456 }, "sv": { "accuracy": 0.0, "cc": -60.92596627415683, "cc_by_type": { "choice": -33.333333333333336, "noul": -100.0, "score": -49.44456548913716 }, "low_n": true, "n": 8, "n_by_type": { "choice": 3, "noul": 4, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.45454545454545453, "cc": -25.14391404022173, "cc_by_type": { "choice": 73.6842105263158, "noul": -33.333333333333336, "score": -115.78261931364766 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 3, "score": 3 }, "score": 0.0 }, "uk": { "accuracy": 0.5454545454545454, "cc": -3.966886884613037, "cc_by_type": { "choice": 34.78260869565218, "noul": -100.0, "score": 53.3167306505087 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 1, "score": 4 }, "score": 0.0 }, "zh": { "accuracy": 0.5, "cc": 27.7843293385466, "cc_by_type": { "choice": 2.702702702702704, "noul": 33.33333333333333, "score": 47.31695197960376 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 27.7843293385466 } }, "type": { "choice": { "accuracy": 0.5546666666666666, "cc": 42.30437035757471, "cc_by_type": { "choice": 42.30437035757471 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 42.30437035757471 }, "noul": { "accuracy": 0.408, "cc": -18.400000000000006, "cc_by_type": { "noul": -18.400000000000006 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.4026666666666667, "cc": 14.785081876637218, "cc_by_type": { "score": 14.785081876637218 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 14.785081876637218 } }, "usecase": { "advertising": { "accuracy": 0.4166666666666667, "cc": -21.322066560908976, "cc_by_type": { "choice": 50.57660626029654, "noul": -60.0, "score": -54.542805943023474 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.25, "cc": -20.883079422436126, "cc_by_type": { "choice": 1.8003273322422266, "noul": -19.999999999999996, "score": -44.4495655995506 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 0.0 }, "customer_support": { "accuracy": 0.55, "cc": 24.46389210511948, "cc_by_type": { "choice": 83.87096774193549, "noul": -50.0, "score": 39.52070857342296 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 8, "score": 4 }, "score": 24.46389210511948 }, "demand_forecasting": { "accuracy": 0.23076923076923078, "cc": -33.81567016039666, "cc_by_type": { "choice": -8.43373493975904, "noul": -100.0, "score": 6.986724458569061 }, "low_n": true, "n": 26, "n_by_type": { "choice": 11, "noul": 4, "score": 11 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.3888888888888889, "cc": 4.454876806713197, "cc_by_type": { "choice": 44.099378881987576, "noul": -60.0, "score": 29.265251538152015 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 5, "score": 6 }, "score": 4.454876806713197 }, "feature_extraction": { "accuracy": 0.3103448275862069, "cc": -6.573360507662842, "cc_by_type": { "choice": 5.9829059829059785, "noul": -19.999999999999996, "score": -5.702987505894508 }, "low_n": true, "n": 29, "n_by_type": { "choice": 15, "noul": 5, "score": 9 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.46153846153846156, "cc": 16.612884611673458, "cc_by_type": { "choice": 47.82608695652174, "noul": 25.0, "score": -22.987433121501375 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 16.612884611673458 }, "gaming": { "accuracy": 0.41379310344827586, "cc": -6.703885975467817, "cc_by_type": { "choice": 40.59405940594059, "noul": -71.42857142857143, "score": 10.722854096227385 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 7, "score": 11 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.21428571428571427, "cc": -33.44343074859378, "cc_by_type": { "choice": 20.634920634920636, "noul": -100.0, "score": -20.965212880701966 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 5, "score": 1 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.17391304347826086, "cc": -31.072633553703692, "cc_by_type": { "choice": -0.7633587786259586, "noul": -100.0, "score": 7.5454581175148805 }, "low_n": true, "n": 23, "n_by_type": { "choice": 14, "noul": 5, "score": 4 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.32, "cc": -1.078959073185669, "cc_by_type": { "choice": 67.47967479674797, "noul": -75.0, "score": 4.283447983695021 }, "low_n": true, "n": 25, "n_by_type": { "choice": 8, "noul": 8, "score": 9 }, "score": 0.0 }, "legal_compliance": { "accuracy": 0.15384615384615385, "cc": -23.678295476921054, "cc_by_type": { "choice": -14.754098360655735, "noul": -100.0, "score": 43.719211929892566 }, "low_n": true, "n": 13, "n_by_type": { "choice": 8, "noul": 1, "score": 4 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.4827586206896552, "cc": 15.868347401942737, "cc_by_type": { "choice": 35.77981651376147, "noul": 25.0, "score": -13.174774307933257 }, "low_n": true, "n": 29, "n_by_type": { "choice": 14, "noul": 8, "score": 7 }, "score": 15.868347401942737 }, "model_routing": { "accuracy": 0.6866666666666666, "cc": 57.504158219489966, "cc_by_type": { "choice": 63.012597158938625, "noul": 59.09090909090908, "score": 50.40896840862219 }, "low_n": false, "n": 150, "n_by_type": { "choice": 74, "noul": 44, "score": 32 }, "score": 57.504158219489966 }, "moderation_trust_safety": { "accuracy": 0.4375, "cc": 7.985104053150214, "cc_by_type": { "choice": 36.84210526315789, "noul": -60.0, "score": 47.113206896292745 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 5, "score": 3 }, "score": 7.985104053150214 }, "recruiting": { "accuracy": 0.38095238095238093, "cc": 0.5006746462399576, "cc_by_type": { "choice": 39.69849246231156, "noul": -66.66666666666667, "score": 28.470198143074978 }, "low_n": true, "n": 21, "n_by_type": { "choice": 9, "noul": 6, "score": 6 }, "score": 0.5006746462399576 }, "risk_assessment": { "accuracy": 0.46153846153846156, "cc": 6.972667050442593, "cc_by_type": { "choice": 43.39622641509434, "noul": -33.333333333333336, "score": 10.855108069566777 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 3, "score": 3 }, "score": 6.972667050442593 }, "scientific_discovery": { "accuracy": 0.25, "cc": -22.282698718421653, "cc_by_type": { "choice": 12.350597609561751, "noul": -71.42857142857143, "score": -7.77012233625527 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.45454545454545453, "cc": -9.604825720085183, "cc_by_type": { "choice": 57.04057279236276, "noul": -100.0, "score": 14.144950047381688 }, "low_n": true, "n": 22, "n_by_type": { "choice": 9, "noul": 6, "score": 7 }, "score": 0.0 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 71.24744830586958, 79.33703742189459 ], "I": [ 18.2163502770018, 25.174206645143013 ], "capability": [ 45.67389319670205, 51.463838439947395 ], "composite": { "A": [ 5.391616814571421, 12.175163701432744 ], "B": [ 4.327224359437512, 10.309753350985918 ], "C": [ 3.744178343452376, 8.454974792661627 ] }, "seed": 18 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 10.073244581339713, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 76.87411794897184, "capability": 49.50665640388703, "intelligence": 22.13919485880221 }, "per_type": { "choice": { "accuracy": 0.5546666666666666, "calibration": 77.37204129541479, "cc": 42.30437035757471, "cc_by_type": { "choice": 42.30437035757471 }, "cc_tiered": 46.38805695454458, "cc_tiered_open": 53.72008003683186, "cc_tiered_sealed": 39.056033872257295, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 42.30437035757471 }, "noul": { "accuracy": 0.408, "calibration": 77.27920881601906, "cc": -18.400000000000006, "cc_by_type": { "noul": -18.400000000000006 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.4026666666666667, "calibration": 75.9711037354817, "cc": 14.785081876637218, "cc_by_type": { "score": 14.785081876637218 }, "cc_tiered": 20.02952762186206, "cc_tiered_open": 29.847390838405925, "cc_tiered_sealed": 10.211664405318189, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 14.785081876637218 } }, "raw_on_A": { "calibration": 72.6387411013128, "capability": 46.47345221333068, "intelligence": 20.308163325348563 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 7.785401938124825, "Emphasis on Accuracy 60:20:20": 5.909418623202946, "Emphasis on Cost 20:20:60": 8.516521034074662, "Emphasis on Speed 20:60:20": 10.132088187487657, "Intelligence only": 4.340556958274228 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Liquid AI", "axes": { "calibration": 62.32062493293682, "cost": 73.1039158657426, "intelligence": 3.2615852143588597, "speed": 94.0744546594334 }, "calibration": { "parts": { "choice": { "ece_hard": 0.22880094970871764, "ece_score": 54.23981005825647, "mean_tvd": null, "n": 750, "n_ece_hard": 328, "n_tvd": 0, "score": 54.23981005825647 }, "noul": { "abstention_rate": 0.7333333333333334, "brier": 0.2769707015707441, "ece": 0.1533565516303061, "n": 375, "score": 69.32868967393878 }, "score": { "interval90_coverage": 0.8293333333333334, "interval90_width": 0.6586666666666664, "n": 375, "nrps": 0.21825554880108286, "score": 63.39337506661522, "top_ece": 0.25693847493330635 } }, "score": 62.32062493293682 }, "capability": 32.79110507364784, "composite_ci95": { "A": [ 0.0050628325646249585, 0.3450163793532752 ], "B": [ 0.003260750472245594, 0.24100666656993436 ], "C": [ 0.003515855947656221, 0.239594707884219 ] }, "cost": { "basis": "ESTIMATE (*est.), size-class reference; self-hosted open weights, no bookable listing for this base. The declared text base LiquidAI/LFM2.5-Encoder-350M is not a key of any frozen pricing table. Method as above: deepinfra:Qwen/Qwen3.5-0.8B at USD 0.01 per 1M input and USD 0.05 per 1M output - the class above this 600M model - applied to its own measured usage, 1,182,027 input tokens and 0 output tokens over 1,500 decisions. The frozen base-size ENCODER rate (0.005, 0.0), which the published tacet-sonata and lev-350m rows carry, would give 0.00394009 and was deliberately NOT used: it prices a text-only encoder, while d1-omni also runs SigLIP2 vision and FastConformer audio towers, so the class above is the conservative choice and keeps the two Liquid rows on one method.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.00788018 }, "display": "Liquid AI d1-omni-600M", "endpoint_condition": "evaluator-owned Lium RTX A6000 48 GB pod (United States), offline after setup (egress cut and proven), author's custom code at the pinned commit via trust_remote_code from the sha256-verified local copy, float16, states cut on the right at 16,384 tokens by the model code", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 4.293220672965477, "I_sealed": 2.229949755752243, "base": 3.2615852143588597, "excess": -8.009973664126479, "gap": 2.063270917213234, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 12.87966201889643, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 10.486486486486504, "hard": 3.743315508021402, "judge": 12.986156888595916, "standard": 32.18920050230222 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": -28.57142857142857, "hard": -73.91304347826086, "judge": -100.0, "standard": -66.66666666666667 } }, "open|score": { "cc": 0.0, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 12.092551202372192, "hard": -10.469003850745073, "judge": 19.089799023912114, "standard": -20.24343611383641 } }, "sealed|choice": { "cc": 6.689849267256729, "n": { "easy": 72, "hard": 260, "judge": 141, "standard": 127 }, "tiers": { "easy": 2.388153092084683, "hard": 3.5396284489392005, "judge": 5.9408452133326675, "standard": 16.2646450723639 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 39, "hard": 132, "judge": 72, "standard": 57 }, "tiers": { "easy": -64.1025641025641, "hard": -72.72727272727273, "judge": -72.22222222222221, "standard": -64.91228070175438 } }, "sealed|score": { "cc": 0.0, "n": { "easy": 33, "hard": 115, "judge": 75, "standard": 77 }, "tiers": { "easy": -14.21064395948739, "hard": -26.382300409685588, "judge": -13.560363505639916, "standard": -9.691900455584058 } } } }, "jevbench_score": 0.049057448753865106, "key": "liquid-d1-omni-600m", "last_measured_on": "2026-10-10", "licence": "LFM Open License v1.0 (commercial use only below USD 10 M annual revenue)", "listing": "ranked", "measured_in": "v1.6.5", "measurement_date_status": "measured", "model_pin": "LiquidAI/d1-omni-600M@02b55d7076f15129e59ab3f94783f32c4b088674", "noul_decisive": { "acc_among_decisive": 0.56, "decisive": 100, "decisive_rate": 0.26666666666666666, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 8, "ranked": true, "ranks": { "A": 8, "B": 8, "C": 8, "capability": 9 }, "repo": "https://huggingface.co/LiquidAI/d1-omni-600M", "scores": { "A": 0.049057448753865106, "B": 0.032554129042754114, "C": 0.03406767274573965 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.19256855882704257, "p50_s_raw": 0.02128427941352129, "p95_s_adjusted": 0.20322126662358642, "p95_s_raw": 0.026610633311793207 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.20689655172413793, "cc": -29.121942106010263, "cc_by_type": { "choice": -8.333333333333334, "noul": -100.0, "score": 20.967507015302544 }, "low_n": true, "n": 29, "n_by_type": { "choice": 16, "noul": 6, "score": 7 }, "score": 0.0 }, "calibration_gen": { "accuracy": 0.1875, "cc": -48.670081675280734, "cc_by_type": { "choice": -0.1552795031055859, "noul": -76.92307692307692, "score": -68.93188859965969 }, "low_n": false, "n": 112, "n_by_type": { "choice": 56, "noul": 26, "score": 30 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.22448979591836735, "cc": -44.09414082419346, "cc_by_type": { "choice": 11.111111111111109, "noul": -78.57142857142857, "score": -64.82210501226291 }, "low_n": false, "n": 98, "n_by_type": { "choice": 36, "noul": 28, "score": 34 }, "score": 0.0 }, "fairness": { "accuracy": 0.2222222222222222, "cc": -39.3458024428534, "cc_by_type": { "choice": 29.411764705882355, "noul": -100.0, "score": -47.44917203444254 }, "low_n": true, "n": 27, "n_by_type": { "choice": 12, "noul": 6, "score": 9 }, "score": 0.0 }, "jaggedness": { "accuracy": 0.2918918918918919, "cc": -18.904988531894233, "cc_by_type": { "choice": 11.069939786938404, "noul": -64.1025641025641, "score": -3.682341280057 }, "low_n": false, "n": 185, "n_by_type": { "choice": 97, "noul": 39, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.3446969696969697, "cc": 1.1432998727522918, "cc_by_type": { "choice": 15.466790524849063, "noul": -28.767123287671236, "score": 16.73023238107905 }, "low_n": false, "n": 264, "n_by_type": { "choice": 131, "noul": 73, "score": 60 }, "score": 1.1432998727522918 }, "long_state": { "accuracy": 0.27586206896551724, "cc": -25.065654575351683, "cc_by_type": { "choice": 16.66666666666667, "noul": -100.0, "score": 8.136369607278283 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 11, "score": 7 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.22580645161290322, "cc": -33.79734650736005, "cc_by_type": { "choice": 7.692307692307688, "noul": -100.0, "score": -9.084347214387822 }, "low_n": false, "n": 31, "n_by_type": { "choice": 15, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.2, "cc": -31.03237123270632, "cc_by_type": { "choice": -1.2091356918943095, "noul": -93.10344827586206, "score": 1.2154702696374198 }, "low_n": false, "n": 315, "n_by_type": { "choice": 146, "noul": 87, "score": 82 }, "score": 0.0 }, "negation_pair_gen": { "accuracy": 0.16071428571428573, "cc": -85.43494727513347, "cc_by_type": { "choice": -4.761904761904764, "noul": -90.0, "score": -161.54293706349563 }, "low_n": false, "n": 56, "n_by_type": { "choice": 28, "noul": 20, "score": 8 }, "score": 0.0 }, "option_order": { "accuracy": 0.2807017543859649, "cc": 8.99001109877914, "cc_by_type": { "choice": 8.99001109877914 }, "low_n": false, "n": 57, "n_by_type": { "choice": 57 }, "score": 8.99001109877914 }, "perturb_pair": { "accuracy": 0.31666666666666665, "cc": -11.077733022650753, "cc_by_type": { "choice": 21.36304062909567, "noul": -85.71428571428572, "score": 31.1180460172378 }, "low_n": false, "n": 60, "n_by_type": { "choice": 34, "noul": 14, "score": 12 }, "score": 0.0 }, "usecase": { "accuracy": 0.2320675105485232, "cc": -25.394686423610008, "cc_by_type": { "choice": 5.184043168240532, "noul": -57.89473684210527, "score": -23.47336559696529 }, "low_n": false, "n": 237, "n_by_type": { "choice": 111, "noul": 57, "score": 69 }, "score": 0.0 } }, "lang": { "ar": { "accuracy": 0.09090909090909091, "cc": -53.76070940660107, "cc_by_type": { "choice": -33.333333333333336, "noul": -100.0, "score": -27.948794886469862 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 4, "score": 2 }, "score": 0.0 }, "cs": { "accuracy": 0.0, "cc": -63.559322033898304, "cc_by_type": { "choice": -27.11864406779661, "noul": -100.0 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 3 }, "score": 0.0 }, "da": { "accuracy": 0.375, "cc": -22.128997159813196, "cc_by_type": { "choice": 17.159763313609464, "noul": -100.0, "score": 16.45324520695096 }, "low_n": true, "n": 24, "n_by_type": { "choice": 11, "noul": 5, "score": 8 }, "score": 0.0 }, "de": { "accuracy": 0.26666666666666666, "cc": -26.699211006624235, "cc_by_type": { "choice": 7.6644538322269185, "noul": -77.77777777777779, "score": -9.984309074321839 }, "low_n": false, "n": 45, "n_by_type": { "choice": 24, "noul": 9, "score": 12 }, "score": 0.0 }, "el": { "accuracy": 0.16666666666666666, "cc": -44.641658892653076, "cc_by_type": { "choice": -8.43373493975904, "noul": -100.0, "score": -25.491241738200188 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 5, "score": 2 }, "score": 0.0 }, "en": { "accuracy": 0.26273458445040215, "cc": -24.234183293883643, "cc_by_type": { "choice": 8.693213684800824, "noul": -62.40601503759399, "score": -18.989748528857753 }, "low_n": false, "n": 1119, "n_by_type": { "choice": 577, "noul": 266, "score": 276 }, "score": 0.0 }, "es": { "accuracy": 0.21428571428571427, "cc": -23.875465734890014, "cc_by_type": { "choice": 8.363636363636363, "noul": -100.0, "score": 20.00996643169359 }, "low_n": true, "n": 28, "n_by_type": { "choice": 10, "noul": 11, "score": 7 }, "score": 0.0 }, "fi": { "accuracy": 0.2222222222222222, "cc": -42.25352112676057, "cc_by_type": { "choice": 15.49295774647887, "noul": -100.0 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 3 }, "score": 0.0 }, "fr": { "accuracy": 0.2, "cc": -32.05973275794293, "cc_by_type": { "choice": -1.5514809590973198, "noul": -100.0, "score": 5.372282685268514 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 4, "score": 6 }, "score": 0.0 }, "hi": { "accuracy": 0.2, "cc": -27.81219448998868, "cc_by_type": { "choice": 13.043478260869566, "noul": -100.0, "score": 3.519938269164402 }, "low_n": false, "n": 30, "n_by_type": { "choice": 11, "noul": 10, "score": 9 }, "score": 0.0 }, "id": { "accuracy": 0.16666666666666666, "cc": -48.387096774193544, "cc_by_type": { "choice": 3.225806451612906, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 0.0 }, "it": { "accuracy": 0.23529411764705882, "cc": -47.53372129710758, "cc_by_type": { "choice": 23.83638928067701, "noul": -100.0, "score": -66.43755317199978 }, "low_n": true, "n": 17, "n_by_type": { "choice": 5, "noul": 4, "score": 8 }, "score": 0.0 }, "ja": { "accuracy": 0.125, "cc": -47.14183058522813, "cc_by_type": { "choice": -13.821138211382111, "noul": -100.0, "score": -27.604353544302263 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "ko": { "accuracy": 0.18181818181818182, "cc": -43.344897587116954, "cc_by_type": { "choice": -31.14754098360656, "noul": -100.0, "score": 1.1128482222557023 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.45454545454545453, "cc": 16.030320645251987, "cc_by_type": { "choice": 56.04395604395604, "noul": -50.0, "score": 42.04700589179993 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 16.030320645251987 }, "nl": { "accuracy": 0.23076923076923078, "cc": -20.35052023493665, "cc_by_type": { "choice": 24.79108635097493, "noul": -66.66666666666667, "score": -19.17598038911821 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 6, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.14285714285714285, "cc": -49.18032786885246, "cc_by_type": { "choice": 1.6393442622950836, "noul": -100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 3 }, "score": 0.0 }, "pl": { "accuracy": 0.225, "cc": -12.618486802534166, "cc_by_type": { "choice": 5.553126366418887, "noul": -60.0, "score": 16.591413225978613 }, "low_n": false, "n": 40, "n_by_type": { "choice": 16, "noul": 10, "score": 14 }, "score": 0.0 }, "pt": { "accuracy": 0.2222222222222222, "cc": -28.382395955071996, "cc_by_type": { "choice": 9.547738693467334, "noul": -66.66666666666667, "score": -28.02825989201665 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 6, "score": 5 }, "score": 0.0 }, "sv": { "accuracy": 0.0, "cc": -48.522135284211906, "cc_by_type": { "choice": -33.333333333333336, "noul": -100.0, "score": -12.233072519302368 }, "low_n": true, "n": 8, "n_by_type": { "choice": 3, "noul": 4, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.36363636363636365, "cc": -18.188721011256614, "cc_by_type": { "choice": -5.26315789473684, "noul": -100.0, "score": 50.696994860967 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 3, "score": 3 }, "score": 0.0 }, "uk": { "accuracy": 0.09090909090909091, "cc": -41.07722171240693, "cc_by_type": { "choice": -30.43478260869565, "noul": -100.0, "score": 7.2031174714748625 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 1, "score": 4 }, "score": 0.0 }, "zh": { "accuracy": 0.375, "cc": -10.272994610640382, "cc_by_type": { "choice": 35.13513513513514, "noul": -100.0, "score": 34.045881032943726 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 0.0 } }, "type": { "choice": { "accuracy": 0.2853333333333333, "cc": 7.410606322335463, "cc_by_type": { "choice": 7.410606322335463 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 7.410606322335463 }, "noul": { "accuracy": 0.14933333333333335, "cc": -70.13333333333334, "cc_by_type": { "noul": -70.13333333333334 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.2853333333333333, "cc": -14.348760038030782, "cc_by_type": { "score": -14.348760038030782 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "usecase": { "advertising": { "accuracy": 0.2916666666666667, "cc": -18.14065897262343, "cc_by_type": { "choice": -8.731466227347617, "noul": -19.999999999999996, "score": -25.690510690522682 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.16666666666666666, "cc": -19.845708892643234, "cc_by_type": { "choice": -8.01963993453355, "noul": -60.0, "score": 8.482513256603841 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 0.0 }, "customer_support": { "accuracy": 0.2, "cc": -36.093708917165806, "cc_by_type": { "choice": 3.225806451612906, "noul": -75.0, "score": -36.506933203110336 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 8, "score": 4 }, "score": 0.0 }, "demand_forecasting": { "accuracy": 0.19230769230769232, "cc": -43.411243099056556, "cc_by_type": { "choice": 15.662650602409636, "noul": -100.0, "score": -45.89637989957931 }, "low_n": true, "n": 26, "n_by_type": { "choice": 11, "noul": 4, "score": 11 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.2222222222222222, "cc": -19.47483978171899, "cc_by_type": { "choice": 6.832298136645961, "noul": -60.0, "score": -5.256817481802933 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 5, "score": 6 }, "score": 0.0 }, "feature_extraction": { "accuracy": 0.10344827586206896, "cc": -52.342876150533066, "cc_by_type": { "choice": -11.111111111111114, "noul": -100.0, "score": -45.917517340488075 }, "low_n": true, "n": 29, "n_by_type": { "choice": 15, "noul": 5, "score": 9 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.38461538461538464, "cc": 0.6176093078177957, "cc_by_type": { "choice": 8.695652173913043, "noul": 0.0, "score": -6.842824250459656 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 0.6176093078177957 }, "gaming": { "accuracy": 0.20689655172413793, "cc": -26.6661863971574, "cc_by_type": { "choice": 16.831683168316832, "noul": -71.42857142857143, "score": -25.40167093121759 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 7, "score": 11 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.2857142857142857, "cc": 4.412051451900017, "cc_by_type": { "choice": 20.634920634920636, "noul": -100.0, "score": 92.60123372077942 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 5, "score": 1 }, "score": 4.412051451900017 }, "knowledge_graphs": { "accuracy": 0.08695652173913043, "cc": -55.71206240218595, "cc_by_type": { "choice": -9.923664122137408, "noul": -100.0, "score": -57.21252308442044 }, "low_n": true, "n": 23, "n_by_type": { "choice": 14, "noul": 5, "score": 4 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.28, "cc": -16.228836402216572, "cc_by_type": { "choice": 18.69918699186992, "noul": -100.0, "score": 32.61430380148036 }, "low_n": true, "n": 25, "n_by_type": { "choice": 8, "noul": 8, "score": 9 }, "score": 0.0 }, "legal_compliance": { "accuracy": 0.23076923076923078, "cc": -29.703887276787327, "cc_by_type": { "choice": 1.6393442622950836, "noul": -100.0, "score": 9.248993907342928 }, "low_n": true, "n": 13, "n_by_type": { "choice": 8, "noul": 1, "score": 4 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.3103448275862069, "cc": -15.934116861404396, "cc_by_type": { "choice": 17.431192660550458, "noul": -50.0, "score": -15.233543244763649 }, "low_n": true, "n": 29, "n_by_type": { "choice": 14, "noul": 8, "score": 7 }, "score": 0.0 }, "model_routing": { "accuracy": 0.4, "cc": 4.006137375452837, "cc_by_type": { "choice": 27.633342267488622, "noul": -40.90909090909091, "score": 25.294160767960793 }, "low_n": false, "n": 150, "n_by_type": { "choice": 74, "noul": 44, "score": 32 }, "score": 4.006137375452837 }, "moderation_trust_safety": { "accuracy": 0.25, "cc": -29.448869234338442, "cc_by_type": { "choice": 5.263157894736842, "noul": -100.0, "score": 6.390234402247841 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 5, "score": 3 }, "score": 0.0 }, "recruiting": { "accuracy": 0.23809523809523808, "cc": -26.169667953406684, "cc_by_type": { "choice": 24.62311557788945, "noul": -100.0, "score": -3.132119438109493 }, "low_n": true, "n": 21, "n_by_type": { "choice": 9, "noul": 6, "score": 6 }, "score": 0.0 }, "risk_assessment": { "accuracy": 0.23076923076923078, "cc": -41.00485732486038, "cc_by_type": { "choice": 5.660377358490565, "noul": -100.0, "score": -28.67494933307171 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 3, "score": 3 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.15625, "cc": -22.981094893304945, "cc_by_type": { "choice": -11.553784860557771, "noul": -71.42857142857143, "score": 14.03907160921436 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.13636363636363635, "cc": -52.2527400518963, "cc_by_type": { "choice": -14.558472553699291, "noul": -100.0, "score": -42.199747601989635 }, "low_n": true, "n": 22, "n_by_type": { "choice": 9, "noul": 6, "score": 7 }, "score": 0.0 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 57.18171398892909, 65.69339495815166 ], "I": [ 1.4982597932705726, 6.488299230052042 ], "capability": [ 29.771515209995528, 35.358321100817776 ], "composite": { "A": [ 0.0050628325646249585, 0.3450163793532752 ], "B": [ 0.003260750472245594, 0.24100666656993436 ], "C": [ 0.003515855947656221, 0.239594707884219 ] }, "seed": 19 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 10.073244581339713, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 62.32062493293682, "capability": 32.79110507364784, "intelligence": 3.2615852143588597 }, "per_type": { "choice": { "accuracy": 0.2853333333333333, "calibration": 54.23981005825647, "cc": 7.410606322335463, "cc_by_type": { "choice": 7.410606322335463 }, "cc_tiered": 9.78475564307658, "cc_tiered_open": 12.87966201889643, "cc_tiered_sealed": 6.689849267256729, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 7.410606322335463 }, "noul": { "accuracy": 0.14933333333333335, "calibration": 69.32868967393878, "cc": -70.13333333333334, "cc_by_type": { "noul": -70.13333333333334 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.2853333333333333, "calibration": 63.39337506661522, "cc": -14.348760038030782, "cc_by_type": { "score": -14.348760038030782 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "raw_on_A": { "calibration": 60.8515976034804, "capability": 31.49910396998157, "intelligence": 2.1466103364827385 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 0.038577217040809395, "Emphasis on Accuracy 60:20:20": 0.022535446033063476, "Emphasis on Cost 20:20:60": 0.05938557623710964, "Emphasis on Speed 20:60:20": 0.06041396685797389, "Intelligence only": 0.013878616661020313 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Shanghua Gao (@GaoShanghua)", "axes": { "calibration": 85.34979344070516, "cost": 31.141652902933714, "intelligence": 61.564156247760415, "speed": 84.71397705257328 }, "calibration": { "parts": { "choice": { "ece_hard": 0.08475596558784464, "ece_score": 83.04880688243107, "mean_tvd": null, "n": 739, "n_ece_hard": 317, "n_tvd": 0, "score": 83.04880688243107 }, "noul": { "abstention_rate": 0.17307692307692313, "brier": 0.10655681075571934, "ece": 0.06711648937487673, "n": 364, "score": 86.57670212502465 }, "score": { "interval90_coverage": 0.9402173913043478, "interval90_width": 0.263586956521739, "n": 368, "nrps": 0.07024138314041055, "score": 86.42387131465975, "top_ece": 0.10064059528319728 } }, "score": 85.34979344070516 }, "capability": 73.45697484423279, "composite_ci95": { "A": [ 21.035782062124166, 21.889081717461654 ], "B": [ 21.212049494503667, 22.466638130307462 ], "C": [ 18.677168153131788, 21.889081717461654 ] }, "cost": { "basis": "ESTIMATE (*est.), base-model market reference. The declared base Qwen/Qwen3.8-27B resolves in the frozen 25 Sep 2026 snapshot through pricing_v15 BASE_REFERENCES to qwen/qwen3.8-27b at USD 0.42 per 1M input and USD 3.00 per 1M output (verified before the pod was rented; r75 COST-BASIS.json). This row reports a real output token count - 1 token per readout, 1,471 in total - so the Gutsy zero-output rule does NOT apply and the output price is charged as measured. This row's own usage on draw v1.6-regular-20261010-a2: 694,381 input and 1,471 output tokens over 1,500 decisions.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.19736867999999996 }, "display": "RSI-Jev v6.1-VL 27B", "endpoint_condition": "evaluator-owned Lium H100 80GB pod (36 min, USD 1.95/h), sealed per chief-of-staff decision #16415 on Florian's YES: the author's own serve.cli inside a loopback-only network namespace (unshare -n; DNS and outbound TCP both proven dead from inside the run namespace), started with env -i and OPENAI_API_KEY empty, plus a pod-level iptables egress cut installed and verified before the sealed input was uploaded. HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE set only to block the download fallbacks the r75 review found; no source line was edited. Server pointed at the LOCAL PINNED SNAPSHOT DIRECTORY, not a repo id, so no revision could resolve to main. Unchanged defaults, --effort auto, 32,768-token cap, no truncation. 54,047 MiB of 81,559 MiB used.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 68.48613704048307, "I_sealed": 54.64217545503776, "base": 61.564156247760415, "excess": 3.7707170041056006, "gap": 13.843961585445314, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 75.8256733782429, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 92.21621621621622, "hard": 73.57659641396666, "judge": 72.31377719182598, "standard": 77.39640016743407 } }, "open|noul": { "cc": 66.15458937198068, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 82.6086956521739, "judge": 39.99999999999999, "standard": 55.55555555555556 } }, "open|score": { "cc": 63.478148371225664, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 67.937072018553, "hard": 66.44443112298649, "judge": 61.06082661163181, "standard": 58.942103683431114 } }, "sealed|choice": { "cc": 70.33653729047333, "n": { "easy": 72, "hard": 260, "judge": 141, "standard": 127 }, "tiers": { "easy": 88.73709458754823, "hard": 64.87853138397273, "judge": 71.69093399624576, "standard": 70.02067539627843 } }, "sealed|noul": { "cc": 41.66298613667034, "n": { "easy": 39, "hard": 132, "judge": 72, "standard": 57 }, "tiers": { "easy": 58.974358974358964, "hard": 21.212121212121215, "judge": 50.0, "standard": 61.403508771929815 } }, "sealed|score": { "cc": 51.92700293796962, "n": { "easy": 33, "hard": 115, "judge": 75, "standard": 77 }, "tiers": { "easy": 68.04656098024147, "hard": 33.730600016105896, "judge": 64.21257035928524, "standard": 61.83167862858772 } } } }, "jevbench_score": 21.58850717988564, "key": "rsi-jev-v6.1-vl-27b", "last_measured_on": "2026-10-10", "licence": "Apache-2.0 (model card); source Shanghua-Gao/RSI-Jev@fc982bbd MIT", "listing": "ranked", "measured_in": "v1.6.5", "measurement_date_status": "measured", "model_pin": "shgao/rsi-jev-v6.1-vl-27b@d0dccc052ee844ab7481ddfacad1b7335ab394e0", "noul_decisive": { "acc_among_decisive": 0.9102990033222591, "decisive": 301, "decisive_rate": 0.8269230769230769, "n": 375, "supported": true, "valid": 364 }, "planned_weight_denominator": 1.0, "rank": 2, "ranked": true, "ranks": { "A": 2, "B": 2, "C": 2, "capability": 1 }, "repo": "https://huggingface.co/shgao/rsi-jev-v6.1-vl-27b", "scores": { "A": 21.58850717988564, "B": 22.011280090800952, "C": 21.58850717988564 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.39549588703666816, "p50_s_raw": 0.12274794351833407, "p95_s_adjusted": 0.8540048229158855, "p95_s_raw": 0.35200241145794275 }, "status": { "answered_ok": 1471, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.9310344827586207, "cc": 86.4125424016998, "cc_by_type": { "choice": 100.0, "noul": 66.66666666666667, "score": 92.5709605384327 }, "low_n": true, "n": 29, "n_by_type": { "choice": 16, "noul": 6, "score": 7 }, "score": 86.4125424016998 }, "calibration_gen": { "accuracy": 0.7232142857142857, "cc": 58.19506668349169, "cc_by_type": { "choice": 67.3913043478261, "noul": 61.53846153846154, "score": 45.655434164187426 }, "low_n": false, "n": 112, "n_by_type": { "choice": 56, "noul": 26, "score": 30 }, "score": 58.19506668349169 }, "composite_gen": { "accuracy": 0.8877551020408163, "cc": 82.46075900003056, "cc_by_type": { "choice": 96.29629629629629, "noul": 78.57142857142858, "score": 72.51455213236682 }, "low_n": false, "n": 98, "n_by_type": { "choice": 36, "noul": 28, "score": 34 }, "score": 82.46075900003056 }, "fairness": { "accuracy": 0.9629629629629629, "cc": 94.1963028792578, "cc_by_type": { "choice": 88.23529411764707, "noul": 100.0, "score": 94.35361452012633 }, "low_n": true, "n": 27, "n_by_type": { "choice": 12, "noul": 6, "score": 9 }, "score": 94.1963028792578 }, "jaggedness": { "accuracy": 0.5081081081081081, "cc": 13.570657148702269, "cc_by_type": { "choice": 38.86058360352014, "noul": -48.71794871794872, "score": 50.56933656053539 }, "low_n": false, "n": 185, "n_by_type": { "choice": 97, "noul": 39, "score": 49 }, "score": 13.570657148702269 }, "llm_router": { "accuracy": 0.8598484848484849, "cc": 81.63698812259595, "cc_by_type": { "choice": 75.84765443567116, "noul": 86.30136986301369, "score": 82.76194006910303 }, "low_n": false, "n": 264, "n_by_type": { "choice": 131, "noul": 73, "score": 60 }, "score": 81.63698812259595 }, "long_state": { "accuracy": 0.0, "cc": -110.31746031746032, "cc_by_type": { "choice": -30.952380952380945, "noul": -100.0, "score": -200.0 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 11, "score": 7 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.6774193548387096, "cc": 31.38414817065086, "cc_by_type": { "choice": 72.30769230769232, "noul": 0.0, "score": 21.844752204260253 }, "low_n": false, "n": 31, "n_by_type": { "choice": 15, "noul": 8, "score": 8 }, "score": 31.38414817065086 }, "multilingual": { "accuracy": 0.7428571428571429, "cc": 58.6257780124704, "cc_by_type": { "choice": 70.44334975369458, "noul": 51.72413793103448, "score": 53.70984635268214 }, "low_n": false, "n": 315, "n_by_type": { "choice": 146, "noul": 87, "score": 82 }, "score": 58.6257780124704 }, "negation_pair_gen": { "accuracy": 0.875, "cc": 63.20741226657981, "cc_by_type": { "choice": 90.47619047619048, "noul": 60.00000000000001, "score": 39.146046323548944 }, "low_n": false, "n": 56, "n_by_type": { "choice": 28, "noul": 20, "score": 8 }, "score": 63.20741226657981 }, "option_order": { "accuracy": 0.9824561403508771, "cc": 97.78024417314094, "cc_by_type": { "choice": 97.78024417314094 }, "low_n": false, "n": 57, "n_by_type": { "choice": 57 }, "score": 97.78024417314094 }, "perturb_pair": { "accuracy": 0.9, "cc": 89.49429032954704, "cc_by_type": { "choice": 84.27260812581913, "noul": 100.0, "score": 84.21026286282202 }, "low_n": false, "n": 60, "n_by_type": { "choice": 34, "noul": 14, "score": 12 }, "score": 89.49429032954704 }, "usecase": { "accuracy": 0.7046413502109705, "cc": 51.83295324947296, "cc_by_type": { "choice": 73.40528040084796, "noul": 36.8421052631579, "score": 45.251474084413026 }, "low_n": false, "n": 237, "n_by_type": { "choice": 111, "noul": 57, "score": 69 }, "score": 51.83295324947296 } }, "lang": { "ar": { "accuracy": 0.8181818181818182, "cc": 69.11348259490397, "cc_by_type": { "choice": 73.33333333333334, "noul": 50.0, "score": 84.00711445137858 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 4, "score": 2 }, "score": 69.11348259490397 }, "cs": { "accuracy": 0.875, "cc": 87.28813559322035, "cc_by_type": { "choice": 74.57627118644068, "noul": 100.0 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 3 }, "score": 87.28813559322035 }, "da": { "accuracy": 0.6666666666666666, "cc": 44.00477315138941, "cc_by_type": { "choice": 52.662721893491124, "noul": 19.999999999999996, "score": 59.351597560677114 }, "low_n": true, "n": 24, "n_by_type": { "choice": 11, "noul": 5, "score": 8 }, "score": 44.00477315138941 }, "de": { "accuracy": 0.8444444444444444, "cc": 72.3799653014141, "cc_by_type": { "choice": 83.70549185274592, "noul": 77.77777777777777, "score": 55.656626273718615 }, "low_n": false, "n": 45, "n_by_type": { "choice": 24, "noul": 9, "score": 12 }, "score": 72.3799653014141 }, "el": { "accuracy": 0.6666666666666666, "cc": -0.8800661948371129, "cc_by_type": { "choice": 75.90361445783132, "noul": -19.999999999999996, "score": -58.54381304234266 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 5, "score": 2 }, "score": 0.0 }, "en": { "accuracy": 0.7497765862377123, "cc": 57.06152962458927, "cc_by_type": { "choice": 72.18171620863711, "noul": 42.85714285714286, "score": 56.14572980798784 }, "low_n": false, "n": 1119, "n_by_type": { "choice": 577, "noul": 266, "score": 276 }, "score": 57.06152962458927 }, "es": { "accuracy": 0.8928571428571429, "cc": 82.61244146186118, "cc_by_type": { "choice": 86.90909090909093, "noul": 100.0, "score": 60.9282334764926 }, "low_n": true, "n": 28, "n_by_type": { "choice": 10, "noul": 11, "score": 7 }, "score": 82.61244146186118 }, "fi": { "accuracy": 0.5555555555555556, "cc": 34.97652582159624, "cc_by_type": { "choice": 36.61971830985915, "noul": 33.33333333333333 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 3 }, "score": 34.97652582159624 }, "fr": { "accuracy": 0.6666666666666666, "cc": 41.96733456321797, "cc_by_type": { "choice": 74.61212976022568, "noul": 0.0, "score": 51.28987392942821 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 4, "score": 6 }, "score": 41.96733456321797 }, "hi": { "accuracy": 0.6666666666666666, "cc": 52.85728930896761, "cc_by_type": { "choice": 62.73291925465839, "noul": 60.00000000000001, "score": 35.83894867224444 }, "low_n": false, "n": 30, "n_by_type": { "choice": 11, "noul": 10, "score": 9 }, "score": 52.85728930896761 }, "id": { "accuracy": 0.8333333333333334, "cc": 83.87096774193549, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 83.87096774193549 }, "it": { "accuracy": 1.0, "cc": 90.29663262626855, "cc_by_type": { "choice": 100.00000000000001, "noul": 100.0, "score": 70.88989787880564 }, "low_n": true, "n": 17, "n_by_type": { "choice": 5, "noul": 4, "score": 8 }, "score": 90.29663262626855 }, "ja": { "accuracy": 0.53125, "cc": 32.05622061795162, "cc_by_type": { "choice": 43.08943089430895, "noul": 14.28571428571428, "score": 38.79351667383163 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 32.05622061795162 }, "ko": { "accuracy": 0.8181818181818182, "cc": 85.45781546660389, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 56.373446399811655 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 2, "score": 5 }, "score": 85.45781546660389 }, "mixed": { "accuracy": 1.0, "cc": 99.9689563002903, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 99.9068689008709 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 99.9689563002903 }, "nl": { "accuracy": 0.6923076923076923, "cc": 40.991198714143415, "cc_by_type": { "choice": 74.93036211699165, "noul": 33.33333333333333, "score": 14.709900692105283 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 6, "score": 2 }, "score": 40.991198714143415 }, "no": { "accuracy": 0.7142857142857143, "cc": 67.21311475409837, "cc_by_type": { "choice": 34.42622950819673, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 3 }, "score": 67.21311475409837 }, "pl": { "accuracy": 0.7, "cc": 51.32398893140217, "cc_by_type": { "choice": 92.12942719720158, "noul": 19.999999999999996, "score": 41.84253959700492 }, "low_n": false, "n": 40, "n_by_type": { "choice": 16, "noul": 10, "score": 14 }, "score": 51.32398893140217 }, "pt": { "accuracy": 0.8333333333333334, "cc": 83.10065347581066, "cc_by_type": { "choice": 63.81909547738693, "noul": 100.0, "score": 85.48286495004504 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 6, "score": 5 }, "score": 83.10065347581066 }, "sv": { "accuracy": 0.625, "cc": 53.29148096066934, "cc_by_type": { "choice": 11.111111111111109, "noul": 50.0, "score": 98.76333177089691 }, "low_n": true, "n": 8, "n_by_type": { "choice": 3, "noul": 4, "score": 1 }, "score": 53.29148096066934 }, "tr": { "accuracy": 0.7272727272727273, "cc": 23.85319314395998, "cc_by_type": { "choice": 73.6842105263158, "noul": 33.33333333333333, "score": -35.457964427769184 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 3, "score": 3 }, "score": 23.85319314395998 }, "uk": { "accuracy": 0.5454545454545454, "cc": 4.1198605058986955, "cc_by_type": { "choice": 34.78260869565218, "noul": -100.0, "score": 77.5769728220439 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 1, "score": 4 }, "score": 4.1198605058986955 }, "zh": { "accuracy": 0.875, "cc": 73.56783818557031, "cc_by_type": { "choice": 100.00000000000001, "noul": 33.33333333333333, "score": 87.37018122337759 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 73.56783818557031 } }, "type": { "choice": { "accuracy": 0.7826666666666666, "cc": 71.84315080324754, "cc_by_type": { "choice": 71.84315080324754 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 71.84315080324754 }, "noul": { "accuracy": 0.7306666666666667, "cc": 46.13333333333334, "cc_by_type": { "noul": 46.13333333333334 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 46.13333333333334 }, "score": { "accuracy": 0.6986666666666667, "cc": 54.531190100580204, "cc_by_type": { "score": 54.531190100580204 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.531190100580204 } }, "usecase": { "advertising": { "accuracy": 0.7083333333333334, "cc": 36.966773340637815, "cc_by_type": { "choice": 70.34596375617794, "noul": -19.999999999999996, "score": 60.55435626573551 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 36.966773340637815 }, "code_linting": { "accuracy": 0.75, "cc": 68.95860502475776, "cc_by_type": { "choice": 70.54009819967267, "noul": 100.0, "score": 36.33571687460061 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 68.95860502475776 }, "customer_support": { "accuracy": 0.75, "cc": 47.499513085946944, "cc_by_type": { "choice": 67.74193548387096, "noul": 75.0, "score": -0.24339622603013478 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 8, "score": 4 }, "score": 47.499513085946944 }, "demand_forecasting": { "accuracy": 0.3076923076923077, "cc": -17.782585557336656, "cc_by_type": { "choice": 15.662650602409636, "noul": -50.0, "score": -19.010407274419606 }, "low_n": true, "n": 26, "n_by_type": { "choice": 11, "noul": 4, "score": 11 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5555555555555556, "cc": 28.57990326207394, "cc_by_type": { "choice": 62.73291925465839, "noul": -19.999999999999996, "score": 43.006790531563425 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 5, "score": 6 }, "score": 28.57990326207394 }, "feature_extraction": { "accuracy": 0.8620689655172413, "cc": 64.36666498139438, "cc_by_type": { "choice": 82.90598290598291, "noul": 19.999999999999996, "score": 90.19401203820024 }, "low_n": true, "n": 29, "n_by_type": { "choice": 15, "noul": 5, "score": 9 }, "score": 64.36666498139438 }, "financial_crime": { "accuracy": 0.7692307692307693, "cc": 69.24951206795548, "cc_by_type": { "choice": 86.95652173913045, "noul": 100.0, "score": 20.792014464735985 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 69.24951206795548 }, "gaming": { "accuracy": 0.5862068965517241, "cc": 25.37172392511331, "cc_by_type": { "choice": 52.475247524752476, "noul": -14.28571428571429, "score": 37.92563853630174 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 7, "score": 11 }, "score": 25.37172392511331 }, "insurance_claims": { "accuracy": 0.5714285714285714, "cc": 31.669815093397148, "cc_by_type": { "choice": 52.38095238095238, "noul": -19.999999999999996, "score": 62.62849289923906 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 5, "score": 1 }, "score": 31.669815093397148 }, "knowledge_graphs": { "accuracy": 0.6956521739130435, "cc": 42.704203038079555, "cc_by_type": { "choice": 72.51908396946564, "noul": 19.999999999999996, "score": 35.59352514477303 }, "low_n": true, "n": 23, "n_by_type": { "choice": 14, "noul": 5, "score": 4 }, "score": 42.704203038079555 }, "lead_generation": { "accuracy": 0.72, "cc": 55.47104563667543, "cc_by_type": { "choice": 100.0, "noul": 25.0, "score": 41.413136910026275 }, "low_n": true, "n": 25, "n_by_type": { "choice": 8, "noul": 8, "score": 9 }, "score": 55.47104563667543 }, "legal_compliance": { "accuracy": 0.6923076923076923, "cc": 8.840061259572431, "cc_by_type": { "choice": 67.21311475409836, "noul": -100.0, "score": 59.30706902461893 }, "low_n": true, "n": 13, "n_by_type": { "choice": 8, "noul": 1, "score": 4 }, "score": 8.840061259572431 }, "llm_guardrails": { "accuracy": 0.9310344827586207, "cc": 89.90702297249, "cc_by_type": { "choice": 90.8256880733945, "noul": 100.0, "score": 78.89538084407552 }, "low_n": true, "n": 29, "n_by_type": { "choice": 14, "noul": 8, "score": 7 }, "score": 89.90702297249 }, "model_routing": { "accuracy": 0.86, "cc": 76.7397155888396, "cc_by_type": { "choice": 85.52666845349772, "noul": 81.81818181818181, "score": 62.874296494839264 }, "low_n": false, "n": 150, "n_by_type": { "choice": 74, "noul": 44, "score": 32 }, "score": 76.7397155888396 }, "moderation_trust_safety": { "accuracy": 0.6875, "cc": 52.07409272558595, "cc_by_type": { "choice": 52.63157894736842, "noul": 60.00000000000001, "score": 43.590699229389415 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 5, "score": 3 }, "score": 52.07409272558595 }, "recruiting": { "accuracy": 0.6666666666666666, "cc": 54.013036775459085, "cc_by_type": { "choice": 54.77386934673366, "noul": 66.66666666666667, "score": 40.59857431297692 }, "low_n": true, "n": 21, "n_by_type": { "choice": 9, "noul": 6, "score": 6 }, "score": 54.013036775459085 }, "risk_assessment": { "accuracy": 0.6153846153846154, "cc": 46.84943347419826, "cc_by_type": { "choice": 62.264150943396224, "noul": 33.33333333333333, "score": 44.95081614586523 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 3, "score": 3 }, "score": 46.84943347419826 }, "scientific_discovery": { "accuracy": 0.59375, "cc": 25.763079128747716, "cc_by_type": { "choice": 60.1593625498008, "noul": -14.28571428571429, "score": 31.41558912215664 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 25.763079128747716 }, "search_retrieval": { "accuracy": 0.9090909090909091, "cc": 86.99121578361938, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 60.97364735085814 }, "low_n": true, "n": 22, "n_by_type": { "choice": 9, "noul": 6, "score": 7 }, "score": 86.99121578361938 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 80.61043166908699, 87.42600376043296 ], "I": [ 56.5653697398953, 65.2038168618822 ], "capability": [ 69.2223091749473, 75.71790383555764 ], "composite": { "A": [ 21.035782062124166, 21.889081717461654 ], "B": [ 21.212049494503667, 22.466638130307462 ], "C": [ 18.677168153131788, 21.889081717461654 ] }, "seed": 20 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 10.073244581339713, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 85.34979344070516, "capability": 73.45697484423279, "intelligence": 61.564156247760415 }, "per_type": { "choice": { "accuracy": 0.7826666666666666, "calibration": 83.04880688243107, "cc": 71.84315080324754, "cc_by_type": { "choice": 71.84315080324754 }, "cc_tiered": 73.08110533435811, "cc_tiered_open": 75.8256733782429, "cc_tiered_sealed": 70.33653729047333, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 71.84315080324754 }, "noul": { "accuracy": 0.7306666666666667, "calibration": 86.57670212502465, "cc": 46.13333333333334, "cc_by_type": { "noul": 46.13333333333334 }, "cc_tiered": 53.90878775432551, "cc_tiered_open": 66.15458937198068, "cc_tiered_sealed": 41.66298613667034, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 46.13333333333334 }, "score": { "accuracy": 0.6986666666666667, "calibration": 86.42387131465975, "cc": 54.531190100580204, "cc_by_type": { "score": 54.531190100580204 }, "cc_tiered": 57.70257565459764, "cc_tiered_open": 63.478148371225664, "cc_tiered_sealed": 51.92700293796962, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.531190100580204 } }, "raw_on_A": { "calibration": 84.95320234939237, "capability": 73.13630404134082, "intelligence": 61.31940573328927 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.014666666666666717, "n": 750 }, "noul": { "invalid_rate": 0.029333333333333322, "n": 375 }, "score": { "invalid_rate": 0.01866666666666672, "n": 375 } }, "views": { "Balanced 33:33:33": 19.344794057306224, "Emphasis on Accuracy 60:20:20": 20.935791627644683, "Emphasis on Cost 20:20:60": 15.593981583505036, "Emphasis on Speed 20:60:20": 23.15453324264071, "Intelligence only": 23.882030176915077 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Standard Thinking", "axes": { "calibration": 85.49831885164681, "cost": 43.21124696592469, "intelligence": 34.464905061466695, "speed": 84.28099014807135 }, "calibration": { "parts": { "choice": { "ece_hard": 0.07091377459977057, "ece_score": 85.8172450800459, "mean_tvd": null, "n": 739, "n_ece_hard": 317, "n_tvd": 0, "score": 85.8172450800459 }, "noul": { "abstention_rate": 0.39010989010989006, "brier": 0.15382831780922446, "ece": 0.0674352050569083, "n": 364, "score": 86.51295898861834 }, "score": { "interval90_coverage": 0.9320652173913043, "interval90_width": 0.5106431159420289, "n": 368, "nrps": 0.11464419484173276, "score": 84.1647524862762, "top_ece": 0.10103037771637155 } }, "score": 85.49831885164681 }, "capability": 59.981611956556755, "composite_ci95": { "A": [ 12.260077031927597, 23.834682052129676 ], "B": [ 10.755937812612656, 21.913672526185227 ], "C": [ 8.513942383283057, 16.551862536201167 ] }, "cost": { "basis": "v1.5-M2 base-model reference: model card base_model StandardThinking/StandardOne-8B, built on mistralai/Ministral-3-8B-Instruct-2512 (not listed on OpenRouter as Standard One). OpenRouter mistralai/ministral-8b-2512 list price USD 0.15/M input and USD 0.15/M output, read 2026-10-10T08:43Z from https://openrouter.ai/api/v1/models. Schema head generates no text: 0 output tokens.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.0781553 }, "display": "Standard One 8B SH (Standard Thinking)", "endpoint_condition": "evaluator-owned Lium GPU pod (H100 80GB), offline read-only container (--network none); author's reference server code/serve_head.py, unchanged defaults", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 42.65983609411336, "I_sealed": 26.269974028820027, "base": 34.464905061466695, "excess": 6.316617483953619, "gap": 16.389862065293332, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 64.76188322786868, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 88.32432432432432, "hard": 56.59012268008808, "judge": 64.40342781806197, "standard": 69.8618668899121 } }, "open|noul": { "cc": 26.588681849551417, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 57.14285714285714, "hard": 39.13043478260869, "judge": 10.000000000000009, "standard": 11.111111111111116 } }, "open|score": { "cc": 36.628943204920006, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 20.91980977072071, "hard": 40.258980287858556, "judge": 46.927298157653716, "standard": 21.775903327041945 } }, "sealed|choice": { "cc": 53.894778364676796, "n": { "easy": 72, "hard": 260, "judge": 141, "standard": 127 }, "tiers": { "easy": 60.57983105641882, "hard": 50.5331427943278, "judge": 51.60062909035564, "standard": 60.716747070985534 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 39, "hard": 132, "judge": 72, "standard": 57 }, "tiers": { "easy": 12.82051282051282, "hard": -9.090909090909093, "judge": 11.111111111111116, "standard": -8.771929824561408 } }, "sealed|score": { "cc": 24.915143721783284, "n": { "easy": 33, "hard": 115, "judge": 75, "standard": 77 }, "tiers": { "easy": 48.44464458742374, "hard": 10.686057111566038, "judge": 28.029772063789803, "standard": 36.93662399638777 } } } }, "jevbench_score": 18.74677558938401, "key": "standard_one_8b_sh", "last_measured_on": "2026-10-10", "licence": "Apache-2.0 (merged weights, schema head and reference server; base Ministral 3 8B Apache-2.0)", "listing": "ranked", "measured_in": "v1.6.5", "measurement_date_status": "measured", "model_pin": "StandardThinking/StandardOne-8B-SH@25e5739bd9635b54f925cf9ba2daa2ec42afd19e", "noul_decisive": { "acc_among_decisive": 0.8828828828828829, "decisive": 222, "decisive_rate": 0.6098901098901099, "n": 375, "supported": true, "valid": 364 }, "planned_weight_denominator": 1.0, "rank": 3, "ranked": true, "ranks": { "A": 3, "B": 3, "C": 3, "capability": 3 }, "repo": "https://huggingface.co/StandardThinking/StandardOne-8B-SH", "scores": { "A": 18.74677558938401, "B": 16.94153091767041, "C": 13.018594159294452 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.27213579495437445, "p50_s_raw": 0.061067897477187216, "p95_s_adjusted": 1.3712458158901424, "p95_s_raw": 0.6106229079450712 }, "status": { "answered_ok": 1471, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.896551724137931, "cc": 76.24972247673712, "cc_by_type": { "choice": 83.33333333333333, "noul": 100.0, "score": 45.41583409687805 }, "low_n": true, "n": 29, "n_by_type": { "choice": 16, "noul": 6, "score": 7 }, "score": 76.24972247673712 }, "calibration_gen": { "accuracy": 0.6339285714285714, "cc": 40.6073873879211, "cc_by_type": { "choice": 60.40372670807453, "noul": 30.76923076923077, "score": 30.64920468645799 }, "low_n": false, "n": 112, "n_by_type": { "choice": 56, "noul": 26, "score": 30 }, "score": 40.6073873879211 }, "composite_gen": { "accuracy": 0.6428571428571429, "cc": 42.991452161771214, "cc_by_type": { "choice": 70.37037037037037, "noul": 28.57142857142858, "score": 30.032557543514706 }, "low_n": false, "n": 98, "n_by_type": { "choice": 36, "noul": 28, "score": 34 }, "score": 42.991452161771214 }, "fairness": { "accuracy": 0.9629629629629629, "cc": 88.7254774240554, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 66.17643227216622 }, "low_n": true, "n": 27, "n_by_type": { "choice": 12, "noul": 6, "score": 9 }, "score": 88.7254774240554 }, "jaggedness": { "accuracy": 0.4702702702702703, "cc": 8.921712067552678, "cc_by_type": { "choice": 34.69198703103288, "noul": -53.84615384615385, "score": 45.919303017779 }, "low_n": false, "n": 185, "n_by_type": { "choice": 97, "noul": 39, "score": 49 }, "score": 8.921712067552678 }, "llm_router": { "accuracy": 0.6477272727272727, "cc": 47.79185607418858, "cc_by_type": { "choice": 57.268927078495125, "noul": 26.027397260273965, "score": 60.07924388379662 }, "low_n": false, "n": 264, "n_by_type": { "choice": 131, "noul": 73, "score": 60 }, "score": 47.79185607418858 }, "long_state": { "accuracy": 0.0, "cc": -110.31746031746032, "cc_by_type": { "choice": -30.952380952380945, "noul": -100.0, "score": -200.0 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 11, "score": 7 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.3225806451612903, "cc": -19.288646333408973, "cc_by_type": { "choice": 26.153846153846153, "noul": -75.0, "score": -9.019785154073068 }, "low_n": false, "n": 31, "n_by_type": { "choice": 15, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.5873015873015873, "cc": 28.891423678941237, "cc_by_type": { "choice": 56.112852664576806, "noul": 8.045977011494255, "score": 22.515441360752643 }, "low_n": false, "n": 315, "n_by_type": { "choice": 146, "noul": 87, "score": 82 }, "score": 28.891423678941237 }, "negation_pair_gen": { "accuracy": 0.5357142857142857, "cc": -27.351155376155678, "cc_by_type": { "choice": 76.19047619047619, "noul": -50.0, "score": -108.24394231894323 }, "low_n": false, "n": 56, "n_by_type": { "choice": 28, "noul": 20, "score": 8 }, "score": 0.0 }, "option_order": { "accuracy": 0.9122807017543859, "cc": 88.90122086570477, "cc_by_type": { "choice": 88.90122086570477 }, "low_n": false, "n": 57, "n_by_type": { "choice": 57 }, "score": 88.90122086570477 }, "perturb_pair": { "accuracy": 0.8333333333333334, "cc": 73.07758294752922, "cc_by_type": { "choice": 76.4089121887287, "noul": 71.42857142857142, "score": 71.39526522528755 }, "low_n": false, "n": 60, "n_by_type": { "choice": 34, "noul": 14, "score": 12 }, "score": 73.07758294752922 }, "usecase": { "accuracy": 0.4978902953586498, "cc": 20.903453186109534, "cc_by_type": { "choice": 45.65426864521104, "noul": 1.7543859649122862, "score": 15.301704948205275 }, "low_n": false, "n": 237, "n_by_type": { "choice": 111, "noul": 57, "score": 69 }, "score": 20.903453186109534 } }, "lang": { "ar": { "accuracy": 0.2727272727272727, "cc": -41.39521348113611, "cc_by_type": { "choice": -6.666666666666665, "noul": 0.0, "score": -117.51897377674166 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 4, "score": 2 }, "score": 0.0 }, "cs": { "accuracy": 0.375, "cc": -4.802259887005647, "cc_by_type": { "choice": 23.72881355932204, "noul": -33.333333333333336 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 3 }, "score": 0.0 }, "da": { "accuracy": 0.5833333333333334, "cc": 23.301107918011525, "cc_by_type": { "choice": 28.994082840236683, "noul": -19.999999999999996, "score": 60.90924091379788 }, "low_n": true, "n": 24, "n_by_type": { "choice": 11, "noul": 5, "score": 8 }, "score": 23.301107918011525 }, "de": { "accuracy": 0.6444444444444445, "cc": 44.56362261696139, "cc_by_type": { "choice": 72.84248642124321, "noul": 55.55555555555556, "score": 5.292825874085382 }, "low_n": false, "n": 45, "n_by_type": { "choice": 24, "noul": 9, "score": 12 }, "score": 44.56362261696139 }, "el": { "accuracy": 0.5, "cc": -22.37065677373447, "cc_by_type": { "choice": 63.855421686746986, "noul": -60.0, "score": -70.96739200795041 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 5, "score": 2 }, "score": 0.0 }, "en": { "accuracy": 0.5978552278820375, "cc": 30.293789733525, "cc_by_type": { "choice": 56.70218732473357, "noul": 3.007518796992481, "score": 31.171663078848944 }, "low_n": false, "n": 1119, "n_by_type": { "choice": 577, "noul": 266, "score": 276 }, "score": 30.293789733525 }, "es": { "accuracy": 0.7142857142857143, "cc": 43.83935759486882, "cc_by_type": { "choice": 100.0, "noul": 45.45454545454546, "score": -13.936472669939004 }, "low_n": true, "n": 28, "n_by_type": { "choice": 10, "noul": 11, "score": 7 }, "score": 43.83935759486882 }, "fi": { "accuracy": 0.2222222222222222, "cc": -42.25352112676057, "cc_by_type": { "choice": 15.49295774647887, "noul": -100.0 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 3 }, "score": 0.0 }, "fr": { "accuracy": 0.4666666666666667, "cc": -0.26931923616449654, "cc_by_type": { "choice": 49.22425952045134, "noul": -100.0, "score": 49.96778277105517 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 4, "score": 6 }, "score": 0.0 }, "hi": { "accuracy": 0.4, "cc": 3.9605189196118, "cc_by_type": { "choice": 25.465838509316765, "noul": -19.999999999999996, "score": 6.415718249518632 }, "low_n": false, "n": 30, "n_by_type": { "choice": 11, "noul": 10, "score": 9 }, "score": 3.9605189196118 }, "id": { "accuracy": 0.5, "cc": -16.12903225806452, "cc_by_type": { "choice": 67.74193548387096, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 0.0 }, "it": { "accuracy": 0.7647058823529411, "cc": 42.698737038013526, "cc_by_type": { "choice": 100.00000000000001, "noul": 0.0, "score": 28.09621111404058 }, "low_n": true, "n": 17, "n_by_type": { "choice": 5, "noul": 4, "score": 8 }, "score": 42.698737038013526 }, "ja": { "accuracy": 0.5, "cc": 9.04085560182647, "cc_by_type": { "choice": 34.959349593495936, "noul": -42.85714285714286, "score": 35.02036006912633 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 9.04085560182647 }, "ko": { "accuracy": 0.6363636363636364, "cc": 51.84513737928095, "cc_by_type": { "choice": 67.21311475409836, "noul": 100.0, "score": -11.677702616255514 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 2, "score": 5 }, "score": 51.84513737928095 }, "mixed": { "accuracy": 1.0, "cc": 95.1901995158086, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 85.57059854742577 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 95.1901995158086 }, "nl": { "accuracy": 0.46153846153846156, "cc": -6.51647728738407, "cc_by_type": { "choice": 49.860724233983284, "noul": 0.0, "score": -69.4101560961355 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 6, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.42857142857142855, "cc": 17.486338797814206, "cc_by_type": { "choice": 1.6393442622950836, "noul": 33.33333333333333 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 3 }, "score": 17.486338797814206 }, "pl": { "accuracy": 0.575, "cc": 41.28299201238961, "cc_by_type": { "choice": 76.38828159160472, "noul": 19.999999999999996, "score": 27.460694445564116 }, "low_n": false, "n": 40, "n_by_type": { "choice": 16, "noul": 10, "score": 14 }, "score": 41.28299201238961 }, "pt": { "accuracy": 0.8333333333333334, "cc": 76.5780001477811, "cc_by_type": { "choice": 81.90954773869345, "noul": 100.0, "score": 47.82445270464984 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 6, "score": 5 }, "score": 76.5780001477811 }, "sv": { "accuracy": 0.5, "cc": 6.320693289140117, "cc_by_type": { "choice": 11.111111111111109, "noul": 50.0, "score": -42.14903124369076 }, "low_n": true, "n": 8, "n_by_type": { "choice": 3, "noul": 4, "score": 1 }, "score": 6.320693289140117 }, "tr": { "accuracy": 0.6363636363636364, "cc": 22.124025198251957, "cc_by_type": { "choice": 73.6842105263158, "noul": 33.33333333333333, "score": -40.64546826489326 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 3, "score": 3 }, "score": 22.124025198251957 }, "uk": { "accuracy": 0.5454545454545454, "cc": -5.12371977540209, "cc_by_type": { "choice": 34.78260869565218, "noul": -100.0, "score": 49.84623197814155 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 1, "score": 4 }, "score": 0.0 }, "zh": { "accuracy": 0.875, "cc": 62.994622040580985, "cc_by_type": { "choice": 100.00000000000001, "noul": 33.33333333333333, "score": 55.65053278840959 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 62.994622040580985 } }, "type": { "choice": { "accuracy": 0.6653333333333333, "cc": 56.64190706512352, "cc_by_type": { "choice": 56.64190706512352 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 56.64190706512352 }, "noul": { "accuracy": 0.5226666666666666, "cc": 4.533333333333323, "cc_by_type": { "noul": 4.533333333333323 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 4.533333333333323 }, "score": { "accuracy": 0.5173333333333333, "cc": 27.98616472792205, "cc_by_type": { "score": 27.98616472792205 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 27.98616472792205 } }, "usecase": { "advertising": { "accuracy": 0.5833333333333334, "cc": 2.1721868800764432, "cc_by_type": { "choice": 50.57660626029654, "noul": -60.0, "score": 15.939954379932786 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 2.1721868800764432 }, "code_linting": { "accuracy": 0.2916666666666667, "cc": 4.537943129820239, "cc_by_type": { "choice": 1.8003273322422266, "noul": 60.00000000000001, "score": -48.186497942781514 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 4.537943129820239 }, "customer_support": { "accuracy": 0.5, "cc": 18.203665219669762, "cc_by_type": { "choice": 35.483870967741936, "noul": -25.0, "score": 44.12712469126735 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 8, "score": 4 }, "score": 18.203665219669762 }, "demand_forecasting": { "accuracy": 0.3076923076923077, "cc": -15.30308720258734, "cc_by_type": { "choice": 27.710843373493972, "noul": -50.0, "score": -23.620104981255995 }, "low_n": true, "n": 26, "n_by_type": { "choice": 11, "noul": 4, "score": 11 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5555555555555556, "cc": 31.8743625129997, "cc_by_type": { "choice": 44.099378881987576, "noul": 19.999999999999996, "score": 31.523708657011518 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 5, "score": 6 }, "score": 31.8743625129997 }, "feature_extraction": { "accuracy": 0.5172413793103449, "cc": 16.864472729203204, "cc_by_type": { "choice": 31.623931623931618, "noul": -19.999999999999996, "score": 38.96948656367799 }, "low_n": true, "n": 29, "n_by_type": { "choice": 15, "noul": 5, "score": 9 }, "score": 16.864472729203204 }, "financial_crime": { "accuracy": 0.5, "cc": 15.494473495061596, "cc_by_type": { "choice": 34.78260869565218, "noul": 0.0, "score": 11.700811789532606 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 15.494473495061596 }, "gaming": { "accuracy": 0.3793103448275862, "cc": -6.714417440674549, "cc_by_type": { "choice": 40.59405940594059, "noul": -42.85714285714286, "score": -17.88016887082138 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 7, "score": 11 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.5, "cc": 28.798707116861394, "cc_by_type": { "choice": 52.38095238095238, "noul": -60.0, "score": 94.0151689696318 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 5, "score": 1 }, "score": 28.798707116861394 }, "knowledge_graphs": { "accuracy": 0.391304347826087, "cc": -0.31602635093334186, "cc_by_type": { "choice": 35.87786259541985, "noul": -60.0, "score": 23.174058351780126 }, "low_n": true, "n": 23, "n_by_type": { "choice": 14, "noul": 5, "score": 4 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.36, "cc": 1.8570013967353607, "cc_by_type": { "choice": 83.73983739837398, "noul": -50.0, "score": -28.168833208167896 }, "low_n": true, "n": 25, "n_by_type": { "choice": 8, "noul": 8, "score": 9 }, "score": 1.8570013967353607 }, "legal_compliance": { "accuracy": 0.3076923076923077, "cc": -26.63894890640322, "cc_by_type": { "choice": 34.42622950819673, "noul": -100.0, "score": -14.343076227406382 }, "low_n": true, "n": 13, "n_by_type": { "choice": 8, "noul": 1, "score": 4 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.7241379310344828, "cc": 51.33693366193592, "cc_by_type": { "choice": 72.4770642201835, "noul": 50.0, "score": 31.533736765624287 }, "low_n": true, "n": 29, "n_by_type": { "choice": 14, "noul": 8, "score": 7 }, "score": 51.33693366193592 }, "model_routing": { "accuracy": 0.68, "cc": 48.65467531366116, "cc_by_type": { "choice": 75.87778075582953, "noul": 36.36363636363635, "score": 33.72260882151758 }, "low_n": false, "n": 150, "n_by_type": { "choice": 74, "noul": 44, "score": 32 }, "score": 48.65467531366116 }, "moderation_trust_safety": { "accuracy": 0.5625, "cc": 25.43524834873031, "cc_by_type": { "choice": 36.84210526315789, "noul": 19.999999999999996, "score": 19.463639783033038 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 5, "score": 3 }, "score": 25.43524834873031 }, "recruiting": { "accuracy": 0.6666666666666666, "cc": 52.73441114090357, "cc_by_type": { "choice": 84.92462311557789, "noul": 33.33333333333333, "score": 39.945276973799494 }, "low_n": true, "n": 21, "n_by_type": { "choice": 9, "noul": 6, "score": 6 }, "score": 52.73441114090357 }, "risk_assessment": { "accuracy": 0.23076923076923078, "cc": -34.30633768347406, "cc_by_type": { "choice": 5.660377358490565, "noul": -100.0, "score": -8.579390408912758 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 3, "score": 3 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.40625, "cc": -3.5610027209667976, "cc_by_type": { "choice": 36.254980079681275, "noul": -71.42857142857143, "score": 24.490583185989767 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.7272727272727273, "cc": 72.3432480209567, "cc_by_type": { "choice": 85.68019093078759, "noul": 100.0, "score": 31.349553132082498 }, "low_n": true, "n": 22, "n_by_type": { "choice": 9, "noul": 6, "score": 7 }, "score": 72.3432480209567 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 79.14914450840455, 86.43859611169106 ], "I": [ 28.905278037674446, 38.23083446327785 ], "capability": [ 55.16459221509819, 61.42516054822835 ], "composite": { "A": [ 12.260077031927597, 23.834682052129676 ], "B": [ 10.755937812612656, 21.913672526185227 ], "C": [ 8.513942383283057, 16.551862536201167 ] }, "seed": 21 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 10.073244581339713, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 85.49831885164681, "capability": 59.981611956556755, "intelligence": 34.464905061466695 }, "per_type": { "choice": { "accuracy": 0.6653333333333333, "calibration": 85.8172450800459, "cc": 56.64190706512352, "cc_by_type": { "choice": 56.64190706512352 }, "cc_tiered": 59.32833079627274, "cc_tiered_open": 64.76188322786868, "cc_tiered_sealed": 53.894778364676796, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 56.64190706512352 }, "noul": { "accuracy": 0.5226666666666666, "calibration": 86.51295898861834, "cc": 4.533333333333323, "cc_by_type": { "noul": 4.533333333333323 }, "cc_tiered": 13.294340924775709, "cc_tiered_open": 26.588681849551417, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 4.533333333333323 }, "score": { "accuracy": 0.5173333333333333, "calibration": 84.1647524862762, "cc": 27.98616472792205, "cc_by_type": { "score": 27.98616472792205 }, "cc_tiered": 30.772043463351643, "cc_tiered_open": 36.628943204920006, "cc_tiered_sealed": 24.915143721783284, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 27.98616472792205 } }, "raw_on_A": { "calibration": 83.94383086837013, "capability": 57.49826297860533, "intelligence": 31.05269508884054 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.014666666666666717, "n": 750 }, "noul": { "invalid_rate": 0.029333333333333322, "n": 375 }, "score": { "invalid_rate": 0.01866666666666672, "n": 375 } }, "views": { "Balanced 33:33:33": 16.62869989264526, "Emphasis on Accuracy 60:20:20": 14.53757347040618, "Emphasis on Cost 20:20:60": 16.08558260894422, "Emphasis on Speed 20:60:20": 20.21989737661098, "Intelligence only": 12.230517698051182 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Nandakishor M", "axes": { "calibration": 62.93608607777928, "cost": 61.47445744011434, "intelligence": 3.0532964296406795, "speed": 89.45598874418484 }, "calibration": { "parts": { "choice": { "ece_hard": 0.20193798693071482, "ece_score": 59.61240261385703, "mean_tvd": null, "n": 750, "n_ece_hard": 328, "n_tvd": 0, "score": 59.61240261385703 }, "noul": { "abstention_rate": 0.6799999999999999, "brier": 0.28375891613054305, "ece": 0.17069524022939242, "n": 375, "score": 65.86095195412152 }, "score": { "interval90_coverage": 0.728, "interval90_width": 0.4346666666666666, "n": 375, "nrps": 0.195768816097399, "score": 63.33490366535929, "top_ece": 0.2687665552977076 } }, "score": 62.93608607777928 }, "capability": 32.99469125370998, "composite_ci95": { "A": [ 0.004091095686330164, 0.24741438251652387 ], "B": [ 0.00263290972469324, 0.17191977795609545 ], "C": [ 0.002841038671062613, 0.17181554341425267 ] }, "cost": { "basis": "v1.5-M2 base-model reference: Vega 0.8B is a decision engine on a frozen Qwen3.5-0.8B backbone (Qwen/Qwen3.5-0.8B, resolves in pricing_v15 to deepinfra:Qwen/Qwen3.5-0.8B). DeepInfra list price USD 0.01/M input, USD 0.05/M output from the dated DeepInfra catalog snapshot 2026-09-25 (jevbench-v15-measure-20260925/receipts/DEEPINFRA-MODELS-20260925.json, sha256 84eb7926acd4e93b71d695be24898e2ca999b20011d1b66cc0348ce7c6c3546f; source https://api.deepinfra.com/models/list), read by the frozen pricing_v15 on 2026-10-10. The model generates no text: 0 output tokens, cost = input tokens x USD 0.01/M.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.019239100000000002 }, "display": "Vega 0.8B", "endpoint_condition": "evaluator-owned Lium RTX A6000 48 GB pod (Kansas City, US), offline after setup (egress cut and proven), reviewed vegaml==0.6.1 wheel, backbone bf16 / engine fp32, context 73,728 (longer states cut on the right), reference PyTorch path for linear attention (no flash-linear-attention / causal-conv1d)", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 4.066515604051393, "I_sealed": 2.040077255229966, "base": 3.0532964296406795, "excess": -8.046806232518286, "gap": 2.0264383488214266, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 12.19954681215418, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 14.378378378378391, "hard": 9.405473419314262, "judge": 16.94133157547792, "standard": 9.585600669736293 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 14.28571428571428, "hard": -73.91304347826086, "judge": -50.0, "standard": -55.55555555555556 } }, "open|score": { "cc": 0.0, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 46.625135136625836, "hard": -0.915170580118807, "judge": -9.170003719084164, "standard": -19.439472746148546 } }, "sealed|choice": { "cc": 6.1202317656898995, "n": { "easy": 72, "hard": 260, "judge": 141, "standard": 127 }, "tiers": { "easy": -8.874752320367087, "hard": 18.874354182697587, "judge": 3.2012581807112888, "standard": -7.512060647829066 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 39, "hard": 132, "judge": 72, "standard": 57 }, "tiers": { "easy": -23.076923076923073, "hard": -69.6969696969697, "judge": -58.33333333333333, "standard": -78.94736842105263 } }, "sealed|score": { "cc": 0.0, "n": { "easy": 33, "hard": 115, "judge": 75, "standard": 77 }, "tiers": { "easy": -0.09009355832638732, "hard": -3.84788500990465, "judge": -4.989943518704232, "standard": -15.600158434735768 } } } }, "jevbench_score": 0.04022166962784317, "key": "vegaml-0.8b", "last_measured_on": "2026-10-10", "licence": "vegaml package Apache-2.0; HF checkpoint repo declares no licence field (implied, not declared); backbone Qwen3.5 Apache-2.0", "listing": "ranked", "measured_in": "v1.6.5", "measurement_date_status": "measured", "model_pin": "nandakishorm/vega-08b-public-intents@73a76a4f6c4fe672e0daa2d45ddc355afc5bc9dd", "noul_decisive": { "acc_among_decisive": 0.6333333333333333, "decisive": 120, "decisive_rate": 0.32, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 9, "ranked": true, "ranks": { "A": 9, "B": 9, "C": 9, "capability": 8 }, "repo": "https://huggingface.co/nandakishorm/vega-08b-public-intents", "scores": { "A": 0.04022166962784317, "B": 0.0266984340300977, "C": 0.027931715019335536 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.3191188830882311, "p50_s_raw": 0.08455944154411554, "p95_s_adjusted": 0.35518009765073655, "p95_s_raw": 0.10259004882536829 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.27586206896551724, "cc": -19.36556571427138, "cc_by_type": { "choice": 25.0, "noul": -66.66666666666667, "score": -16.43003047614746 }, "low_n": true, "n": 29, "n_by_type": { "choice": 16, "noul": 6, "score": 7 }, "score": 0.0 }, "calibration_gen": { "accuracy": 0.19642857142857142, "cc": -24.988246708550232, "cc_by_type": { "choice": -4.813664596273289, "noul": -76.92307692307692, "score": 6.772001393699512 }, "low_n": false, "n": 112, "n_by_type": { "choice": 56, "noul": 26, "score": 30 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.15306122448979592, "cc": -36.524602578293646, "cc_by_type": { "choice": -3.7037037037037055, "noul": -100.0, "score": -5.870104031177226 }, "low_n": false, "n": 98, "n_by_type": { "choice": 36, "noul": 28, "score": 34 }, "score": 0.0 }, "fairness": { "accuracy": 0.2222222222222222, "cc": -26.181811039531453, "cc_by_type": { "choice": -5.882352941176475, "noul": -100.0, "score": 27.336919822582118 }, "low_n": true, "n": 27, "n_by_type": { "choice": 12, "noul": 6, "score": 9 }, "score": 0.0 }, "jaggedness": { "accuracy": 0.2864864864864865, "cc": -17.791678833140427, "cc_by_type": { "choice": 4.12227883279296, "noul": -23.076923076923073, "score": -34.42039225529116 }, "low_n": false, "n": 185, "n_by_type": { "choice": 97, "noul": 39, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.22348484848484848, "cc": -10.709273113952321, "cc_by_type": { "choice": -3.111936832326967, "noul": -47.945205479452056, "score": 18.929322969922058 }, "low_n": false, "n": 264, "n_by_type": { "choice": 131, "noul": 73, "score": 60 }, "score": 0.0 }, "long_state": { "accuracy": 0.1724137931034483, "cc": -34.941907856494, "cc_by_type": { "choice": 4.761904761904764, "noul": -81.81818181818181, "score": -27.769446513204944 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 11, "score": 7 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.41935483870967744, "cc": -14.749178737264756, "cc_by_type": { "choice": 7.692307692307688, "noul": 25.0, "score": -76.93984390410196 }, "low_n": false, "n": 31, "n_by_type": { "choice": 15, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.2634920634920635, "cc": -18.949056458195674, "cc_by_type": { "choice": 12.225705329153609, "noul": -67.81609195402298, "score": -1.2567827497176554 }, "low_n": false, "n": 315, "n_by_type": { "choice": 146, "noul": 87, "score": 82 }, "score": 0.0 }, "negation_pair_gen": { "accuracy": 0.19642857142857142, "cc": -42.402473689854006, "cc_by_type": { "choice": -9.523809523809524, "noul": -60.0, "score": -57.6836115457525 }, "low_n": false, "n": 56, "n_by_type": { "choice": 28, "noul": 20, "score": 8 }, "score": 0.0 }, "option_order": { "accuracy": 0.3157894736842105, "cc": 13.42952275249723, "cc_by_type": { "choice": 13.42952275249723 }, "low_n": false, "n": 57, "n_by_type": { "choice": 57 }, "score": 13.42952275249723 }, "perturb_pair": { "accuracy": 0.45, "cc": 6.179349639681301, "cc_by_type": { "choice": 29.22673656618611, "noul": -14.28571428571429, "score": 3.5970266385720873 }, "low_n": false, "n": 60, "n_by_type": { "choice": 34, "noul": 14, "score": 12 }, "score": 6.179349639681301 }, "usecase": { "accuracy": 0.29535864978902954, "cc": -19.00704443525642, "cc_by_type": { "choice": 21.37213335902873, "noul": -71.9298245614035, "score": -6.46344210339449 }, "low_n": false, "n": 237, "n_by_type": { "choice": 111, "noul": 57, "score": 69 }, "score": 0.0 } }, "lang": { "ar": { "accuracy": 0.45454545454545453, "cc": -7.714580910237068, "cc_by_type": { "choice": -6.666666666666665, "noul": 0.0, "score": -16.47707606404454 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 4, "score": 2 }, "score": 0.0 }, "cs": { "accuracy": 0.25, "cc": -38.135593220338976, "cc_by_type": { "choice": 23.72881355932204, "noul": -100.0 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 3 }, "score": 0.0 }, "da": { "accuracy": 0.2916666666666667, "cc": -23.813705445205983, "cc_by_type": { "choice": 5.325443786982243, "noul": -100.0, "score": 23.23343987739981 }, "low_n": true, "n": 24, "n_by_type": { "choice": 11, "noul": 5, "score": 8 }, "score": 0.0 }, "de": { "accuracy": 0.26666666666666666, "cc": -9.48435284148812, "cc_by_type": { "choice": 13.095956547978272, "noul": -33.333333333333336, "score": -8.215681739109293 }, "low_n": false, "n": 45, "n_by_type": { "choice": 24, "noul": 9, "score": 12 }, "score": 0.0 }, "el": { "accuracy": 0.4444444444444444, "cc": -38.08299402731068, "cc_by_type": { "choice": 39.7590361445783, "noul": -60.0, "score": -94.00801822651034 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 5, "score": 2 }, "score": 0.0 }, "en": { "accuracy": 0.2529043789097408, "cc": -19.4845221445307, "cc_by_type": { "choice": 6.449803701626394, "noul": -60.902255639097746, "score": -4.00111449612075 }, "low_n": false, "n": 1119, "n_by_type": { "choice": 577, "noul": 266, "score": 276 }, "score": 0.0 }, "es": { "accuracy": 0.35714285714285715, "cc": -3.020542459003753, "cc_by_type": { "choice": -17.81818181818182, "noul": 9.090909090909083, "score": -0.3343546497385219 }, "low_n": true, "n": 28, "n_by_type": { "choice": 10, "noul": 11, "score": 7 }, "score": 0.0 }, "fi": { "accuracy": 0.4444444444444444, "cc": 1.6431924882629083, "cc_by_type": { "choice": 36.61971830985915, "noul": -33.333333333333336 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 3 }, "score": 1.6431924882629083 }, "fr": { "accuracy": 0.06666666666666667, "cc": -42.623688574546726, "cc_by_type": { "choice": -26.93935119887165, "noul": -100.0, "score": -0.9317145247685232 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 4, "score": 6 }, "score": 0.0 }, "hi": { "accuracy": 0.2, "cc": -28.83602734828803, "cc_by_type": { "choice": -11.801242236024846, "noul": -40.0, "score": -34.70683980883924 }, "low_n": false, "n": 30, "n_by_type": { "choice": 11, "noul": 10, "score": 9 }, "score": 0.0 }, "id": { "accuracy": 0.0, "cc": -64.51612903225806, "cc_by_type": { "choice": -29.032258064516125, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 0.0 }, "it": { "accuracy": 0.5294117647058824, "cc": 5.113250586139728, "cc_by_type": { "choice": 49.22425952045134, "noul": 0.0, "score": -33.88450776203216 }, "low_n": true, "n": 17, "n_by_type": { "choice": 5, "noul": 4, "score": 8 }, "score": 5.113250586139728 }, "ja": { "accuracy": 0.21875, "cc": -21.637722673747195, "cc_by_type": { "choice": 2.4390243902439046, "noul": -71.42857142857143, "score": 4.076379017085941 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "ko": { "accuracy": 0.18181818181818182, "cc": -20.186550397257573, "cc_by_type": { "choice": 1.6393442622950836, "noul": 0.0, "score": -62.1989954540678 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.2727272727272727, "cc": -1.2255967997920887, "cc_by_type": { "choice": -9.890109890109892, "noul": -50.0, "score": 56.213319490733625 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 0.0 }, "nl": { "accuracy": 0.07692307692307693, "cc": -33.67879710336313, "cc_by_type": { "choice": -0.2785515320334245, "noul": -100.0, "score": -0.7578397780559687 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 6, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.0, "cc": -65.57377049180329, "cc_by_type": { "choice": -31.14754098360656, "noul": -100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 3 }, "score": 0.0 }, "pl": { "accuracy": 0.275, "cc": -16.97531167960543, "cc_by_type": { "choice": 29.164844774814167, "noul": -80.0, "score": -0.0907798136304594 }, "low_n": false, "n": 40, "n_by_type": { "choice": 16, "noul": 10, "score": 14 }, "score": 0.0 }, "pt": { "accuracy": 0.5, "cc": 21.729761287554258, "cc_by_type": { "choice": 81.90954773869345, "noul": -66.66666666666667, "score": 49.946402790636014 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 6, "score": 5 }, "score": 21.729761287554258 }, "sv": { "accuracy": 0.25, "cc": -8.670532037806744, "cc_by_type": { "choice": 11.111111111111109, "noul": -100.0, "score": 62.87729277546865 }, "low_n": true, "n": 8, "n_by_type": { "choice": 3, "noul": 4, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.18181818181818182, "cc": -58.40033972849334, "cc_by_type": { "choice": -5.26315789473684, "noul": -100.0, "score": -69.93786129074316 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 3, "score": 3 }, "score": 0.0 }, "uk": { "accuracy": 0.18181818181818182, "cc": -43.03199883446496, "cc_by_type": { "choice": 13.043478260869565, "noul": -100.0, "score": -42.13947476426443 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 1, "score": 4 }, "score": 0.0 }, "zh": { "accuracy": 0.5, "cc": 18.615495552140683, "cc_by_type": { "choice": 35.13513513513514, "noul": -33.333333333333336, "score": 54.04468485462024 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 18.615495552140683 } }, "type": { "choice": { "accuracy": 0.2866666666666667, "cc": 7.583347728450512, "cc_by_type": { "choice": 7.583347728450512 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 7.583347728450512 }, "noul": { "accuracy": 0.20266666666666666, "cc": -59.46666666666667, "cc_by_type": { "noul": -59.46666666666667 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.264, "cc": -4.914138326410811, "cc_by_type": { "score": -4.914138326410811 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "usecase": { "advertising": { "accuracy": 0.25, "cc": -42.23115369530359, "cc_by_type": { "choice": 11.037891268533771, "noul": -100.0, "score": -37.73135235444454 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.4166666666666667, "cc": 10.053783668961861, "cc_by_type": { "choice": 21.440261865793783, "noul": -19.999999999999996, "score": 28.721089141091795 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 10.053783668961861 }, "customer_support": { "accuracy": 0.25, "cc": -14.695878083746338, "cc_by_type": { "choice": -12.90322580645161, "noul": -50.0, "score": 18.815591555212595 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 8, "score": 4 }, "score": 0.0 }, "demand_forecasting": { "accuracy": 0.07692307692307693, "cc": -52.52931897614329, "cc_by_type": { "choice": -20.481927710843376, "noul": -100.0, "score": -37.10602921758648 }, "low_n": true, "n": 26, "n_by_type": { "choice": 11, "noul": 4, "score": 11 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.1111111111111111, "cc": -38.90478330755122, "cc_by_type": { "choice": 6.832298136645961, "noul": -100.0, "score": -23.546648059299624 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 5, "score": 6 }, "score": 0.0 }, "feature_extraction": { "accuracy": 0.3793103448275862, "cc": -13.328851245398967, "cc_by_type": { "choice": 31.623931623931618, "noul": -60.0, "score": -11.61048536012852 }, "low_n": true, "n": 29, "n_by_type": { "choice": 15, "noul": 5, "score": 9 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.34615384615384615, "cc": -18.503379574597428, "cc_by_type": { "choice": 47.82608695652174, "noul": -75.0, "score": -28.33622568031402 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 0.0 }, "gaming": { "accuracy": 0.27586206896551724, "cc": -25.77506638432793, "cc_by_type": { "choice": -6.9306930693069315, "noul": -71.42857142857143, "score": 1.0340653448945747 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 7, "score": 11 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.14285714285714285, "cc": -30.119048798099303, "cc_by_type": { "choice": -11.11111111111111, "noul": -100.0, "score": 20.7539647168132 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 5, "score": 1 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.17391304347826086, "cc": -34.69531570651282, "cc_by_type": { "choice": -0.7633587786259586, "noul": -100.0, "score": -3.3225883409125245 }, "low_n": true, "n": 23, "n_by_type": { "choice": 14, "noul": 5, "score": 4 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.48, "cc": 8.261797778580346, "cc_by_type": { "choice": 67.47967479674797, "noul": -25.0, "score": -17.694281461006934 }, "low_n": true, "n": 25, "n_by_type": { "choice": 8, "noul": 8, "score": 9 }, "score": 8.261797778580346 }, "legal_compliance": { "accuracy": 0.23076923076923078, "cc": -31.134132674360444, "cc_by_type": { "choice": 18.032786885245905, "noul": -100.0, "score": -11.435184908327244 }, "low_n": true, "n": 13, "n_by_type": { "choice": 8, "noul": 1, "score": 4 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.3103448275862069, "cc": -9.67272641203819, "cc_by_type": { "choice": 17.431192660550458, "noul": -75.0, "score": 28.550628103334972 }, "low_n": true, "n": 29, "n_by_type": { "choice": 14, "noul": 8, "score": 7 }, "score": 0.0 }, "model_routing": { "accuracy": 0.29333333333333333, "cc": 2.0858721093715658, "cc_by_type": { "choice": 1.9029750737067974, "noul": -18.181818181818176, "score": 22.536459436226075 }, "low_n": false, "n": 150, "n_by_type": { "choice": 74, "noul": 44, "score": 32 }, "score": 2.0858721093715658 }, "moderation_trust_safety": { "accuracy": 0.3125, "cc": -22.66548316017918, "cc_by_type": { "choice": 36.84210526315789, "noul": -100.0, "score": -4.8385547436954335 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 5, "score": 3 }, "score": 0.0 }, "recruiting": { "accuracy": 0.38095238095238093, "cc": 1.0762044705122094, "cc_by_type": { "choice": 69.84924623115576, "noul": -66.66666666666667, "score": 0.046033847047532994 }, "low_n": true, "n": 21, "n_by_type": { "choice": 9, "noul": 6, "score": 6 }, "score": 1.0762044705122094 }, "risk_assessment": { "accuracy": 0.3076923076923077, "cc": -21.111851703106247, "cc_by_type": { "choice": 24.528301886792452, "noul": -100.0, "score": 12.136143003888812 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 3, "score": 3 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.1875, "cc": -22.40823864613203, "cc_by_type": { "choice": 4.382470119521911, "noul": -71.42857142857143, "score": -0.17861462934656291 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.2727272727272727, "cc": -35.0856709518421, "cc_by_type": { "choice": 14.08114558472553, "noul": -66.66666666666667, "score": -52.671491773585146 }, "low_n": true, "n": 22, "n_by_type": { "choice": 9, "noul": 6, "score": 7 }, "score": 0.0 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 56.95703626645148, 66.84035944861385 ], "I": [ 1.394797842409951, 5.784545846198247 ], "capability": [ 29.939799695068736, 35.75319772071871 ], "composite": { "A": [ 0.004091095686330164, 0.24741438251652387 ], "B": [ 0.00263290972469324, 0.17191977795609545 ], "C": [ 0.002841038671062613, 0.17181554341425267 ] }, "seed": 22 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 10.073244581339713, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 62.93608607777928, "capability": 32.99469125370998, "intelligence": 3.0532964296406795 }, "per_type": { "choice": { "accuracy": 0.2866666666666667, "calibration": 59.61240261385703, "cc": 7.583347728450512, "cc_by_type": { "choice": 7.583347728450512 }, "cc_tiered": 9.15988928892204, "cc_tiered_open": 12.19954681215418, "cc_tiered_sealed": 6.1202317656898995, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 7.583347728450512 }, "noul": { "accuracy": 0.20266666666666666, "calibration": 65.86095195412152, "cc": -59.46666666666667, "cc_by_type": { "noul": -59.46666666666667 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.264, "calibration": 63.33490366535929, "cc": -4.914138326410811, "cc_by_type": { "score": -4.914138326410811 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "raw_on_A": { "calibration": 59.68341710343884, "capability": 30.94517056072779, "intelligence": 2.206924018016739 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 0.03151658754618596, "Emphasis on Accuracy 60:20:20": 0.018460810494659353, "Emphasis on Cost 20:20:60": 0.048117449859240324, "Emphasis on Speed 20:60:20": 0.04941520456803552, "Intelligence only": 0.011385887829608154 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Nandakishor M", "axes": { "calibration": 58.68489087403218, "cost": 47.16081979852447, "intelligence": 16.689214224692094, "speed": 84.39386641436894 }, "calibration": { "parts": { "choice": { "ece_hard": 0.2574125103434302, "ece_score": 48.517497931313954, "mean_tvd": null, "n": 750, "n_ece_hard": 328, "n_tvd": 0, "score": 48.517497931313954 }, "noul": { "abstention_rate": 0.30400000000000005, "brier": 0.2781281471262366, "ece": 0.2285300562531464, "n": 375, "score": 54.293988749370726 }, "score": { "interval90_coverage": 0.8346666666666667, "interval90_width": 0.49288888888888904, "n": 375, "nrps": 0.16226214991492005, "score": 73.24318594141187, "top_ece": 0.18643706562842138 } }, "score": 58.68489087403218 }, "capability": 37.687052549362136, "composite_ci95": { "A": [ 1.6852320401608203, 6.813845133691371 ], "B": [ 1.3079888804084314, 5.761661265907114 ], "C": [ 1.1703000278894586, 4.731836898396786 ] }, "cost": { "basis": "v1.5-M2 base-model reference: Vega 4B is a decision engine on a frozen Qwen3.5-4B backbone (Qwen/Qwen3.5-4B, resolves in pricing_v15 to deepinfra:Qwen/Qwen3.5-4B). DeepInfra list price USD 0.03/M input, USD 0.15/M output from the dated DeepInfra catalog snapshot 2026-09-25 (jevbench-v15-measure-20260925/receipts/DEEPINFRA-MODELS-20260925.json, sha256 84eb7926acd4e93b71d695be24898e2ca999b20011d1b66cc0348ce7c6c3546f; source https://api.deepinfra.com/models/list), read by the frozen pricing_v15 on 2026-10-10. The listing is flagged by pricing_v15 as deprecated after the 25 Sep cut-off (DEPRECATED_AFTER_CUTOFF) and stays frozen with that note. The model generates no text: 0 output tokens, cost = input tokens x USD 0.03/M.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.057717300000000006 }, "display": "Vega 4B", "endpoint_condition": "evaluator-owned Lium RTX A6000 48 GB pod (Des Moines, US), offline after setup (egress cut and proven), reviewed vegaml==0.6.1 wheel, backbone bf16 / engine fp32, context 73,728 (longer states cut on the right), reference PyTorch path for linear attention (no flash-linear-attention / causal-conv1d)", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 22.023529046873755, "I_sealed": 11.354899402510437, "base": 16.689214224692094, "excess": 0.5953850630236044, "gap": 10.668629644363318, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 42.15141777710706, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 45.51351351351353, "hard": 30.16671909405474, "judge": 48.58272907053395, "standard": 54.79280033486815 } }, "open|noul": { "cc": 9.665976535541756, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 57.14285714285714, "hard": 30.434782608695656, "judge": -19.999999999999996, "standard": -11.111111111111116 } }, "open|score": { "cc": 14.253192827972441, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 36.64297240164319, "hard": 6.501670939484594, "judge": 20.791162375101624, "standard": 8.754392497418984 } }, "sealed|choice": { "cc": 25.42491346722165, "n": { "easy": 72, "hard": 260, "judge": 141, "standard": 127 }, "tiers": { "easy": 32.4225675252894, "hard": 23.821039903264804, "judge": 22.378367409060942, "standard": 29.70365265334253 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 39, "hard": 132, "judge": 72, "standard": 57 }, "tiers": { "easy": 7.692307692307687, "hard": -22.72727272727273, "judge": 8.333333333333325, "standard": -12.28070175438597 } }, "sealed|score": { "cc": 8.639784740309665, "n": { "easy": 33, "hard": 115, "judge": 75, "standard": 77 }, "tiers": { "easy": 23.787017462891814, "hard": 6.903920787097362, "judge": 10.43509835324301, "standard": 1.8449258660431789 } } } }, "jevbench_score": 3.603902446043355, "key": "vegaml-4b", "last_measured_on": "2026-10-10", "licence": "vegaml package Apache-2.0; HF checkpoint repo declares no licence field (implied, not declared); backbone Qwen3.5 Apache-2.0", "listing": "ranked", "measured_in": "v1.6.5", "measurement_date_status": "measured", "model_pin": "nandakishorm/vega-08b-public-intents@73a76a4f6c4fe672e0daa2d45ddc355afc5bc9dd", "noul_decisive": { "acc_among_decisive": 0.6819923371647509, "decisive": 261, "decisive_rate": 0.696, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 6, "ranked": true, "ranks": { "A": 6, "B": 6, "C": 6, "capability": 7 }, "repo": "https://huggingface.co/nandakishorm/vega-08b-public-intents", "scores": { "A": 3.603902446043355, "B": 2.9164270830443, "C": 2.502710031974551 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.5304726889356971, "p50_s_raw": 0.19023634446784854, "p95_s_adjusted": 0.6854098306968808, "p95_s_raw": 0.2677049153484404 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.5862068965517241, "cc": 32.10254092129096, "cc_by_type": { "choice": 50.0, "noul": 0.0, "score": 46.30762276387288 }, "low_n": true, "n": 29, "n_by_type": { "choice": 16, "noul": 6, "score": 7 }, "score": 32.10254092129096 }, "calibration_gen": { "accuracy": 0.3482142857142857, "cc": -7.764599070714067, "cc_by_type": { "choice": 16.14906832298137, "noul": -38.46153846153846, "score": -0.9813270735851143 }, "low_n": false, "n": 112, "n_by_type": { "choice": 56, "noul": 26, "score": 30 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.35714285714285715, "cc": 4.1474482072018475, "cc_by_type": { "choice": 7.407407407407411, "noul": -7.14285714285714, "score": 12.177794357055271 }, "low_n": false, "n": 98, "n_by_type": { "choice": 36, "noul": 28, "score": 34 }, "score": 4.1474482072018475 }, "fairness": { "accuracy": 0.7037037037037037, "cc": 46.819844941220424, "cc_by_type": { "choice": 64.70588235294117, "noul": 0.0, "score": 75.75365247072008 }, "low_n": true, "n": 27, "n_by_type": { "choice": 12, "noul": 6, "score": 9 }, "score": 46.819844941220424 }, "jaggedness": { "accuracy": 0.34054054054054056, "cc": -9.581651647472881, "cc_by_type": { "choice": 19.407132931912926, "noul": -28.205128205128204, "score": -19.946959669203366 }, "low_n": false, "n": 185, "n_by_type": { "choice": 97, "noul": 39, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.5303030303030303, "cc": 35.11456970853259, "cc_by_type": { "choice": 34.9744542498839, "noul": 28.767123287671236, "score": 41.602131588042646 }, "low_n": false, "n": 264, "n_by_type": { "choice": 131, "noul": 73, "score": 60 }, "score": 35.11456970853259 }, "long_state": { "accuracy": 0.3448275862068966, "cc": -4.35719562611893, "cc_by_type": { "choice": 16.66666666666667, "noul": -27.27272727272727, "score": -2.4655262722961924 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 11, "score": 7 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.3548387096774194, "cc": -22.26927167429365, "cc_by_type": { "choice": 35.38461538461538, "noul": -75.0, "score": -27.192430407496325 }, "low_n": false, "n": 31, "n_by_type": { "choice": 15, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.4603174603174603, "cc": 15.9575022436564, "cc_by_type": { "choice": 35.512763098970005, "noul": -1.1494252873563204, "score": 13.509168919355519 }, "low_n": false, "n": 315, "n_by_type": { "choice": 146, "noul": 87, "score": 82 }, "score": 15.9575022436564 }, "negation_pair_gen": { "accuracy": 0.32142857142857145, "cc": -18.937881655648187, "cc_by_type": { "choice": -9.523809523809524, "noul": 0.0, "score": -47.28983544313503 }, "low_n": false, "n": 56, "n_by_type": { "choice": 28, "noul": 20, "score": 8 }, "score": 0.0 }, "option_order": { "accuracy": 0.5614035087719298, "cc": 44.50610432852386, "cc_by_type": { "choice": 44.50610432852386 }, "low_n": false, "n": 57, "n_by_type": { "choice": 57 }, "score": 44.50610432852386 }, "perturb_pair": { "accuracy": 0.5333333333333333, "cc": 41.1419841739421, "cc_by_type": { "choice": 25.29488859764089, "noul": 42.85714285714286, "score": 55.273921067042544 }, "low_n": false, "n": 60, "n_by_type": { "choice": 34, "noul": 14, "score": 12 }, "score": 41.1419841739421 }, "usecase": { "accuracy": 0.379746835443038, "cc": -1.691456792175881, "cc_by_type": { "choice": 28.30988629793796, "noul": -22.807017543859654, "score": -10.577239130605953 }, "low_n": false, "n": 237, "n_by_type": { "choice": 111, "noul": 57, "score": 69 }, "score": 0.0 } }, "lang": { "ar": { "accuracy": 0.5454545454545454, "cc": -14.905334686220934, "cc_by_type": { "choice": 73.33333333333334, "noul": -50.0, "score": -68.04933739199615 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 4, "score": 2 }, "score": 0.0 }, "cs": { "accuracy": 0.375, "cc": -4.802259887005647, "cc_by_type": { "choice": 23.72881355932204, "noul": -33.333333333333336 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 3 }, "score": 0.0 }, "da": { "accuracy": 0.4583333333333333, "cc": 22.143800617963134, "cc_by_type": { "choice": 64.49704142011835, "noul": 19.999999999999996, "score": -18.065639566228953 }, "low_n": true, "n": 24, "n_by_type": { "choice": 11, "noul": 5, "score": 8 }, "score": 22.143800617963134 }, "de": { "accuracy": 0.4444444444444444, "cc": 3.742298256027752, "cc_by_type": { "choice": 29.390464695232346, "noul": -33.333333333333336, "score": 15.169763406184245 }, "low_n": false, "n": 45, "n_by_type": { "choice": 24, "noul": 9, "score": 12 }, "score": 3.742298256027752 }, "el": { "accuracy": 0.5, "cc": -7.267096986083452, "cc_by_type": { "choice": 51.80722891566265, "noul": -60.0, "score": -13.608519873913005 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 5, "score": 2 }, "score": 0.0 }, "en": { "accuracy": 0.4289544235924933, "cc": 10.591540002599032, "cc_by_type": { "choice": 26.86483454851368, "noul": -5.263157894736848, "score": 10.17294335402027 }, "low_n": false, "n": 1119, "n_by_type": { "choice": 577, "noul": 266, "score": 276 }, "score": 10.591540002599032 }, "es": { "accuracy": 0.5357142857142857, "cc": 23.735811235182705, "cc_by_type": { "choice": 47.63636363636363, "noul": 9.090909090909083, "score": 14.480160978275391 }, "low_n": true, "n": 28, "n_by_type": { "choice": 10, "noul": 11, "score": 7 }, "score": 23.735811235182705 }, "fi": { "accuracy": 0.2222222222222222, "cc": -19.483568075117372, "cc_by_type": { "choice": -5.63380281690141, "noul": -33.333333333333336 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 3 }, "score": 0.0 }, "fr": { "accuracy": 0.5333333333333333, "cc": 9.81849210967231, "cc_by_type": { "choice": 49.22425952045134, "noul": -50.0, "score": 30.231216808565588 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 4, "score": 6 }, "score": 9.81849210967231 }, "hi": { "accuracy": 0.36666666666666664, "cc": -2.9723800154374516, "cc_by_type": { "choice": 37.888198757763966, "noul": -40.0, "score": -6.805338804076322 }, "low_n": false, "n": 30, "n_by_type": { "choice": 11, "noul": 10, "score": 9 }, "score": 0.0 }, "id": { "accuracy": 0.16666666666666666, "cc": -14.516129032258062, "cc_by_type": { "choice": -29.032258064516125, "noul": 0.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 0.0 }, "it": { "accuracy": 0.5882352941176471, "cc": 41.890588956237984, "cc_by_type": { "choice": 74.61212976022568, "noul": 50.0, "score": 1.059637108488276 }, "low_n": true, "n": 17, "n_by_type": { "choice": 5, "noul": 4, "score": 8 }, "score": 41.890588956237984 }, "ja": { "accuracy": 0.34375, "cc": 6.462836027337623, "cc_by_type": { "choice": -5.691056910569103, "noul": -14.28571428571429, "score": 39.36527927829626 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 6.462836027337623 }, "ko": { "accuracy": 0.36363636363636365, "cc": 18.2762217539951, "cc_by_type": { "choice": 34.42622950819673, "noul": 0.0, "score": 20.402435753788563 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 2, "score": 5 }, "score": 18.2762217539951 }, "mixed": { "accuracy": 0.7272727272727273, "cc": 59.83071388979721, "cc_by_type": { "choice": 56.04395604395604, "noul": 100.0, "score": 23.448185625435602 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 59.83071388979721 }, "nl": { "accuracy": 0.15384615384615385, "cc": -18.26355956150215, "cc_by_type": { "choice": -25.34818941504178, "noul": -66.66666666666667, "score": 37.224177397202 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 6, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.7142857142857143, "cc": 67.21311475409837, "cc_by_type": { "choice": 34.42622950819673, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 3 }, "score": 67.21311475409837 }, "pl": { "accuracy": 0.525, "cc": 43.18418848568871, "cc_by_type": { "choice": 68.51770878880629, "noul": 60.00000000000001, "score": 1.0348566682598204 }, "low_n": false, "n": 40, "n_by_type": { "choice": 16, "noul": 10, "score": 14 }, "score": 43.18418848568871 }, "pt": { "accuracy": 0.4444444444444444, "cc": 15.083741051175762, "cc_by_type": { "choice": 27.638190954773865, "noul": -33.333333333333336, "score": 50.94636553208676 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 6, "score": 5 }, "score": 15.083741051175762 }, "sv": { "accuracy": 0.125, "cc": -24.998495011569247, "cc_by_type": { "choice": -33.333333333333336, "noul": -50.0, "score": 8.337848298625605 }, "low_n": true, "n": 8, "n_by_type": { "choice": 3, "noul": 4, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.6363636363636364, "cc": 20.74230849140807, "cc_by_type": { "choice": 47.368421052631575, "noul": 100.0, "score": -85.14149557840736 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 3, "score": 3 }, "score": 20.74230849140807 }, "uk": { "accuracy": 0.18181818181818182, "cc": -41.58877432033782, "cc_by_type": { "choice": 13.043478260869565, "noul": -100.0, "score": -37.80980122188302 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 1, "score": 4 }, "score": 0.0 }, "zh": { "accuracy": 0.75, "cc": 65.40226528868891, "cc_by_type": { "choice": 67.56756756756756, "noul": 33.33333333333333, "score": 95.30589496516582 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 65.40226528868891 } }, "type": { "choice": { "accuracy": 0.4493333333333333, "cc": 28.65779927448609, "cc_by_type": { "choice": 28.65779927448609 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 28.65779927448609 }, "noul": { "accuracy": 0.4746666666666667, "cc": -5.066666666666664, "cc_by_type": { "noul": -5.066666666666664 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.3626666666666667, "cc": 9.783689886391333, "cc_by_type": { "score": 9.783689886391333 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 9.783689886391333 } }, "usecase": { "advertising": { "accuracy": 0.3333333333333333, "cc": -35.1534607494542, "cc_by_type": { "choice": 20.922570016474467, "noul": -100.0, "score": -26.382952264837066 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.5416666666666666, "cc": 15.916918386842573, "cc_by_type": { "choice": 41.08019639934533, "noul": 19.999999999999996, "score": -13.329441238817608 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 15.916918386842573 }, "customer_support": { "accuracy": 0.45, "cc": 8.926626674691754, "cc_by_type": { "choice": 35.483870967741936, "noul": 25.0, "score": -33.703990943666675 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 8, "score": 4 }, "score": 8.926626674691754 }, "demand_forecasting": { "accuracy": 0.23076923076923078, "cc": -37.51418716919469, "cc_by_type": { "choice": 3.614457831325295, "noul": -100.0, "score": -16.15701933890936 }, "low_n": true, "n": 26, "n_by_type": { "choice": 11, "noul": 4, "score": 11 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.2222222222222222, "cc": -13.643840746791454, "cc_by_type": { "choice": 25.465838509316768, "noul": -60.0, "score": -6.397360749691128 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 5, "score": 6 }, "score": 0.0 }, "feature_extraction": { "accuracy": 0.27586206896551724, "cc": -6.646648832243309, "cc_by_type": { "choice": 5.9829059829059785, "noul": -19.999999999999996, "score": -5.922852479635909 }, "low_n": true, "n": 29, "n_by_type": { "choice": 15, "noul": 5, "score": 9 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.6538461538461539, "cc": 30.68478198913924, "cc_by_type": { "choice": 86.95652173913045, "noul": 25.0, "score": -19.902175771712738 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 30.68478198913924 }, "gaming": { "accuracy": 0.27586206896551724, "cc": -8.539638885229605, "cc_by_type": { "choice": -6.9306930693069315, "noul": -42.85714285714286, "score": 24.16891927076098 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 7, "score": 11 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.21428571428571427, "cc": -26.327496496464544, "cc_by_type": { "choice": 4.761904761904763, "noul": -60.0, "score": -23.744394251298395 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 5, "score": 1 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.21739130434782608, "cc": -7.707530000989269, "cc_by_type": { "choice": -0.7633587786259586, "noul": -19.999999999999996, "score": -2.359231224341851 }, "low_n": true, "n": 23, "n_by_type": { "choice": 14, "noul": 5, "score": 4 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.52, "cc": 19.542316672389997, "cc_by_type": { "choice": 51.21951219512196, "noul": 25.0, "score": -17.592562177951955 }, "low_n": true, "n": 25, "n_by_type": { "choice": 8, "noul": 8, "score": 9 }, "score": 19.542316672389997 }, "legal_compliance": { "accuracy": 0.23076923076923078, "cc": -17.21253914360014, "cc_by_type": { "choice": 18.032786885245905, "noul": -100.0, "score": 30.32959568395368 }, "low_n": true, "n": 13, "n_by_type": { "choice": 8, "noul": 1, "score": 4 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.41379310344827586, "cc": -0.8829038829667036, "cc_by_type": { "choice": 35.77981651376147, "noul": -50.0, "score": 11.571471837338422 }, "low_n": true, "n": 29, "n_by_type": { "choice": 14, "noul": 8, "score": 7 }, "score": 0.0 }, "model_routing": { "accuracy": 0.5, "cc": 24.299647249468666, "cc_by_type": { "choice": 42.106673813990895, "noul": 22.72727272727273, "score": 8.064995207142367 }, "low_n": false, "n": 150, "n_by_type": { "choice": 74, "noul": 44, "score": 32 }, "score": 24.299647249468666 }, "moderation_trust_safety": { "accuracy": 0.5, "cc": 21.15669006850672, "cc_by_type": { "choice": 21.052631578947366, "noul": 19.999999999999996, "score": 22.417438626572796 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 5, "score": 3 }, "score": 21.15669006850672 }, "recruiting": { "accuracy": 0.42857142857142855, "cc": 4.28970309222692, "cc_by_type": { "choice": 69.84924623115576, "noul": -33.333333333333336, "score": -23.646803621141665 }, "low_n": true, "n": 21, "n_by_type": { "choice": 9, "noul": 6, "score": 6 }, "score": 4.28970309222692 }, "risk_assessment": { "accuracy": 0.5384615384615384, "cc": 13.524055414252595, "cc_by_type": { "choice": 43.39622641509434, "noul": 33.33333333333333, "score": -36.15739350566989 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 3, "score": 3 }, "score": 13.524055414252595 }, "scientific_discovery": { "accuracy": 0.21875, "cc": -20.974690029781613, "cc_by_type": { "choice": 20.31872509960159, "noul": -71.42857142857143, "score": -11.814223760374997 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.5, "cc": 2.15502085448014, "cc_by_type": { "choice": 57.04057279236276, "noul": -33.333333333333336, "score": -17.242176895589 }, "low_n": true, "n": 22, "n_by_type": { "choice": 9, "noul": 6, "score": 7 }, "score": 2.15502085448014 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 53.44068752171121, 63.00430146367479 ], "I": [ 12.4680138524395, 21.489496876115624 ], "capability": [ 33.5959160465955, 41.3340258561405 ], "composite": { "A": [ 1.6852320401608203, 6.813845133691371 ], "B": [ 1.3079888804084314, 5.761661265907114 ], "C": [ 1.1703000278894586, 4.731836898396786 ] }, "seed": 23 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 10.073244581339713, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 58.68489087403218, "capability": 37.687052549362136, "intelligence": 16.689214224692094 }, "per_type": { "choice": { "accuracy": 0.4493333333333333, "calibration": 48.517497931313954, "cc": 28.65779927448609, "cc_by_type": { "choice": 28.65779927448609 }, "cc_tiered": 33.78816562216436, "cc_tiered_open": 42.15141777710706, "cc_tiered_sealed": 25.42491346722165, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 28.65779927448609 }, "noul": { "accuracy": 0.4746666666666667, "calibration": 54.293988749370726, "cc": -5.066666666666664, "cc_by_type": { "noul": -5.066666666666664 }, "cc_tiered": 4.832988267770878, "cc_tiered_open": 9.665976535541756, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.3626666666666667, "calibration": 73.24318594141187, "cc": 9.783689886391333, "cc_by_type": { "score": 9.783689886391333 }, "cc_tiered": 11.446488784141053, "cc_tiered_open": 14.253192827972441, "cc_tiered_sealed": 8.639784740309665, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 9.783689886391333 } }, "raw_on_A": { "calibration": 60.28751716714082, "capability": 38.880493243091394, "intelligence": 17.47346931904197 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 3.198325647386112, "Emphasis on Accuracy 60:20:20": 2.3288005981136943, "Emphasis on Cost 20:20:60": 3.6607417516659844, "Emphasis on Speed 20:60:20": 4.247790339721065, "Intelligence only": 1.654208859284027 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "org2ai", "axes": { "calibration": 86.58998638479137, "cost": 46.287007162302075, "intelligence": 51.951877017848766, "speed": 93.07746270925374 }, "calibration": { "parts": { "choice": { "ece_hard": 0.05465848238633132, "ece_score": 89.06830352273374, "mean_tvd": null, "n": 750, "n_ece_hard": 328, "n_tvd": 0, "score": 89.06830352273374 }, "noul": { "abstention_rate": 0.0, "brier": 0.16249185347848963, "ece": 0.09805762809316519, "n": 375, "score": 80.38847438136696 }, "score": { "interval90_coverage": 0.9813333333333333, "interval90_width": 0.6019999999999996, "n": 375, "nrps": 0.1056056821458286, "score": 90.31318125027343, "top_ece": 0.04406534642435141 } }, "score": 86.58998638479137 }, "capability": 69.27093170132007, "composite_ci95": { "A": [ 46.18008336139371, 55.5000390400574 ], "B": [ 43.548061690071826, 53.8197076967788 ], "C": [ 32.06950233430119, 48.51580772757769 ] }, "cost": { "basis": "Self-hosted open weights, no bookable tariff. v1.5-M2 base-model reference rule: base model Qwen/Qwen3.5-4B (model card base_model) -> DeepInfra Qwen/Qwen3.5-4B, USD 0.03/M input and USD 0.15/M output, frozen 25 Sep 2026 price cut-off snapshot (METHOD-v1.5-ADDENDUM-PRICING-INTERPRETATION-1 I-2/I-3; listing deprecated after the cut-off, stays the frozen v1.5 reference). Same basis as the published Wald 4B (v2.0) row. Read 2026-10-10.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.06172102 }, "display": "Wald 4B v2.1", "endpoint_condition": "evaluator-owned Lium GPU pod (RTX PRO 6000 96GB), offline read-only container (--network none); official vllm/vllm-openai:v0.30.0@sha256:8a69ffad... image; author's server wald_serve.native_vision from the model repo, --prompt-format plain (author RUNBOOK's JevBench format), serving.json defaults otherwise (effort none = one pass, noul_decisive_floor 0.81)", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 56.69080677700682, "I_sealed": 47.21294725869071, "base": 51.951877017848766, "excess": -0.5953850630236044, "gap": 9.477859518316109, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 70.9521688095628, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 84.43243243243245, "hard": 64.1396665618119, "judge": 76.26895187870798, "standard": 69.8618668899121 } }, "open|noul": { "cc": 66.16425120772948, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 56.52173913043479, "judge": 60.00000000000001, "standard": 77.77777777777777 } }, "open|score": { "cc": 32.9560003137282, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 42.310346490056276, "hard": 31.56850185540615, "judge": 32.18036106421194, "standard": 32.21728301648263 } }, "sealed|choice": { "cc": 55.12375685402749, "n": { "easy": 72, "hard": 260, "judge": 141, "standard": 127 }, "tiers": { "easy": 62.45698195849411, "hard": 49.543805650214345, "judge": 57.07980315559839, "standard": 59.6829772570641 } }, "sealed|noul": { "cc": 53.117408906882595, "n": { "easy": 39, "hard": 132, "judge": 72, "standard": 57 }, "tiers": { "easy": 48.71794871794872, "hard": 50.0, "judge": 55.55555555555556, "standard": 57.89473684210527 } }, "sealed|score": { "cc": 33.39767601516204, "n": { "easy": 33, "hard": 115, "judge": 75, "standard": 77 }, "tiers": { "easy": 38.95455244599806, "hard": 28.60554068871497, "judge": 38.85138751678833, "standard": 32.02294120019875 } } } }, "jevbench_score": 54.28713266716134, "key": "wald-4b-v2-1", "last_measured_on": "2026-10-10", "licence": "Apache-2.0 (weights and server code)", "listing": "ranked", "measured_in": "v1.6.5", "measurement_date_status": "measured", "model_pin": "org2ai/Wald-4B@ce3286f7efd332530e5dccd8a17d6496e029f714", "noul_decisive": { "acc_among_decisive": 0.7813333333333333, "decisive": 375, "decisive_rate": 1.0, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 1, "ranked": true, "ranks": { "A": 1, "B": 1, "C": 1, "capability": 2 }, "repo": "https://huggingface.co/org2ai/Wald-4B", "scores": { "A": 54.28713266716134, "B": 52.0059503801659, "C": 40.700232429112376 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.20302085168659686, "p50_s_raw": 0.026510425843298435, "p95_s_adjusted": 0.24250074867159127, "p95_s_raw": 0.046250374335795645 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.7586206896551724, "cc": 63.68096433104207, "cc_by_type": { "choice": 75.0, "noul": 66.66666666666667, "score": 49.376226326459516 }, "low_n": true, "n": 29, "n_by_type": { "choice": 16, "noul": 6, "score": 7 }, "score": 63.68096433104207 }, "calibration_gen": { "accuracy": 0.6160714285714286, "cc": 44.10064431077017, "cc_by_type": { "choice": 46.42857142857144, "noul": 61.53846153846154, "score": 24.334899965277536 }, "low_n": false, "n": 112, "n_by_type": { "choice": 56, "noul": 26, "score": 30 }, "score": 44.10064431077017 }, "composite_gen": { "accuracy": 0.6836734693877551, "cc": 57.621281961194256, "cc_by_type": { "choice": 74.07407407407408, "noul": 64.28571428571428, "score": 34.50405752379443 }, "low_n": false, "n": 98, "n_by_type": { "choice": 36, "noul": 28, "score": 34 }, "score": 57.621281961194256 }, "fairness": { "accuracy": 1.0, "cc": 89.90132796385994, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 69.70398389157982 }, "low_n": true, "n": 27, "n_by_type": { "choice": 12, "noul": 6, "score": 9 }, "score": 89.90132796385994 }, "jaggedness": { "accuracy": 0.5459459459459459, "cc": 28.653270418657403, "cc_by_type": { "choice": 47.19777674849467, "noul": 12.82051282051282, "score": 25.94152168696473 }, "low_n": false, "n": 185, "n_by_type": { "choice": 97, "noul": 39, "score": 49 }, "score": 28.653270418657403 }, "llm_router": { "accuracy": 0.7424242424242424, "cc": 68.06655796712668, "cc_by_type": { "choice": 56.33999071063632, "noul": 83.56164383561644, "score": 64.29803935512727 }, "low_n": false, "n": 264, "n_by_type": { "choice": 131, "noul": 73, "score": 60 }, "score": 68.06655796712668 }, "long_state": { "accuracy": 0.3793103448275862, "cc": 4.748724836384873, "cc_by_type": { "choice": 40.476190476190474, "noul": -27.27272727272727, "score": 1.0427113056914128 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 11, "score": 7 }, "score": 4.748724836384873 }, "ml_usecase": { "accuracy": 0.5161290322580645, "cc": 22.614498009598538, "cc_by_type": { "choice": 35.38461538461538, "noul": 50.0, "score": -17.541121355819755 }, "low_n": false, "n": 31, "n_by_type": { "choice": 15, "noul": 8, "score": 8 }, "score": 22.614498009598538 }, "multilingual": { "accuracy": 0.6412698412698413, "cc": 46.51177232484513, "cc_by_type": { "choice": 55.217196596506945, "noul": 58.62068965517242, "score": 25.69743072285603 }, "low_n": false, "n": 315, "n_by_type": { "choice": 146, "noul": 87, "score": 82 }, "score": 46.51177232484513 }, "negation_pair_gen": { "accuracy": 0.6964285714285714, "cc": 33.02311941019134, "cc_by_type": { "choice": 61.904761904761905, "noul": 50.0, "score": -12.835403674187873 }, "low_n": false, "n": 56, "n_by_type": { "choice": 28, "noul": 20, "score": 8 }, "score": 33.02311941019134 }, "option_order": { "accuracy": 0.8245614035087719, "cc": 77.80244173140953, "cc_by_type": { "choice": 77.80244173140953 }, "low_n": false, "n": 57, "n_by_type": { "choice": 57 }, "score": 77.80244173140953 }, "perturb_pair": { "accuracy": 0.8666666666666667, "cc": 84.22071661549569, "cc_by_type": { "choice": 76.4089121887287, "noul": 100.0, "score": 76.25323765775835 }, "low_n": false, "n": 60, "n_by_type": { "choice": 34, "noul": 14, "score": 12 }, "score": 84.22071661549569 }, "usecase": { "accuracy": 0.6033755274261603, "cc": 41.10751808936778, "cc_by_type": { "choice": 57.217190210059755, "noul": 43.85964912280702, "score": 22.24571493523657 }, "low_n": false, "n": 237, "n_by_type": { "choice": 111, "noul": 57, "score": 69 }, "score": 41.10751808936778 } }, "lang": { "ar": { "accuracy": 0.8181818181818182, "cc": 42.61103847074362, "cc_by_type": { "choice": 73.33333333333334, "noul": 50.0, "score": 4.49978207889753 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 4, "score": 2 }, "score": 42.61103847074362 }, "cs": { "accuracy": 0.625, "cc": 61.864406779661024, "cc_by_type": { "choice": 23.72881355932204, "noul": 100.0 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 3 }, "score": 61.864406779661024 }, "da": { "accuracy": 0.5416666666666666, "cc": 42.74539505383657, "cc_by_type": { "choice": 28.994082840236683, "noul": 60.00000000000001, "score": 39.24210232127302 }, "low_n": true, "n": 24, "n_by_type": { "choice": 11, "noul": 5, "score": 8 }, "score": 42.74539505383657 }, "de": { "accuracy": 0.7333333333333333, "cc": 55.337376093377294, "cc_by_type": { "choice": 78.27398913699457, "noul": 55.55555555555556, "score": 32.18258358758174 }, "low_n": false, "n": 45, "n_by_type": { "choice": 24, "noul": 9, "score": 12 }, "score": 55.337376093377294 }, "el": { "accuracy": 0.5555555555555556, "cc": 11.934924140935678, "cc_by_type": { "choice": 39.7590361445783, "noul": 60.00000000000001, "score": -63.954263721771284 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 5, "score": 2 }, "score": 11.934924140935678 }, "en": { "accuracy": 0.6657730116175157, "cc": 49.96242193822483, "cc_by_type": { "choice": 59.169938306225426, "noul": 53.38345864661653, "score": 37.33386886183252 }, "low_n": false, "n": 1119, "n_by_type": { "choice": 577, "noul": 266, "score": 276 }, "score": 49.96242193822483 }, "es": { "accuracy": 0.8571428571428571, "cc": 71.78869471940175, "cc_by_type": { "choice": 73.81818181818181, "noul": 100.0, "score": 41.54790234002343 }, "low_n": true, "n": 28, "n_by_type": { "choice": 10, "noul": 11, "score": 7 }, "score": 71.78869471940175 }, "fi": { "accuracy": 0.7777777777777778, "cc": 78.87323943661971, "cc_by_type": { "choice": 57.74647887323943, "noul": 100.0 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 3 }, "score": 78.87323943661971 }, "fr": { "accuracy": 0.7333333333333333, "cc": 53.603908086020034, "cc_by_type": { "choice": 74.61212976022568, "noul": 50.0, "score": 36.19959449783442 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 4, "score": 6 }, "score": 53.603908086020034 }, "hi": { "accuracy": 0.43333333333333335, "cc": 25.416831317917744, "cc_by_type": { "choice": 13.043478260869566, "noul": 60.00000000000001, "score": 3.2070156928836524 }, "low_n": false, "n": 30, "n_by_type": { "choice": 11, "noul": 10, "score": 9 }, "score": 25.416831317917744 }, "id": { "accuracy": 0.6666666666666666, "cc": 33.87096774193548, "cc_by_type": { "choice": 67.74193548387096, "noul": 0.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 33.87096774193548 }, "it": { "accuracy": 0.7647058823529411, "cc": 54.2758437211133, "cc_by_type": { "choice": 74.61212976022568, "noul": 50.0, "score": 38.21540140311422 }, "low_n": true, "n": 17, "n_by_type": { "choice": 5, "noul": 4, "score": 8 }, "score": 54.2758437211133 }, "ja": { "accuracy": 0.65625, "cc": 49.17418342165371, "cc_by_type": { "choice": 51.21951219512196, "noul": 71.42857142857142, "score": 24.874466641267755 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 49.17418342165371 }, "ko": { "accuracy": 0.5454545454545454, "cc": 55.08222770188039, "cc_by_type": { "choice": 67.21311475409836, "noul": 100.0, "score": -1.9664316484571787 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 2, "score": 5 }, "score": 55.08222770188039 }, "mixed": { "accuracy": 0.9090909090909091, "cc": 82.54649269370202, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 47.63947808110607 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 82.54649269370202 }, "nl": { "accuracy": 0.38461538461538464, "cc": 9.711332583883976, "cc_by_type": { "choice": -0.2785515320334245, "noul": 33.33333333333333, "score": -3.920784049647974 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 6, "score": 2 }, "score": 9.711332583883976 }, "no": { "accuracy": 0.5714285714285714, "cc": 50.81967213114754, "cc_by_type": { "choice": 1.6393442622950836, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 3 }, "score": 50.81967213114754 }, "pl": { "accuracy": 0.575, "cc": 34.47099050276897, "cc_by_type": { "choice": 76.38828159160472, "noul": 19.999999999999996, "score": 7.024689916702187 }, "low_n": false, "n": 40, "n_by_type": { "choice": 16, "noul": 10, "score": 14 }, "score": 34.47099050276897 }, "pt": { "accuracy": 0.7222222222222222, "cc": 74.79648342566703, "cc_by_type": { "choice": 63.81909547738693, "noul": 100.0, "score": 60.570354799614194 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 6, "score": 5 }, "score": 74.79648342566703 }, "sv": { "accuracy": 0.375, "cc": -8.847329747252749, "cc_by_type": { "choice": -33.333333333333336, "noul": 50.0, "score": -43.20865590842491 }, "low_n": true, "n": 8, "n_by_type": { "choice": 3, "noul": 4, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.7272727272727273, "cc": 39.204683185393755, "cc_by_type": { "choice": 73.6842105263158, "noul": 33.33333333333333, "score": 10.596505696532155 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 3, "score": 3 }, "score": 39.204683185393755 }, "uk": { "accuracy": 0.5454545454545454, "cc": -4.6691240891629535, "cc_by_type": { "choice": 56.52173913043478, "noul": -100.0, "score": 29.47088860207636 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 1, "score": 4 }, "score": 0.0 }, "zh": { "accuracy": 0.75, "cc": 52.01465673807612, "cc_by_type": { "choice": 67.56756756756756, "noul": 100.0, "score": -11.523597353339188 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 52.01465673807612 } }, "type": { "choice": { "accuracy": 0.6786666666666666, "cc": 58.36932112627397, "cc_by_type": { "choice": 58.36932112627397 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 58.36932112627397 }, "noul": { "accuracy": 0.7813333333333333, "cc": 56.266666666666666, "cc_by_type": { "noul": 56.266666666666666 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 56.266666666666666 }, "score": { "accuracy": 0.5066666666666667, "cc": 33.21347273527731, "cc_by_type": { "score": 33.21347273527731 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 33.21347273527731 } }, "usecase": { "advertising": { "accuracy": 0.6666666666666666, "cc": 47.43548710573032, "cc_by_type": { "choice": 60.46128500823723, "noul": 60.00000000000001, "score": 21.845176308953697 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 47.43548710573032 }, "code_linting": { "accuracy": 0.5833333333333334, "cc": 41.491312000045305, "cc_by_type": { "choice": 41.08019639934533, "noul": 60.00000000000001, "score": 23.393739600790585 }, "low_n": true, "n": 24, "n_by_type": { "choice": 13, "noul": 5, "score": 6 }, "score": 41.491312000045305 }, "customer_support": { "accuracy": 0.7, "cc": 48.534908440165786, "cc_by_type": { "choice": 51.61290322580645, "noul": 50.0, "score": 43.9918220946909 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 8, "score": 4 }, "score": 48.534908440165786 }, "demand_forecasting": { "accuracy": 0.38461538461538464, "cc": 7.558009820211829, "cc_by_type": { "choice": 39.7590361445783, "noul": 0.0, "score": -17.085006683942815 }, "low_n": true, "n": 26, "n_by_type": { "choice": 11, "noul": 4, "score": 11 }, "score": 7.558009820211829 }, "ecommerce_marketplaces": { "accuracy": 0.6111111111111112, "cc": 60.831849874775266, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 22.495549624325818 }, "low_n": true, "n": 18, "n_by_type": { "choice": 7, "noul": 5, "score": 6 }, "score": 60.831849874775266 }, "feature_extraction": { "accuracy": 0.5517241379310345, "cc": 33.80012553149724, "cc_by_type": { "choice": 48.71794871794871, "noul": 19.999999999999996, "score": 32.68242787654302 }, "low_n": true, "n": 29, "n_by_type": { "choice": 15, "noul": 5, "score": 9 }, "score": 33.80012553149724 }, "financial_crime": { "accuracy": 0.6153846153846154, "cc": 42.38729210014478, "cc_by_type": { "choice": 34.78260869565218, "noul": 100.0, "score": -7.620732395217833 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 42.38729210014478 }, "gaming": { "accuracy": 0.5172413793103449, "cc": 29.097480905970304, "cc_by_type": { "choice": 28.712871287128706, "noul": 42.85714285714286, "score": 15.722428573639357 }, "low_n": true, "n": 29, "n_by_type": { "choice": 11, "noul": 7, "score": 11 }, "score": 29.097480905970304 }, "insurance_claims": { "accuracy": 0.5714285714285714, "cc": 59.854855365826985, "cc_by_type": { "choice": 20.634920634920636, "noul": 60.00000000000001, "score": 98.92964546256033 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 5, "score": 1 }, "score": 59.854855365826985 }, "knowledge_graphs": { "accuracy": 0.4782608695652174, "cc": 34.55736075618144, "cc_by_type": { "choice": 26.717557251908392, "noul": 60.00000000000001, "score": 16.954525016635912 }, "low_n": true, "n": 23, "n_by_type": { "choice": 14, "noul": 5, "score": 4 }, "score": 34.55736075618144 }, "lead_generation": { "accuracy": 0.48, "cc": 32.637731364341626, "cc_by_type": { "choice": 83.73983739837398, "noul": 25.0, "score": -10.826643305349105 }, "low_n": true, "n": 25, "n_by_type": { "choice": 8, "noul": 8, "score": 9 }, "score": 32.637731364341626 }, "legal_compliance": { "accuracy": 0.3076923076923077, "cc": -22.12982828671353, "cc_by_type": { "choice": 34.42622950819673, "noul": -100.0, "score": -0.8157143683373169 }, "low_n": true, "n": 13, "n_by_type": { "choice": 8, "noul": 1, "score": 4 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.8275862068965517, "cc": 70.67750613389478, "cc_by_type": { "choice": 81.65137614678899, "noul": 75.0, "score": 55.381142254895344 }, "low_n": true, "n": 29, "n_by_type": { "choice": 14, "noul": 8, "score": 7 }, "score": 70.67750613389478 }, "model_routing": { "accuracy": 0.7866666666666666, "cc": 68.49684059479237, "cc_by_type": { "choice": 79.09407665505226, "noul": 86.36363636363636, "score": 40.03280876568852 }, "low_n": false, "n": 150, "n_by_type": { "choice": 74, "noul": 44, "score": 32 }, "score": 68.49684059479237 }, "moderation_trust_safety": { "accuracy": 0.5625, "cc": 32.325588967854976, "cc_by_type": { "choice": 52.63157894736842, "noul": 60.00000000000001, "score": -15.654812043803478 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 5, "score": 3 }, "score": 32.325588967854976 }, "recruiting": { "accuracy": 0.6666666666666666, "cc": 38.753237007491286, "cc_by_type": { "choice": 69.84924623115576, "noul": 33.33333333333333, "score": 13.077131457984759 }, "low_n": true, "n": 21, "n_by_type": { "choice": 9, "noul": 6, "score": 6 }, "score": 38.753237007491286 }, "risk_assessment": { "accuracy": 0.46153846153846156, "cc": 18.588938121855957, "cc_by_type": { "choice": 24.528301886792452, "noul": 33.33333333333333, "score": -2.0948208545578995 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 3, "score": 3 }, "score": 18.588938121855957 }, "scientific_discovery": { "accuracy": 0.375, "cc": 6.8446821135291955, "cc_by_type": { "choice": 28.286852589641434, "noul": -14.28571428571429, "score": 6.532908036660445 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 7, "score": 9 }, "score": 6.8446821135291955 }, "search_retrieval": { "accuracy": 0.8636363636363636, "cc": 76.70730935415533, "cc_by_type": { "choice": 85.68019093078759, "noul": 100.0, "score": 44.44173713167841 }, "low_n": true, "n": 22, "n_by_type": { "choice": 9, "noul": 6, "score": 7 }, "score": 76.70730935415533 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 81.02077877084074, 87.93508325091142 ], "I": [ 46.93348131417222, 56.20932983529756 ], "capability": [ 64.67436255217773, 71.31817290976798 ], "composite": { "A": [ 46.18008336139371, 55.5000390400574 ], "B": [ 43.548061690071826, 53.8197076967788 ], "C": [ 32.06950233430119, 48.51580772757769 ] }, "seed": 24 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 10.073244581339713, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 86.58998638479137, "capability": 69.27093170132007, "intelligence": 51.951877017848766 }, "per_type": { "choice": { "accuracy": 0.6786666666666666, "calibration": 89.06830352273374, "cc": 58.36932112627397, "cc_by_type": { "choice": 58.36932112627397 }, "cc_tiered": 63.03796283179515, "cc_tiered_open": 70.9521688095628, "cc_tiered_sealed": 55.12375685402749, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 58.36932112627397 }, "noul": { "accuracy": 0.7813333333333333, "calibration": 80.38847438136696, "cc": 56.266666666666666, "cc_by_type": { "noul": 56.266666666666666 }, "cc_tiered": 59.64083005730603, "cc_tiered_open": 66.16425120772948, "cc_tiered_sealed": 53.117408906882595, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 56.266666666666666 }, "score": { "accuracy": 0.5066666666666667, "calibration": 90.31318125027343, "cc": 33.21347273527731, "cc_by_type": { "score": 33.21347273527731 }, "cc_tiered": 33.176838164445115, "cc_tiered_open": 32.9560003137282, "cc_tiered_sealed": 33.39767601516204, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 33.21347273527731 } }, "raw_on_A": { "calibration": 86.69118099549732, "capability": 69.21287139307924, "intelligence": 51.734561790661175 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 49.82850819150803, "Emphasis on Accuracy 60:20:20": 47.56124181789482, "Emphasis on Cost 20:20:60": 45.197608867054875, "Emphasis on Speed 20:60:20": 58.630700497081925, "Intelligence only": 44.522489120413574 } } ], "tier_weights": { "easy": 0.1, "hard": 0.4, "judge": 0.3, "standard": 0.2 }, "types": { "choice": 0.3333333333333333, "noul": 0.3333333333333333, "score": 0.3333333333333333 }, "v16": { "a2_supplement": null, "baseline_sha256": "279efaa2fef1e8cbae2413adc7a452bf80e04cdc9cd041999a9bbca5bb298074", "capability_formula": "mean(Intelligence, Calibration)", "cost_basis": null, "counts": { "A": 300, "P": 300, "S": 1200, "api_input": 600, "api_smoke_public": 5, "core": 900, "selfhosted_input": 1500 }, "draw_digests": { "A": "a0fe704bdba955ab6f5b847f5c87d4fac801945c17454a73028abdb1682f8de9", "P": "7fa9e2592fcfaf24a959bf56eb247bda8adf60e7b2a1bf7c0a46156189820f68", "S": "66c9d7e43e8db2ca346eee887aee46203dc307fa55d1e7fa28355d7162e97600", "core": "545ebfea24bde76dc92e4068bb4c01b41074375fb0f3b1bf88c9ca9dd95877c7" }, "draw_release": "v1.6-regular-20261010-a2", "equating": { "equating_flag": null, "min_pool": 5, "offsets": { "C": 1.2215243847226134, "I": 0.24475051447114282 }, "pool": [ "clef-omni", "gutsy-0.8b-v0.3", "liquid-d1-3b", "liquid-d1-omni-600m", "rsi-jev-v6.1-vl-27b", "standard_one_8b_sh", "vegaml-0.8b", "vegaml-4b", "wald-4b-v2-1" ], "pool_n": 9, "rule": "median over complete full-coverage self-hosted systems of metric(S u P) - metric(A u P), per metric (Intelligence, Calibration); API value = clip(raw_A + offset)" }, "equating_A2": null, "freeze_manifest_sha256": "e100afd2cc1d80819761a344592578cafd0ca8b6a79e9d9c706a562f70b29feb", "g_med_rule": "median gap over ranked self-hosted systems; API systems use the same systems' P-vs-A gap", "offset_bootstrap_B": 0, "ranked_selfhosted": [ "clef-omni", "gutsy-0.8b-v0.3", "liquid-d1-3b", "liquid-d1-omni-600m", "rsi-jev-v6.1-vl-27b", "standard_one_8b_sh", "vegaml-0.8b", "vegaml-4b", "wald-4b-v2-1" ], "release": "v1.6-regular-20261010-a2", "scorer_sources_sha256": { "noul_method_v16.py": "119e59486ab3300b972fbcbadb0fe1dceceb124287607375aa61c66063c607d0", "score_v15.py": "0931e05c6410b1b971201834397cb609cdb25f942723c50375632952f6dcc447", "score_v15_headlineA.py": "826f2cbcb57007ec8a161e6eb0111182b8a656c0638027c6c78022a858fd8e2c", "score_v16.py": "02132271117653e6302c509232a581d0094f0e78eac8b1bfc67596fd5d5097e9" }, "scoring_parameters": { "B": 1000, "g_med_fixed": 10.073244581339713, "noul_method": "O1S", "only": null, "seed": 16, "test_allow_partial": false, "workers": 4 }, "seed_commitment": "2926f8a975d7f3f51068ed770b8ed0710aa56121bdb5a4e91119de7dff14fff2", "set_manifest_sha256": { "A": "442364621a9755bd3ea9831c50c720a7ff8d05b1a79dcc0cb69da0cb46bb7f50", "P": "6b2321f8359d87769a95c8e52135542d95ccdc353197e1e5c96abb9b616ce5ba", "S": "8814e4943c5440e7c932111b05b2205acc4019ef45dd0671223bee4c63dd3f01" } }, "views": [ "Balanced 33:33:33", "Emphasis on Accuracy 60:20:20", "Emphasis on Cost 20:20:60", "Emphasis on Speed 20:60:20", "Intelligence only" ] }