{ "G_med": 9.839526770355711, "G_med_api_basis_P_vs_A": 8.5949815619838, "G_med_flag_gt10": false, "benchmark": "JevBench", "board": { "A": { "markers": [], "order": [ "kettei-fat", "liquid-d1-ultra", "kettei-chub", "torchcast-decision-4b-v1.1", "sage-1.4.0", "attest-31b-v1.0", "metask_jev_rain_12b_trained", "metask_jev_rain_31b_trained", "messier-one-v0.4", "metask_jev_rain_31b_trained_v3", "celeris-1-decision", "subq-snap-preview", "respan-span-02", "nimble-v4", "nimble-v4-max", "jebadiah-9b-v2-1", "decision2-lux-9b", "uniphore-4b-decider", "startlux-decision-v1.1-27b-instant", "standard_one_3b_sh", "amos-1-medium", "arsenelupin-v1.1", "pjev-2b", "noma", "startlux-decision-v1.1-27b-autothink", "imajev-9b", "krino-2b", "imajev-2b", "nev-2b", "vllm-sr-d3", "amos-1-large", "purplemist-mini-1.0", "phocinae-largha-150m-v1", "julia-1" ] }, "B": { "markers": [], "order": [ "kettei-fat", "liquid-d1-ultra", "kettei-chub", "sage-1.4.0", "metask_jev_rain_12b_trained", "attest-31b-v1.0", "metask_jev_rain_31b_trained", "torchcast-decision-4b-v1.1", "metask_jev_rain_31b_trained_v3", "messier-one-v0.4", "celeris-1-decision", "subq-snap-preview", "respan-span-02", "nimble-v4-max", "nimble-v4", "jebadiah-9b-v2-1", "decision2-lux-9b", "uniphore-4b-decider", "startlux-decision-v1.1-27b-instant", "standard_one_3b_sh", "amos-1-medium", "arsenelupin-v1.1", "pjev-2b", "startlux-decision-v1.1-27b-autothink", "noma", "imajev-9b", "krino-2b", "imajev-2b", "vllm-sr-d3", "amos-1-large", "nev-2b", "purplemist-mini-1.0", "phocinae-largha-150m-v1", "julia-1" ] }, "C": { "markers": [], "order": [ "kettei-fat", "liquid-d1-ultra", "kettei-chub", "sage-1.4.0", "attest-31b-v1.0", "metask_jev_rain_12b_trained", "metask_jev_rain_31b_trained", "metask_jev_rain_31b_trained_v3", "celeris-1-decision", "subq-snap-preview", "torchcast-decision-4b-v1.1", "respan-span-02", "messier-one-v0.4", "nimble-v4-max", "nimble-v4", "startlux-decision-v1.1-27b-instant", "jebadiah-9b-v2-1", "decision2-lux-9b", "uniphore-4b-decider", "standard_one_3b_sh", "amos-1-medium", "arsenelupin-v1.1", "startlux-decision-v1.1-27b-autothink", "pjev-2b", "noma", "imajev-9b", "krino-2b", "vllm-sr-d3", "amos-1-large", "imajev-2b", "nev-2b", "purplemist-mini-1.0", "phocinae-largha-150m-v1", "julia-1" ] }, "capability": { "order": [ "kettei-fat", "vllm-sr-d3", "startlux-decision-v1.1-27b-instant", "subq-snap-preview", "liquid-d1-ultra", "startlux-decision-v1.1-27b-autothink", "nimble-v4-max", "metask_jev_rain_31b_trained_v3", "amos-1-large", "metask_jev_rain_31b_trained", "metask_jev_rain_12b_trained", "sage-1.4.0", "celeris-1-decision", "kettei-chub", "attest-31b-v1.0", "respan-span-02", "torchcast-decision-4b-v1.1", "nimble-v4", "jebadiah-9b-v2-1", "amos-1-medium", "messier-one-v0.4", "imajev-9b", "decision2-lux-9b", "pjev-2b", "uniphore-4b-decider", "standard_one_3b_sh", "arsenelupin-v1.1", "noma", "imajev-2b", "nev-2b", "purplemist-mini-1.0", "krino-2b", "phocinae-largha-150m-v1", "julia-1" ] } }, "bootstrap": { "B": 1000, "bootstrap_seed": 16, "g_med_fixed": true }, "headline": "A", "label": "JevBench v1.6.25 · four measured systems join the 34 full-draw rows of v1.6.24 on the same frozen draw: Jebadiah 9B v2.1 (Frontier Infra, self-hosted, ranked), the paid hosted Celeris-1 Decision (Celeris, ranked) and StartLux-Decision v1.1 27B Instant and AutoThink (StartLux Labs, self-hosted, ranked over the cost cap, weights non-commercial); every carried point estimate and the unranked A2 measurement are unchanged", "measurement_notes": [ "Refusals, errors and context-bound failures stay in the full 1,500 denominator. Every row kept its own package, model, runtime and context settings; nothing was re-run for presentation.", "NOT DIRECTLY COMPARABLE WITH THE LIVE v1.6.1 BOARD OR WITH THE EARLIER FRESH DRAWS. This release is a different draw with its own small-field median: G_med = 9.839526770355711, frozen from the four v1.6.10 members (attest-31B-v1.0 does not move it), against 2.573387642438244 on the live field. G_med_flag_gt10 is false. The gap penalty is max(0, 1 - max(0, (gap - G_med) - 8)/100), so this field's threshold is G_med + 8 = 17.84. On the merged live board this draw is placed with offset 0 while its anchor runs are pending, exactly like the earlier fresh draws. Measured, not estimated: Applying the same penalty formula to the stored base and gap at the live field's median 2.573387642438244, attest-31B-v1.0 Intelligence 65.238 would be 65.238 (gap -0.628); the original four rows are as stated in the v1.6.10 notes: standard_one_3b_sh Intelligence 32.786 would be 31.294 and metask_jev_rain_12b_trained 67.826 would be 67.587; krino-2b and metask_jev_rain_31b_trained are unchanged.", "Cost: no item is excluded from the cost basis on this draw. Every row is priced from its own measured usage block at a clearly-labelled ESTIMATE rate fixed before scoring (r99, r101, r103 and r106 pre-registrations); none is a bill any of these systems issued. Every row is under the published v1.6 cost ceiling of USD 0.0646 per 1,000 decisions. The cost normalizer C0 is USD 0.001 per 1,000 decisions with 30 cost points per decade.", "No subject-topic radar in this release: this draw's items have not been topic-labelled yet, and the shared topic labels of earlier draws do not cover its items. Families, use cases and languages are authoring metadata and are shown.", "v1.6.13 ADDS ONE ROW, attest-31B-v1.0, to the six of v1.6.12 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so all six earlier rows keep identical point scores, axes and capability; ranks and bootstrap confidence intervals are recomputed for the seven-row roster, and the existing raw_on_A and API-basis offset diagnostics are unchanged because the new row is full-set API, not a new self-hosted equating-pool member. No API offset enters its score.", "krino-2b: All 1,500 items answered HTTP 200 with native probabilities. The 15 longest items of the draw (the same 15 that standard_one_3b_sh refuses) ran with the state truncated to 65,536 tokens, as the author's submitted serve command documents (KEV_TRUNCATE_STATES=1); every other item is at most 1,802 input tokens. Reference kernels were used (KEV_FUSED=0) because the documented pip install has no flash-linear-attention. Kev reports usage.output_tokens as the token count of the serialized answer JSON although nothing is generated (165,276 over 1,500 decisions); it is priced as reported at the frozen DeepInfra Qwen/Qwen3.5-2B reference, USD 0.02/0.10 per 1M. Measured on an evaluator-owned Lium H100 pod in an offline container (network none).", "metask_jev_rain_12b_trained: All 1,500 items answered HTTP 200. The author's service runs the authors' own int4 (W4A16) quantization of Google's Gemma 4 12B QAT checkpoint with their own rank-16 LoRA applied unmerged by vLLM; per its model card it takes a second pass on part of the items, so it reports generated output tokens (84,238 over 1,500 decisions) and its latency has a long tail (raw p95 8.1 s against a raw p50 of 0.13 s). Measured on an evaluator-owned Lium H100 80GB pod (IP geolocated to Nizhny Novgorod, Russia) in an offline container (network none). google/gemma-4-12b-it is not listed on OpenRouter, so the cost uses the nearest listed sibling, google/gemma-3-12b-it at USD 0.05/0.15 per 1M, times its own usage. It is a different system from the earlier metask_jev_rain_12b row.", "metask_jev_rain_31b_trained: All 1,500 items answered HTTP 200. The author's service runs Google's unchanged W4A16 Gemma 4 31B QAT checkpoint with the authors' own rank-16 LoRA applied unmerged by vLLM, at max_model_len 6,656 (per the measuring operator's handoff): longer inputs are shortened by the service's own documented extraction rule rather than refused, so the longest items were answered on a shortened state. The authors tested it on H200; it was measured on an evaluator-owned Lium H100 80GB pod (IP geolocated to Nizhny Novgorod, Russia) in an offline container (network none) with unchanged settings. The service reports 1 output token per decision; cost is the OpenRouter google/gemma-4-31b-it list price (USD 0.09/0.34 per 1M) times its own usage.", "standard_one_3b_sh: 15 of 1,500 items returned HTTP 422: they are the 15 longest items of the draw and exceed the model's documented 32,768-token context, so the author's server refuses them rather than truncating; scored wrong per method. Measured on an evaluator-owned Lium H100 pod in an offline container (network none). Priced at the M2 base-model floor, OpenRouter mistralai/ministral-3b-2512 at USD 0.10/0.10 per 1M, because the author's own list price (USD 0.038 per 1M input) is below that floor; the schema head generates no text, so output tokens are 0.", "torchcast-decision-4b-v1.1: evaluator-owned Lium GPU pod (H100), offline read-only container (--network none); author's torchcast_decision server unchanged at tag v1.1.0 with the bundled decision_config.json (bf16, one forward pass reading option letters, calibration temperature 0.9, yes/no commit floor 0.801) on the documented stack torch 2.9.1+cu128 / transformers 5.8.1 / flash-linear-attention 0.5.2 / causal-conv1d 1.7.0 (fast kernels active, CUDA graphs on) Paid fast-lane order 529be5d5 (public). Qwen3.5-4B fine-tune served by the author's own torchcast_decision package. Runtime: official pytorch/pytorch:2.9.1-cuda12.8-cudnn9-devel image (digest-pinned) plus an offline hash-locked overlay of the documented packages (Python 3.11 instead of the documented 3.12). decision_config.json sets a disclosed yes/no commit floor of 0.801: a yes/no probability with max(P, 1-P) below 0.801 is moved to 0.801 on its own side (answer unchanged; same mechanism as h2o-lightning-4b). Prompts over 16,384 tokens keep the head and the tail of the state (documented). Output tokens are 0 (no generation). CC BY-NC 4.0 (weights); base Qwen/Qwen3.5-4B Apache-2.0; serving package Apache-2.0. ESTIMATE, v1.5-M2 base-model reference rule: self-hosted open weights, no bookable tariff. Base model Qwen/Qwen3.5-4B (model card base_model). OpenRouter does not list Qwen3.5-4B (host snapshot source/public-docs/OPENROUTER-MODELS.json, fetched 2026-10-12T08:15:26Z, read 2026-10-12). Binding precedent INTERPRETATION-1 I-2/I-3: DeepInfra Qwen/Qwen3.5-4B, USD 0.03/M input, USD 0.15/M output, frozen 25 Sep 2026 price cut-off snapshot (listing later deprecated; frozen reference kept). Logit readout, no generation: server reports 0 output tokens. Isolation is container-level (--network none); no separate host-level netcut probe was taken. The input tar was on the rented pod host disk during the run, and the pod was destroyed after measurement (CUSTODY-RECEIPT.json).", "uniphore-4b-decider: 15 of 1,500 inputs exceed the author's documented 16,384-token wrapper limit; the author's code refuses them (HTTP 422, no truncation) and they are scored as wrong per method, so 1,485 items were answered. The native readout is one forward pass, so output tokens are 0 and the cost is the input tokens at the frozen DeepInfra Qwen/Qwen3.5-4B reference, USD 0.03/0.15 per 1M (that listing is flagged deprecated in the frozen snapshot but remains the reference). Measured on an evaluator-owned Lium H100 pod in an offline container (network none), in the vllm/vllm-openai v0.31.0 image with transformers 5.17 and reference PyTorch kernels in place of the documented flash-linear-attention / causal-conv1d kernels, so its latency was measured on the fallback path. The author discloses that the public JevBench set was inspected repeatedly during development.", "attest-31b-v1.0: Paid public fast-lane order 07d13315. Vendor-hosted attest.engineering /v1/systemone API; latency measured from Sandy (Germany), using the official per-item latency subset without API adjustment. Full S1200+P300 draw, not API A300 supplement; no API equating. 15 of 1,500 requests returned HTTP 500 after 8 retry invocations and are scored as wrong in the full denominator. Public model documentation declares frozen google/gemma-4-31B-it @842da3794eaa0b77d5f08bae87a17459d91ff475, unmodified bf16, no adapter or quantization; single forward pass reading option-label logits, no generated tokens, one fitted temperature (value not documented). Hosting implementation and deployed weights were not independently inspected. Hosted API access is not an open-weights release by the vendor; base weights are under the Gemma 4 licence. ESTIMATE, v1.5-M2 base-model reference rule: vendor-hosted API, customer list price USD 0.048/M input and USD 0.00/M output effective 2026-10-11 has been in force fewer than 30 days. Base model google/gemma-4-31b-it reference on OpenRouter USD 0.09/M input / USD 0.34/M output, snapshot fetched 2026-10-12T17:55:05Z (SHA-256 81eae2f3f94bf299c8d4499260f842c357b7a869ce82dfa1741ad6a89cf57014), same base-model estimate precedent as metask_jev_rain_31b_trained. Applied to measured response usage; failed HTTP 500 usage/billing may be unreported. No output generation; reported output tokens 0. Estimated, not a vendor bill.", "Microsoft A2 re-run: 300 sealed supplementary A2 + 300 public items, not the full S1200 set. Unranked measurement; the official Microsoft row is retained on the merged boards. Equating pool four of six: Krino, Standard One, Uniphore, MeTask 12B. MeTask 31B and Torchcast were not completed after four unsuccessful allocation attempts (price/creation/controller failures); no inferred scores for missing pool systems. Seven HTTP 422 responses are wrong in the 600-item denominator. Pool re-runs used H200 in Helsinki, versus H100 parents; wide equating confidence intervals are reported. Latency from Sandy (Germany). Cost is the Azure Foundry tariff USD 0.042/M input, USD 0/M output; invoice/warmup/refusal charges unknown. No category cells or ranks are assigned to the A2 measurement.", "v1.6.15 ADDS THREE SELF-HOSTED CPU ROWS to the seven official rows of v1.6.14 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so all seven earlier rows keep identical point scores, axes, capability and cost objects. Ranks and bootstrap confidence intervals are recomputed over the new roster - the scorer draws ONE paired resample for the whole field, so adding rows moves a carried row's confidence interval while no point estimate changes; six of the eight carried rows' intervals moved (attest-31b-v1.0 and the carried A2 measurement did not). One DIAGNOSTIC also moved, and it is named here rather than left to be found: the API-basis median diagnostic is computed over the ranked field, so it moved from 9.628699345543296 to 9.158662835848954 when three rows joined, and that moves the stored A-basis diagnostic (the row field v16.raw_on_A) of exactly one carried row, standard_one_3b_sh: intelligence 29.3339 -> 29.1921, capability 53.1539 -> 53.0830. That diagnostic is not part of any score, rank or axis on this release, and no carried row's composite, axis, capability or cost object changes. The unranked A2 measurement of v1.6.14 is carried through byte for byte and is not rescored. reflex-instinct-0.6b is LISTED BUT NOT RANKED: CoS deputy decision #18761 accepts its Intelligence, Calibration and Cost only and declares its Speed figure non-official, and the composite weights Speed. All three new rows were measured at USD 0 on the evaluator's own shared CPU host, so their Speed figures are loaded-host lower bounds; each row logs its own one-minute load trace and carries it in its note.", "Where to look for the row that is shown but not ranked, and what it does NOT mean. reflex-instinct-0.6b is deliberately absent from the ranking charts, the Composite and Capability boards, the compare view and the category radars, because those are ranked views and this row has no rank. It is named in this release's own notes and header, and it appears in the collapsed \"All data\" table further up the page, after the Noul table. DISCLOSED: in that table the row still shows its composite as the scorer computed it, and a Jev-class verdict the site derives from its cost per decision and its MEDIAN LATENCY against the frozen Jev-class caps. The composite weights the Speed axis, and that latency is the very measurement CoS deputy decision #18761 declares non-official for this row, so neither figure is an official placement. They are not an official placement and must not be quoted as one. Its accepted figures are Intelligence 3.960, Calibration 73.153, Capability 38.556 and Cost USD 0.00630127 per 1,000 decisions. A dedicated row group for listed-not-ranked rows, with Speed greyed out and the composite hidden, is the right presentation and is a site change this release does not make.", "Cost on the three new rows: each is priced from its own measured usage block at a clearly labelled ESTIMATE rate fixed before scoring, on the full 1,500-row basis, and each is far under the published v1.6 ceiling of USD 0.0646 per 1,000 decisions - phocinae-largha-150m-v1 USD 0.00143980, reflex-instinct-0.6b USD 0.00630127, julia-1 USD 0.00217445. None is a bill any of these systems issued. julia-1's declared base jhu-clsp/mmBERT-small has no market listing; adding it to the frozen pricing module's reviewed unlisted-base set was authorised by #18761, re-frozen, hashed, proved behaviour-identical over every registry entry on the evaluator's machine and independently reviewed before this row was scored.", "julia-1: 1,500 attempted, 1,483 answered. The 17 misses are the author's own strict_encoding policy refusing rather than truncating (15 items above their lossless context budget, 2 with an option above their 48-token option contract); the totals AND the per-type split were pre-registered before the run and came back exact. All answered items native; the Speed figures are the published method's percentiles over this draw's open standard and judge items. 652,334 input tokens and 0 output tokens by construction (one encoder forward pass read at marker positions, nothing generated). The author ships no server, so the HTTP surface is the evaluator's - but their own julia.typed.predict_typed already speaks JevBench's typed-question shape, so no mapping decision of the evaluator's enters this row. Their hierarchical Router, which their own card says is not a global probability distribution, is neither used nor reachable here: every choice item of this draw has 3 to 9 options, inside their native 2-20 contract, so each decision is one call with one global softmax, and the probabilities are their raw values rather than their presentation helper's. Measured on the evaluator's own shared CPU host at their own settings read back from the run itself (float32, 4 threads, torch backend); one-minute load 10.87-13.78 logged, so Speed is a loaded-shared-CPU lower bound with the standard self-hosted x2 + 0.15 s adjustment on top. Cost is a clearly labelled size-class ESTIMATE: the declared base jhu-clsp/mmBERT-small has no market listing, and the base-size-encoder line of USD 0.005 per 1M input / 0 output that modernbert-jev-v2, tacet-sonata and watt-flash-0.1 carry in the PUBLISHED v1.6 field is used unchanged - no row of this draw carries it - and it is not a bill.", "phocinae-largha-150m-v1: 1,500 attempted, 1,125 answered. The 375 score items are recorded as HTTP 422 misses by the EVALUATOR'S shim, under a rule pre-committed in writing before the run (PREREG-R92 section 4.6), because the author's API answers only a fixed nine-level 2..10 scale that does not fit this item type; the shim refuses them in about 0.7 ms without calling the author's server at all, so no claim is made that the author's code refused anything. They stay in the full 1,500 denominator as misses - the figure 1,125 was pre-registered and came back exact. All answered items native. The Speed figures on this row are the published method's percentiles over this draw's open standard and judge items, not a median over every attempt - and DISCLOSED: 37 of those 124 items are the shim's instant score refusals, which cost no model time, so the figures are faster than the model itself is. Over the 87 items the model actually answered the Speed axis would read 75.75 instead of 77.12, which changes no rank. Measured on the evaluator's own shared CPU host (AMD Zen 2, float32), NOT a dedicated machine: other jobs ran beside it and the one-minute load average was logged for the whole window (3.75 to 9.89, mean 5.81, 12 samples - the quietest of this release's three new rows). Speed is therefore a loaded-shared-CPU figure and a lower bound for this model on a dedicated core or a GPU; the standard self-hosted x2 + 0.15 s adjustment is applied on top and the raw percentiles are kept. DISCLOSED: the author's server places every short-option request in its own 512-token budget tier, so the state is truncated far below the 8,192 its MAX_POS advertises - the author's design, not the evaluator's, and visible in their own usage histogram. A SECOND mapping decision of the evaluator's, footnoted on the row: the item's instruction text (and for noul its true/false criteria) is appended to the state, because state is the only text field the author's API has. Cost is a clearly labelled size-class ESTIMATE on the base-size-encoder line the PUBLISHED v1.6 field carries - modernbert-jev-v2 (measured in v1.6.8 on the earlier a2 draw) and the v1.6.1 rows tacet-sonata and watt-flash-0.1 - and no row of this draw carries it. It is not a bill.", "reflex-instinct-0.6b: 1,500 attempted, 1,500 answered, zero failures of any class - the pre-registered figure, exact. All answered items native; the Speed figures are the published method's percentiles over this draw's open standard and judge items. NOT RANKED, and here is exactly why: CoS deputy decision #18761 accepts this row's Intelligence, Calibration and Cost only and declares its Speed figure (58.972) NON-OFFICIAL, because the one-minute load on the shared host ranged 9.42-33.12 (mean 15.28, 155 samples) during the window and the evaluator's own preparation activity overlapped it. The published composite weights Speed, so this row is shown with its axes and its Capability 38.556 - which is Intelligence and Calibration only and contains no Speed - but it carries no rank and no composite placement. Its Capability 38.556 is the highest of this release's three new rows, above phocinae-largha-150m-v1's 30.240 - said here because the Capability board does not show it: that board contains no Speed at all, so a reader could reasonably ask why this row is absent from it. The answer is that THIS RELEASE applies #18761 to the whole row rather than axis by axis - the deputy's decision excludes the Speed figure, and whether a row with a non-official Speed may still be ranked on Capability alone is the open question #18825, which has no answer yet. The conservative reading ships. DISCLOSED: the author's own encode() truncates state lines above its own max_len of 16,384 instead of refusing them, so this row answers the 15 items of 106,804-122,936 characters that four other rows of this draw recorded as context refusals (attest-31b-v1.0, julia-1, standard_one_3b_sh and uniphore-4b-decider); that is the author's design choice, not the evaluator's, and those items are the slowest in the row. Cost is a clearly labelled size-class ESTIMATE on the sub-1B line the PUBLISHED v1.6 field carries (liquid-d1-omni-600m, jiwo-0.8b, vegaml-0.8b) - no row of this draw carries it - and it is not a bill.", "v1.6.16 ADDS ONE SELF-HOSTED GPU ROW, vllm-sr-d3 (vLLM-SR d3), to the ten full-draw rows of v1.6.15 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so every carried row keeps identical point scores, axes, capability, cost and speed objects. What moved on carried rows, counted exactly. RANKS: vllm-sr-d3 has the highest Capability of this draw, so all nine ranked carried rows move down one place in this release's Capability order (attest-31b-v1.0 3 -> 4, julia-1 9 -> 10, krino-2b 7 -> 8, metask_jev_rain_12b_trained 2 -> 3, metask_jev_rain_31b_trained 1 -> 2, phocinae-largha-150m-v1 8 -> 9, standard_one_3b_sh 6 -> 7, torchcast-decision-4b-v1.1 4 -> 5, uniphore-4b-decider 5 -> 6); its composite (A 5.570) sits between krino-2b and phocinae-largha-150m-v1, so phocinae-largha-150m-v1 (8 -> 9) and julia-1 (9 -> 10) also move down one place on composites A, B and C and in their headline rank; no other composite rank moves. CONFIDENCE INTERVALS: none of the ten carried rows' bootstrap confidence intervals moved. The scorer seeds each row's bootstrap with the shared seed 16 plus the row's position in the key-sorted roster, and vllm-sr-d3 sorts last, so no carried row's position changed (in v1.6.15 the three new keys sorted into the middle of the roster, which is why six carried intervals moved then). Correction to the v1.6.15 note above: it says the scorer draws ONE paired resample for the whole field; it does not - the scorer seeds each row's resample by its position (seed 16 + index in the key-sorted roster), each row with its own resample, which is why rows that changed position moved in v1.6.15 and none moved here. The same mechanism is why d3's interval here differs from its private single-row preview while its point values are identical: Capability 95% interval [78.928, 84.441] here, [79.337, 84.362] in the preview. DIAGNOSTICS: the API-basis median diagnostic is computed over the ranked field, so it moved from 9.158662835848954 to 9.40342052601045, which moves the stored A-basis diagnostic (the row field v16.raw_on_A) of exactly one carried row, standard_one_3b_sh: intelligence 29.1921 -> 29.2660, capability 53.0830 -> 53.1199; and the release-level equating-offset diagnostic moved from I 0.1329 / C 1.0866 to I 0.2288 / C 1.2340. Neither is part of any score, rank or axis on this release: the only API row scored in this invocation, attest-31b-v1.0, is a full-set row and is not equated, and the carried A2 measurement keeps its own A2 equating and is not rescored. reflex-instinct-0.6b stays LISTED BUT NOT RANKED under #18761 and the unranked A2 measurement of v1.6.14 is carried; neither is changed.", "Cost on vllm-sr-d3, and the scope of the cost note above: that note's \"every row is under the published v1.6 cost ceiling of USD 0.0646 per 1,000 decisions\" is true of the ten rows of v1.6.15 and is NOT true of vllm-sr-d3. d3 is priced as an ESTIMATE at the frozen Qwen/Qwen3.8-27B list reference of the published v1.6 field (USD 0.42 per 1M input and USD 3.00 per 1M output, the same pair the published Perplexity Decider v1.1 27B and Torchcast-Decision-27B rows carry) times its own measured tokens on the full 1,500-row basis: 1,951,205 input tokens and 0 output tokens, USD 0.5463374 per 1,000 decisions, about 8.46x that ceiling. It is not a bill and not a quote from the author or any host. Consequences: its Cost axis is 17.876, below 50, so the squared cost gate of the composite applies and its composite A is only 5.570 (rank 8 of 10 here) despite the highest Capability of this draw; and on the merged live board it is far above the official Jev-class cost cap, so it is NOT on the default (cost-capped) Capability ranking.", "Board effect, and why this release needed an explicit GO. On the merged live board, with the default caps, the top five of composites A, B and C and of the cost-capped Capability ranking are unchanged on the mixed board, on the open-weights board and on the API board (vllm-sr-d3 is outside the cost cap and its composite is far down). The views that change are Capability views with the cost cap switched off: with the latency cap at 2x and open-weights and API models mixed, vllm-sr-d3 becomes #1 on Capability, 82.966 against Perplexity Decider v1.1 27B at 82.792; on the open-weights board it is #1 with the cost cap off whether the latency cap is 2x or off; on the mixed board with both caps off it is #6 and that top five is unchanged. The #1 is not a resolved order: d3's 95% interval here is [78.93, 84.44] and Perplexity Decider's is [79.18, 84.71], they were measured on different draws (this draw sits on the merged board with offset 0 while its anchor runs are pending), and d3's Speed was timed on different hardware (see its note). A change to a public top five needs Florian's GO; he gave it on 13 Oct 2026 at 15:12Z (card 17392).", "vllm-sr-d3: Model vllm-sr/d3 at revision 5ce995dfb7ff07da00fec4211eb7f0560f52eaf8 (vLLM Semantic Router team), a fine-tune of the base Qwen/Qwen3.8-27B; its model card states 26.09B parameters including a 0.46B vision encoder. Weights Apache-2.0. The training data is not disclosed by the author, so the weight licence alone says nothing about the provenance of the training data. 1,500 attempted, 1,500 answered (all HTTP 200), 0 errors, 0 missing; the author's server sets no input-length cap, so it also answered the draw's 15 longest items that several rows of this draw refuse. Determinism check: 50 items re-asked, 50/50 bit-identical probabilities, 0 argmax flips. HARDWARE DIFFERS FROM THIS DRAW'S OTHER GPU ROWS AND FROM PERPLEXITY DECIDER AND TORCHCAST-27B: measured offline on one rented NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB GPU (Verda), while this draw's other GPU rows, Perplexity Decider and Torchcast-Decision-27B were timed on a single H100 80 GB, because no single H100 was available; the Speed axis (87.369; p50 0.1087 s raw, 0.3675 s adjusted, over the scorer's 124 open standard and judge items) therefore rests on different hardware, and the larger 96 GB memory may also matter for the longest inputs. The plain CUDA/PyTorch path of the author's unmodified d3_server.py was measured; the author's AMD ROCm fast path does not apply on CUDA. The own-server adjustment x2 + 0.15 s is an assumption, not a measurement. Cost is an ESTIMATE, not a bill: the frozen Qwen3.8-27B list reference (the same basis as Perplexity Decider and Torchcast-Decision-27B) times d3's own measured tokens, USD 0.5463374 per 1,000 decisions, far above the official Jev-class cost cap, so d3 is NOT on the default (capped) Capability ranking and its composite is small because of the squared cost gate. Source review of the pinned code before the run: PASS (no network on the text path, no benchmark-specific branches, no secret reads).", "v1.6.17 ADDS 7 ROWS (Amos 1 large (Scira) (amos-1-large), Amos 1 medium (Scira) (amos-1-medium), metask-jev-rain-31B-trained service v3 (metask_jev_rain_31b_trained_v3), pjev-2b, PurpleMIST-Mini-1.0 (Olaverse) (purplemist-mini-1.0), Sage 1.4 (Levanto Labs) (sage-1.4.0), SubQ Snap (preview, SubQ) (subq-snap-preview)) to the eleven full-draw rows of v1.6.16 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so every carried row keeps identical point scores, axes, capability, cost and speed objects, and the listed-not-ranked reflex-instinct-0.6b and the unranked A2 measurement keep their wording. What moved on carried rows, counted exactly. RANKS: rows ranked in this release - amos-1-large, amos-1-medium, metask_jev_rain_31b_trained_v3, pjev-2b, purplemist-mini-1.0 - sit above some carried rows, so ranks recompute over the new roster and each carried rank moves only by the number of newly ranked rows above it: attest-31b-v1.0 capability 4 -> 6, julia-1 A 10 -> 15; B 10 -> 15; C 10 -> 15; capability 10 -> 15, krino-2b A 7 -> 10; B 7 -> 10; C 7 -> 10; capability 8 -> 13, metask_jev_rain_12b_trained capability 3 -> 5, metask_jev_rain_31b_trained capability 2 -> 4, phocinae-largha-150m-v1 A 9 -> 14; B 9 -> 14; C 9 -> 14; capability 9 -> 14, standard_one_3b_sh A 6 -> 7; B 6 -> 7; C 6 -> 7; capability 7 -> 11, torchcast-decision-4b-v1.1 C 4 -> 5; capability 5 -> 7, uniphore-4b-decider A 5 -> 6; B 5 -> 6; C 5 -> 6; capability 6 -> 10, vllm-sr-d3 A 8 -> 11; B 8 -> 11; C 8 -> 11. Rows listed but not ranked take no place in any order. CONFIDENCE INTERVALS: unlike v1.6.16, new keys sort into the middle of the key-sorted roster, and the scorer seeds each row's bootstrap with the shared seed 16 plus the row's position in that roster (each row has its own resample; the v1.6.15 note above that says the scorer draws one paired resample is corrected by the v1.6.16 note), so a carried row whose position changed gets a different interval while no point estimate moves. Carried rows whose intervals moved: attest-31b-v1.0, julia-1, krino-2b, metask_jev_rain_12b_trained, metask_jev_rain_31b_trained, phocinae-largha-150m-v1, reflex-instinct-0.6b, standard_one_3b_sh, torchcast-decision-4b-v1.1, uniphore-4b-decider, vllm-sr-d3. DIAGNOSTICS: the API-basis median diagnostic is computed over the ranked field, so it moved from 9.40342052601045 to 8.245659392674778; that moves the stored A-basis diagnostic (the row field v16.raw_on_A) of exactly the carried rows named here: standard_one_3b_sh: intelligence 29.2660 -> 28.9168, capability 53.1199 -> 52.9453; and the equating pool gains metask_jev_rain_31b_trained_v3 and pjev-2b and purplemist-mini-1.0 (10 -> 13 members; the self-hosted ranked field changes with it) and the release-level equating-offset diagnostic moved from I 0.2288 / C 1.2340 to I 0.1329 / C 1.5418. Neither is part of any score, rank or axis on this release: the hosted-API rows scored in this invocation are full-set rows and are not equated, and the carried A2 measurement keeps its own A2 equating and is not rescored. reflex-instinct-0.6b stays LISTED BUT NOT RANKED under #18761 and the unranked A2 measurement of v1.6.14 is carried; neither is changed.", "Cost on amos-1-large, and the scope of the cost notes above: the inherited note's \"every row is under the published v1.6 cost ceiling of USD 0.0646 per 1,000 decisions\" is NOT true of amos-1-large (cost ceiling exceeded, as it was not true of vllm-sr-d3 in v1.6.16). Amos 1 large (Scira) is priced as an ESTIMATE at its frozen base-model market reference times the provider-reported tokens on the full 1,500-row basis: 1,938,726 input tokens and 0 output tokens, USD 0.5428433 per 1,000 decisions, 8.40x that ceiling. It is not a bill: the endpoint is free and keyless. Its Cost axis is 17.960, below 50, so the squared cost gate of the composite applies (composite A 5.536, rank 12 of 15), and on the merged live board it is above the official Jev-class cost cap, so it is NOT on the default (cost-capped) Capability ranking.", "Cost on amos-1-medium, and the scope of the cost notes above: the inherited note's \"every row is under the published v1.6 cost ceiling of USD 0.0646 per 1,000 decisions\" is NOT true of amos-1-medium (cost ceiling exceeded, as it was not true of vllm-sr-d3 in v1.6.16). Amos 1 medium (Scira) is priced as an ESTIMATE at its frozen base-model market reference times the provider-reported tokens on the full 1,500-row basis: 1,938,726 input tokens and 0 output tokens, USD 0.1292484 per 1,000 decisions, 2.00x that ceiling. It is not a bill: the endpoint is free and keyless. Its Cost axis is 36.657, below 50, so the squared cost gate of the composite applies (composite A 16.594, rank 8 of 15), and on the merged live board it is above the official Jev-class cost cap, so it is NOT on the default (cost-capped) Capability ranking.", "Board effect, and the basis for publishing it. Compared on the merged live board (the v1.6.1 feed plus the published fresh-draw cohort rows) before and after this release, over the mixed, open-weights and API boards: the top five of composites A, B and C is unchanged on all three boards; unchanged top fives also: Capability with no cost and no latency cap. The top five changes in: the mixed board (open-weights and API together): Capability with the official cost and latency caps (the default ranking) - before quyet-1-0-large, jeff_1_0_large, decisio-gemma-4-31b-v080, metask-jev-rain-31B-trained, sage-1.3.0; after quyet-1-0-large, jeff_1_0_large, metask-jev-rain-31B-trained service v3, decisio-gemma-4-31b-v080, metask-jev-rain-31B-trained (metask-jev-rain-31B-trained service v3 enters at #3) | the mixed board (open-weights and API together): Capability with the official cost cap and no latency cap - before quyet-1-0-large, jeff_1_0_large, decisio-gemma-4-31b-v080, wity-1, metask-jev-rain-31B-trained; after quyet-1-0-large, jeff_1_0_large, metask-jev-rain-31B-trained service v3, decisio-gemma-4-31b-v080, wity-1 (metask-jev-rain-31B-trained service v3 enters at #3) | the open-weights board: Capability with the official cost and latency caps (the default ranking); the open-weights board: Capability with the official cost cap and no latency cap - before quyet-1-0-large, jeff_1_0_large, decisio-gemma-4-31b-v080, metask-jev-rain-31B-trained, metask-jev-rain-12B-trained; after quyet-1-0-large, jeff_1_0_large, metask-jev-rain-31B-trained service v3, decisio-gemma-4-31b-v080, metask-jev-rain-31B-trained (metask-jev-rain-31B-trained service v3 enters at #3) | the API board: Capability with the cost cap off and latency 2x (Florian's settings) - before spx-cd-pro, sage-1.3.0, jev-1.13.0, fastino-glide, liquid-d1; after Amos 1 large (Scira), spx-cd-pro, sage-1.3.0, jev-1.13.0, fastino-glide (Amos 1 large (Scira) enters at #1). Added rows that enter a top five: Amos 1 large (Scira), metask-jev-rain-31B-trained service v3 (metask-jev-rain-31B-trained service v3 - a paid fast-lane row, order 4d9fef10); Amos 1 large (Scira) enters on Florian's explicit GO of 13 Oct 2026 (Telegram card #17487, Publish), for this row and this view, distinct from card 17392 (vllm-sr-d3). The other added rows (Amos 1 medium (Scira), pjev-2b, PurpleMIST-Mini-1.0 (Olaverse), Sage 1.4 (Levanto Labs), SubQ Snap (preview, SubQ)) appear in no top five. Amos 1 large's entry is not a paid order: it is a hosted row ranked above the cost ceiling and published on Florian's explicit GO of 13 Oct 2026 (Telegram card #17487, Publish), for this row and this view, distinct from card 17392 (vllm-sr-d3). The cause of the paid row's entry is the paid fast-lane order 4d9fef10 (a re-submission of the weights of order 147c5e18, disclosed in its row note): a paid public fast-lane row publishes automatically under the paid exception even when it changes a top five, exactly as the two paid metask rows did in v1.6.10 (whose release authority records the same top-five change); the change is disclosed here and in the proof authority.", "amos-1-large: 1,500 of 1,500 attempted answered (HTTP 200: 1,500); 0 not answered. Hosted API: the row is the vendor-hosted api.amos.scira.ai endpoint, not self-hosted weights - open weights are declared at mukaddamzaid/amos-1-large (Apache-2.0, revision declared by the submitter, not verified), but this measurement is of the endpoint, whose deployed weights were not independently inspected, and the vendor reports its token usage. Free, keyless, one request per second for the documented 60 requests/minute; latency measured from Sandy (Germany) on the official per-item latency subset without API adjustment: p50 0.5090 s, p95 0.8543 s. All 1,200 sealed items were logged to provider scira before the first request. CUSTODY CONDITION: after this hosted exposure, no self-hosted measurement of Amos weights on this draw may be shown as a clean measurement. Cost is an ESTIMATE: the exact frozen base-model market reference times the provider-reported 1,938,726 input tokens (output 0), USD 0.542843 per 1,000 decisions, 8.40x the official cost cap, disclosed in the cost notes above; it is not a bill. Speed figures are the published method's percentiles over this draw's open standard and judge items: p50 raw 0.5090 s, p95 raw 0.8543 s.", "amos-1-medium: 1,500 of 1,500 attempted answered (HTTP 200: 1,500); 0 not answered. Hosted API: the row is the vendor-hosted api.amos.scira.ai endpoint, not self-hosted weights - open weights are declared at mukaddamzaid/amos-1-medium (Apache-2.0, revision declared by the submitter, not verified), but this measurement is of the endpoint, whose deployed weights were not independently inspected, and the vendor reports its token usage. Free, keyless, one request per second for the documented 60 requests/minute; latency measured from Sandy (Germany) on the official per-item latency subset without API adjustment: p50 0.4435 s, p95 0.5068 s. All 1,200 sealed items were logged to provider scira before the first request. CUSTODY CONDITION: after this hosted exposure, no self-hosted measurement of Amos weights on this draw may be shown as a clean measurement. Cost is an ESTIMATE: the exact frozen base-model market reference times the provider-reported 1,938,726 input tokens (output 0), USD 0.129248 per 1,000 decisions, 2.00x the official cost cap, disclosed in the cost notes above; it is not a bill. Speed figures are the published method's percentiles over this draw's open standard and judge items: p50 raw 0.4435 s, p95 raw 0.5068 s.", "metask_jev_rain_31b_trained_v3: 1,500 of 1,500 attempted answered (HTTP 200: 1,500); 0 not answered. PAID PUBLIC FAST-LANE ORDER 4d9fef10, published under the paid exception. RE-SUBMISSION, DISCLOSED: the same LoRA weights as the published metask_jev_rain_31b_trained row (order 147c5e18, v1.6.10) with a changed service/ layer (for example its yes/no constant); those changes may have been tuned on that row's published v1.6.10 aggregates (custodian #18945). Both rows are shown, each with its own numbers; the earlier row is carried unchanged, and no third same-weights row is admitted on this draw. HARDWARE DIFFERS FROM THE EARLIER ROW: this service v3 was measured on an evaluator-owned Lium RTX PRO 6000 96 GB pod (its reviewed recipe needs 96 GB), in an offline read-only container (--network none, verified by docker inspect before the input upload), while the earlier metask rows of this draw were timed on an H100 80GB, so the Speed axis (84.666; p50 raw 0.1808 s, adjusted 0.5116 s under the x2 + 0.15 s assumption) rests on different hardware; vllm-sr-d3 (v1.6.16) was also timed on an RTX PRO 6000 96 GB class GPU (Verda), so two rows of this draw share that class while the other self-hosted GPU rows are H100. The service reports generated output tokens (840,049 input / 17,397 output). Cost is the v1.5-M2 base-model reference for the exact base google/gemma-4-31B-it (OpenRouter list price USD 0.09/M input, 0.34/M output, snapshot 2026-10-13T12:10:25Z) times the service's own measured usage: USD 0.054346 per 1,000 decisions, under the cost ceiling, estimated and not charged. Weights file shas: model 1b9b1d62..., adapter 2b956899..., code tree 6a1234ee...", "pjev-2b: 1,485 of 1,500 attempted answered (HTTP 200: 1,485; HTTP 422: 15); 15 not answered. The 15 misses are HTTP 422 refusals by the author's own letter_adapter.py, which raises rather than truncating a prompt above its 16,000-token contract; they are EXACTLY the same 15 items (106,804-122,936 characters) that uniphore-4b-decider (HTTP 422), attest-31b-v1.0 (HTTP 500) and standard_one_3b_sh (HTTP 422) failed on in the published v1.6.16 runs, verified set-equal, and julia-1's 17 refusals are a strict superset. Correction of an earlier entry text: torchcast-decision-4b-v1.1 and both metask_jev_rain rows answered all 1,500 and are NOT among the rows that refused them. The totals (1,485 answered, 15 HTTP 422, 0 other failures) were pre-registered before the run and came back exact; all answered items native. SPLIT WINDOW, DISCLOSED: the row was measured across four sittings under different host load on the evaluator's shared CPU (09:07-13:16Z until a pressure guard stopped it at load 57; two short attempts stopped by swap another job had left on the host; and 18:36-19:41Z to the end; one-minute load 6.96-54.44, mean 15.09 over 323 samples), resumed by task id so no item was answered twice. Its Speed is therefore a loaded-shared-CPU figure with the same kind of interference as reflex-instinct-0.6b, and a declared lower bound against the author's own H100 reference of 20-62 ms; the standard self-hosted x2 + 0.15 s adjustment (an assumption) is applied on top and the raw p50 10.06 s / p95 30.19 s are kept. Weights run in float32 (a strict upcast of the stored bfloat16, via the pre-registered PJEV_DTYPE one-line change that defaults to the author's bfloat16); serve.py and decision_core.py are byte-identical to the author's. Cost is a size-class ESTIMATE on the frozen DeepInfra 2-4B line (USD 0.03/M input, 0.0/M output; the same treatment the board gives decision-2b on the same MiniCPM5-2B base), 813,676 input tokens, USD 0.016274 per 1,000 decisions, not a bill. Speed figures are the published method's percentiles over this draw's open standard and judge items: p50 raw 10.0633 s, p95 raw 30.1948 s.", "purplemist-mini-1.0: 1,485 of 1,500 attempted answered (HTTP 200: 1,485; HTTP 422: 15); 15 not answered. The refusals are honest HTTP 422s: with --no-truncate the author's code refuses rather than truncates a prompt over 65,536 tokens, and the draw's 15 longest items (76-80 k tokens) are scored as wrong; they are the same 15 items that uniphore-4b-decider, attest-31b-v1.0, standard_one_3b_sh and pjev-2b failed on. DISCLOSED: the pre-registration predicted 0 refusals from a character-based estimate; the offline token count and the live 422s both show 15. TOKENS ARE AN OFFLINE COUNT, not provider usage (CoS deputy #18974): the author's server reports no usage, so input tokens were counted with the author's own pinned tokenizer and unchanged encode() on the exact prompts (independent recount PASS): 700,492 input tokens over the 1,485 answered items, output 0 (one forward pass, nothing generated). Measured on the evaluator's own shared CPU (AMD Zen 2, float32), not a dedicated machine, with the one-minute load logged; Speed is a loaded-shared-CPU lower bound and the standard x2 + 0.15 s adjustment (an assumption) is applied on top. Cost is an ESTIMATE on the exact base-model reference, USD 0.009340 per 1,000 decisions, not a bill. Speed figures are the published method's percentiles over this draw's open standard and judge items: p50 raw 3.6021 s, p95 raw 8.9970 s.", "sage-1.4.0: 1,483 of 1,500 attempted answered (HTTP 200: 1,483; HTTP 400: 15; HTTP 504: 2); 17 not answered. Hosted API (Levanto Labs), measured from Sandy (Germany) without API adjustment; native probabilities on all three question types; all 1,200 sealed items were logged to provider levanto before the first request; 665,989 input tokens and 0 output tokens. Treatment of its cost axis: UNPRICED under the frozen pricing_v15 path (sha256 dae86740...): proprietary hosted model with no base-model reference, and levanto-sage-v1.4 is in neither AGE_QUALIFIED_API nor LAUNCH_LIST_PRICE_API. sage-1.3.0's published row carries Levanto's list tariff (USD 0.05/M input, USD 10/M output), set before these sets bound new rows; it is not carried over here. Its Capability 74.773 is higher than that of 10 of the 15 ranked rows of this draw (it would sit at #6 of 16 in the draw's Capability order); it is withheld from every ranking, not rated lower. Ranking it would put it into every public Capability ranking, which needs Florian's review. Speed figures are the published method's percentiles over this draw's open standard and judge items: p50 raw 1.7411 s, p95 raw 7.5714 s.", "subq-snap-preview: 1,500 of 1,500 attempted answered (HTTP 200: 1,500); 0 not answered. NOT RANKED, and here is exactly why: CoS deputy decision #18975 accepts the measurement but gives the row no cost axis until a frozen-pricing change for its preview tariff (USD 0.05/M input, output free) passes re-freeze, unchanged-row proof and independent review; its model is proprietary with no base-model reference and the days-old preview tariff is in neither frozen tariff set, so under the frozen pricing path the cost is unpriced. Without a cost axis there is no composite, so the row shows Intelligence 72.250, Calibration 90.697 and Capability 81.474 (and Speed 86.405) and carries no rank, no composite and no place in the Capability ranking (the same conservative reading applied to reflex-instinct-0.6b under #18761). Its Capability 81.474 is higher than that of 14 of the 15 ranked rows of this draw (it would sit at #2 of 16 in the draw's Capability order); it is withheld from every ranking, not rated lower. Ranking it would put it into every public Capability ranking, which needs Florian's review. Where to look: it appears in the \"All data\" table and its detail view only. Hosted API, measured from Sandy (Germany) without API adjustment; 1,943,705 input tokens and 0 output tokens (nothing generated). SubQ retains API inputs indefinitely by default; acceptable only because every sealed item of this draw was already retired, and all 1,200 sealed items were logged to the provider before the first request. Hosting implementation and deployed weights were not independently inspected. Speed figures are the published method's percentiles over this draw's open standard and judge items: p50 raw 0.4656 s, p95 raw 0.4915 s.", "v1.6.18 ADDS 3 ROWS (Bespoke Nimble-v4 (Bespoke Labs) (nimble-v4), Bespoke Nimble-v4-max (Bespoke Labs) (nimble-v4-max), Respan Span-02 (respan-span-02)) to the eighteen full-draw rows of v1.6.17 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so every carried row keeps identical point scores, axes, capability, cost and speed objects, and the listed-not-ranked reflex-instinct-0.6b and the unranked A2 measurement keep their wording. What moved on carried rows, counted exactly. RANKS: amos-1-large A 12 -> 13; B 12 -> 13; C 12 -> 13, amos-1-medium A 8 -> 9; B 8 -> 9; C 8 -> 9; capability 8 -> 9, julia-1 A 15 -> 16; B 15 -> 16; C 15 -> 16; capability 15 -> 16, krino-2b A 10 -> 11; B 10 -> 11; C 10 -> 11; capability 13 -> 14, phocinae-largha-150m-v1 A 14 -> 15; B 14 -> 15; C 14 -> 15; capability 14 -> 15, pjev-2b A 9 -> 10; B 9 -> 10; C 9 -> 10; capability 9 -> 10, purplemist-mini-1.0 A 13 -> 14; B 13 -> 14; C 13 -> 14; capability 12 -> 13, standard_one_3b_sh A 7 -> 8; B 7 -> 8; C 7 -> 8; capability 11 -> 12, torchcast-decision-4b-v1.1 capability 7 -> 8, uniphore-4b-decider A 6 -> 7; B 6 -> 7; C 6 -> 7; capability 10 -> 11, vllm-sr-d3 A 11 -> 12; B 11 -> 12; C 11 -> 12. Each moved rank shifts only by the newly ranked rows above it. Rows listed but not ranked take no place in any order. CONFIDENCE INTERVALS: as in v1.6.17, new keys sort into the middle of the key-sorted roster, and the scorer seeds each row's bootstrap with the shared seed 16 plus the row's position in that roster (each row has its own resample; the v1.6.15 note above that says the scorer draws one paired resample is corrected by the v1.6.16 note), so a carried row whose position changed gets a different interval while no point estimate moves. Carried rows whose intervals moved: phocinae-largha-150m-v1, pjev-2b, purplemist-mini-1.0, reflex-instinct-0.6b, sage-1.4.0, standard_one_3b_sh, subq-snap-preview, torchcast-decision-4b-v1.1, uniphore-4b-decider, vllm-sr-d3. DIAGNOSTICS: the API-basis median diagnostic is 7.703968975298743 (previous 8.245659392674778); carried v16.raw_on_A moves: standard_one_3b_sh: intelligence 28.9168 -> 28.7534, capability 52.9453 -> 52.8636. Equating offsets are I 0.1005 / C 1.4616 (previous I 0.1329 / C 1.5418); the equating pool gains respan-span-02 (13 -> 14 members; the self-hosted ranked field changes with it) and the equating pool has 14 members. These diagnostics are not part of any score, rank or axis on this release: the hosted-API rows scored in this invocation are full-set rows and are not equated, and the carried A2 measurement keeps its own A2 equating and is not rescored. reflex-instinct-0.6b stays LISTED BUT NOT RANKED under #18761 and the unranked A2 measurement of v1.6.14 is carried; neither is changed.", "Cost on respan-span-02, and the scope of the cost notes above: the inherited note's \"every row is under the published v1.6 cost ceiling of USD 0.0646 per 1,000 decisions\" is NOT true of respan-span-02 (cost ceiling exceeded, as it was not true of vllm-sr-d3 in v1.6.17). Respan Span-02 is priced as an ESTIMATE at its frozen base-model market reference times the provider-reported tokens on the full 1,500-row basis: 776,560 input tokens and 102328 output tokens, USD 0.0670592 per 1,000 decisions, 1.04x that ceiling. It is not a bill: the endpoint is self-hosted on an evaluator-owned rented pod. Its Cost axis is 45.206, below 50, so the squared cost gate of the composite applies (composite A 51.804, rank 6 of 16), and on the merged live board it is above the official Jev-class cost cap, so it is NOT on the default (cost-capped) Capability ranking.", "Board effect: compared on the merged live board before and after v1.6.18, over the mixed, open-weights and API boards, composites A/B/C and Capability with official caps, no caps, cost off + latency 2x (Florian settings), and official cost + no latency cap: No top five changes in any view. Neither Nimble-v4 nor Nimble-v4-max enters any ranked view; both are listed but not ranked under #19084. Respan Span-02 enters no top five and is outside all official-cost-cap Capability views.", "nimble-v4: 1,500 of 1,500 attempted answered (HTTP 200: 1,500); 0 not answered. Vendor-hosted api.bespokelabs.ai /v1/systemone API, model `nimble-v4`, through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) on the official per-item latency subset without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `bespoke` before the first request (CoS deputy #18998/#19084, custodian #18999; every sealed item of this draw was already retired). Native probabilities on all three question types. Hosting implementation and deployed weights were not independently inspected. Measured as our own add after Bespoke Labs' public announcement of 13 Oct 2026 (no paid order); key on Benchmark Heaven's own account with Bespoke's free credit. CoS deputy decision #19084 (one-time ruling, the unranked SubQ Snap precedent #18975/#19046): measured, but the frozen pricing path gives it no eligible price, so it carries no cost axis until a separate pricing change passes re-freeze, unchanged-row proof and independent review. Without a cost axis there is no composite, so the row shows Intelligence, Calibration, Capability and Speed and is not ranked. UNPRICED under the frozen pricing_v15 path (sha256 dae86740...): proprietary hosted model with no documented base model (Bespoke's Hugging Face repos for v4 are private), and `nimble-v4` is in neither AGE_QUALIFIED_API nor LAUNCH_LIST_PRICE_API. Bespoke's own list tariff (USD 0.04/M input, output free) is recorded here for information only and is not used. Measured Intelligence 41.405, Calibration 86.634, Capability 64.020 and Speed 88.399 only; NOT RANKED: no Cost, no Composite, no rank in any view, including the Capability order. Where to look: the collapsed \"All data\" table and the detail view. Provider-reported usage: 1,935,351 input tokens and 3,000 output tokens (usage, not an inspected implementation). Latency on the official subset, without API adjustment: p50 raw 0.2324 s, p95 raw 0.6222 s. Exposure was logged to provider bespoke before dispatch. No top five changes; both rows are excluded from every ranked view.", "nimble-v4-max: 1,500 of 1,500 attempted answered (HTTP 200: 1,500); 0 not answered. Vendor-hosted api.bespokelabs.ai /v1/systemone API, model `nimble-v4-max`, through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) on the official per-item latency subset without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `bespoke` before the first request (CoS deputy #18998/#19084, custodian #18999; every sealed item of this draw was already retired). Native probabilities on all three question types. Hosting implementation and deployed weights were not independently inspected. Measured as our own add after Bespoke Labs' public announcement of 13 Oct 2026 (no paid order); key on Benchmark Heaven's own account with Bespoke's free credit. CoS deputy decision #19084 (one-time ruling, the unranked SubQ Snap precedent #18975/#19046): measured, but the frozen pricing path gives it no eligible price, so it carries no cost axis until a separate pricing change passes re-freeze, unchanged-row proof and independent review. Without a cost axis there is no composite, so the row shows Intelligence, Calibration, Capability and Speed and is not ranked. UNPRICED under the frozen pricing_v15 path (sha256 dae86740...): proprietary hosted model with no documented base model (Bespoke's Hugging Face repos for v4 are private), and `nimble-v4-max` is in neither AGE_QUALIFIED_API nor LAUNCH_LIST_PRICE_API. Bespoke's own list tariff (USD 0.09/M input, output free) is recorded here for information only and is not used. Measured Intelligence 68.185, Calibration 91.250, Capability 79.717 and Speed 91.451 only; NOT RANKED: no Cost, no Composite, no rank in any view, including the Capability order. Where to look: the collapsed \"All data\" table and the detail view. Provider-reported usage: 1,935,351 input tokens and 3,000 output tokens (usage, not an inspected implementation). Latency on the official subset, without API adjustment: p50 raw 0.2545 s, p95 raw 0.2813 s. Exposure was logged to provider bespoke before dispatch. No top five changes; both rows are excluded from every ranked view.", "respan-span-02: Regular queue GitHub issue #215, no paid order. Custodian #19055 admitted one use of the frozen draw input; container isolation was proven at 05:45:26Z (pod clock); the sealed file reached the pod host at 05:45:48Z (evaluator clock); the measuring container was re-created and re-proven at 05:45:50Z (pod clock), with the first sealed item entering at 05:45:55Z (pod clock). The host kept its network for ssh; cross-clock skew cannot be excluded. CoS deputy #19087/#19088: one EU RTX 5090, hard cap USD 2.00; actual approximately USD 0.20. Speed measured on an RTX 5090 with a Blackwell-only engine, NOT the H100 most GPU rows used. evaluator-owned rented Lium GPU pod, 1x NVIDIA RTX 5090 32 GB (Romania), NOT the 1x H100 80 GB on which most of this draw's GPU rows were timed: the author's engine runs only on Blackwell (sm_120a), so this row's Speed axis rests on different hardware. The author's own Rust/CUDA dllm-server, built unchanged from their Dockerfile at the pinned revision (weights model.safetensors sha256 5105d39d..., tokenizer cc8d3a0c..., public MTP draft google/gemma-4-26B-A4B-it-assistant pre-staged), started by their own launch.sh with their documented Decision Index settings (container environment HOST=127.0.0.1, PORT=8000; weights mounted read-only at /model, the pre-staged draft at /data) in a container with --network none. Isolation timeline: the container's cut was proven by docker inspect and a failing outbound probe at 05:45:26Z (pod clock) before the sealed input file was copied to the pod host at 05:45:48Z (evaluator clock); the measuring container was then re-created with the same --network none setting and re-proven at 05:45:50Z before the first sealed item was sent at 05:45:55Z. The pod host itself kept its network for the evaluator's ssh session, as for every self-hosted row; no request left the container; one disclosed invocation change: launch.sh is stored without the executable bit in the repository (mode 100644), so the unchanged file was run through the interpreter its own shebang names (/bin/bash) instead of being exec'd directly. Driven by the unchanged run_v15 runner and TypeSafe adapter against /v1/systemone, sequentially, after a 4-item synthetic smoke gate. The standard self-hosted x2 + 0.15 s own-server adjustment (an assumption, not measured) applies on top of the raw p50/p95. 15 of 1,500 requests returned HTTP 400 because prompt + 256-token cap exceeds the server's 16,384-token context: exactly the 15 task ids pre-registered from an offline count with the author's own tokenizer (the same 15 that Attest-31B, Standard One 3B SH, Uniphore-4B-Decider and PurpleMIST-Mini failed on); they are scored as wrong in the full denominator. Measured 14 Oct 2026 on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720077ef7c41ace367524896507c7f44ce71306fb3e659e8b85069d995b); pod removed afterwards. Regular-queue request GitHub issue #215; CoS deputy #19087/#19088 (spend), custodian #19055 (draw use). ESTIMATE (base-model reference; self-hosted open weights, no public tariff for these weights): the frozen market reference google/gemma-4-26b-a4b-it USD 0.09/M input, USD 0.3/M output for the declared base Gemma 4 26B-A4B (catalog id google/gemma-4-26b-a4b-it, supplied by the evaluator because the frozen path matches base names case-sensitively; the submission's model card spells it google/gemma-4-26B-A4B-it) x this row's own measured tokens on the full 1,500-row basis (776,560 input and 102,328 output tokens as reported by the author's server; the 15 refused items count as zero). Estimated, not charged; not a quote from the author or any host. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.067059 per 1,000 decisions is 1.04x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 precedent) and falls outside the official-caps board views by construction. Pre-registration matched exactly: 1,485 of 1,500 answered; the 15 HTTP 400 refusals are exactly the 15 pre-registered over-length task ids; 776,560 input tokens. r111/PREREG-R111.md sha256 9aa388b0…; receipts r111/receipts/TOKCOUNT-SPAN02-R111.json and r112/receipts/ROW-RESPAN-SPAN-02-R112.json. Native probabilities on all three types. Official-subset latency: p50 raw 0.0270 s, p95 raw 0.6446 s; standard self-hosted x2 + 0.15 s adjustment (assumption, not measured). Ranked with over-cap disclosure and excluded from official-caps views by construction. No top five changes.", "SubQ Snap PRICE-ONLY ADDENDUM: its measurement remains v1.6.17 (last measured 2026-10-13); no model was called again. The vendor publishes USD 0.05 per million input tokens and no output charge; the measured 1,943,705 input tokens and 0 output tokens produce USD 0.0647901667 per 1,000 decisions, 1.003x the official cap. The public basis discloses the capacity-limited public preview and waitlist under deputy ruling #19174. The initial candidate preview GO was Telegram card #17513. Florian approved publication of v1.6.20 on 2026-10-15T16:53Z in a direct message. The chief of staff recorded that as the answer \"Publish (recommended)\" to Telegram card #17515 (DECISIONS.md 2026-10-15 16:53 UTC; board #19649) - the card the refreshed merged-board full ranking and radar had been sent on, with preview PNG sha256 8cb19a14e4304228b3fccbd1a4a12feabb53b5a9fb884434a9a95c6a7639d88e attached. The card's own button was not pressed. The candidate and preview agree on SubQ point estimates: A 57.1609, Capability 81.4737, cost axis 45.6547. It is ranked outside official cost-capped views. Carried rank movements from this repricing: amos-1-large: A 13 → 14, B 13 → 14, C 13 → 14, capability 3 → 4, amos-1-medium: A 9 → 10, B 9 → 10, C 9 → 10, capability 9 → 10, attest-31b-v1.0: capability 6 → 7, julia-1: A 16 → 17, B 16 → 17, C 16 → 17, capability 16 → 17, krino-2b: A 11 → 12, B 11 → 12, C 11 → 12, capability 14 → 15, metask_jev_rain_12b_trained: capability 5 → 6, metask_jev_rain_31b_trained: capability 4 → 5, metask_jev_rain_31b_trained_v3: capability 2 → 3, phocinae-largha-150m-v1: A 15 → 16, B 15 → 16, C 15 → 16, capability 15 → 16, pjev-2b: A 10 → 11, B 10 → 11, C 10 → 11, capability 10 → 11, purplemist-mini-1.0: A 14 → 15, B 14 → 15, C 14 → 15, capability 13 → 14, respan-span-02: A 6 → 7, B 6 → 7, C 6 → 7, capability 7 → 8, standard_one_3b_sh: A 8 → 9, B 8 → 9, C 8 → 9, capability 12 → 13, torchcast-decision-4b-v1.1: C 5 → 6, capability 8 → 9, uniphore-4b-decider: A 7 → 8, B 7 → 8, C 7 → 8, capability 11 → 12, vllm-sr-d3: A 12 → 13, B 12 → 13, C 12 → 13.", "Board effect: merged-live-board comparison before/after v1.6.20: 20 of 21 top-five lists are unchanged. On the API Capability view with the cost cap off and the official 2x latency setting, SubQ Snap enters at #1 of 23 (81.474), ahead of Amos 1 large (79.359). It remains excluded from every official cost-capped view because its estimated USD 0.064790 per 1,000 decisions is above the USD 0.064595 cap.", "v1.6.21 ADDS 1 ROW (d1-ultra (Liquid AI, liquid-d1-ultra)) to the 21 full-draw rows of v1.6.20 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so every carried row keeps identical point scores, axes, capability, cost and speed objects, and the listed-not-ranked rows (reflex-instinct-0.6b, nimble-v4, nimble-v4-max, sage-1.4.0) and the unranked A2 measurement keep their wording. What moved on carried rows, counted exactly. RANKS: amos-1-large A 14 -> 15; B 14 -> 15; C 14 -> 15; capability 4 -> 5, amos-1-medium A 10 -> 11; B 10 -> 11; C 10 -> 11; capability 10 -> 11, attest-31b-v1.0 A 2 -> 3; B 2 -> 3; C 1 -> 2; capability 7 -> 8, julia-1 A 17 -> 18; B 17 -> 18; C 17 -> 18; capability 17 -> 18, krino-2b A 12 -> 13; B 12 -> 13; C 12 -> 13; capability 15 -> 16, metask_jev_rain_12b_trained A 3 -> 4; B 1 -> 2; C 2 -> 3; capability 6 -> 7, metask_jev_rain_31b_trained A 4 -> 5; B 3 -> 4; C 3 -> 4; capability 5 -> 6, metask_jev_rain_31b_trained_v3 A 5 -> 6; B 5 -> 6; C 4 -> 5; capability 3 -> 4, phocinae-largha-150m-v1 A 16 -> 17; B 16 -> 17; C 16 -> 17; capability 16 -> 17, pjev-2b A 11 -> 12; B 11 -> 12; C 11 -> 12; capability 11 -> 12, purplemist-mini-1.0 A 15 -> 16; B 15 -> 16; C 15 -> 16; capability 14 -> 15, respan-span-02 A 7 -> 8; B 7 -> 8; C 7 -> 8; capability 8 -> 9, standard_one_3b_sh A 9 -> 10; B 9 -> 10; C 9 -> 10; capability 13 -> 14, subq-snap-preview A 6 -> 7; B 6 -> 7; C 5 -> 6, torchcast-decision-4b-v1.1 A 1 -> 2; B 4 -> 5; C 6 -> 7; capability 9 -> 10, uniphore-4b-decider A 8 -> 9; B 8 -> 9; C 8 -> 9; capability 12 -> 13, vllm-sr-d3 A 13 -> 14; B 13 -> 14; C 13 -> 14. Each moved rank shifts only by the newly ranked row above it. Rows listed but not ranked take no place in any order. CONFIDENCE INTERVALS: as in v1.6.17 and v1.6.18, the new key sorts into the middle of the key-sorted roster, and the scorer seeds each row's bootstrap with the shared seed 16 plus the row's position in that roster, so a carried row whose position changed gets a different interval while no point estimate moves. Carried rows whose intervals moved: metask_jev_rain_12b_trained, metask_jev_rain_31b_trained, metask_jev_rain_31b_trained_v3, nimble-v4, nimble-v4-max, phocinae-largha-150m-v1, pjev-2b, purplemist-mini-1.0, reflex-instinct-0.6b, respan-span-02, sage-1.4.0, standard_one_3b_sh, subq-snap-preview, torchcast-decision-4b-v1.1, uniphore-4b-decider, vllm-sr-d3. DIAGNOSTICS: the API-basis median diagnostic is 7.703968975298743 (previous 7.703968975298743); carried v16.raw_on_A moves: none. Equating offsets are I 0.1005 / C 1.4616 (previous I 0.1005 / C 1.4616); the equating pool keeps its 14 members (a full-set API row is not equated and does not join it). These diagnostics are not part of any score, rank or axis on this release. reflex-instinct-0.6b stays LISTED BUT NOT RANKED under #18761 and the unranked A2 measurement of v1.6.14 is carried; neither is changed.", "Board effect: merged-live-board top-five comparison before/after v1.6.21 over the mixed, open-weights and API boards, composites A/B/C and Capability with official caps, no caps, cost off + latency 2x (Florian settings) and official cost cap + no latency cap: 11 of 21 top-five lists change, all of them because d1-ultra enters. mixed board (open weights + API), composite A: d1-ultra (Liquid AI) enters at #1, decisio v0.8.0 on gemma-4-31B-it leaves the top five | mixed board (open weights + API), composite B: d1-ultra (Liquid AI) enters at #1, wity-1 (Wity, reasoning auto) leaves the top five | mixed board (open weights + API), composite C: d1-ultra (Liquid AI) enters at #1, decisio v0.8.0 on gemma-4-31B-it leaves the top five | mixed board (open weights + API), Capability, official cost + latency caps (default): d1-ultra (Liquid AI) enters at #3, metask-jev-rain-31B-trained leaves the top five | mixed board (open weights + API), Capability, official cost cap + no latency cap: d1-ultra (Liquid AI) enters at #3, wity-1 (Wity, reasoning auto) leaves the top five | API board, composite A: d1-ultra (Liquid AI) enters at #1, wity-1 (Wity, reasoning auto) leaves the top five | API board, composite B: d1-ultra (Liquid AI) enters at #1, Jev 1.13.0 (TypeSafe AI) leaves the top five | API board, composite C: d1-ultra (Liquid AI) enters at #1, wity-1 (Wity, reasoning auto) leaves the top five | API board, Capability, official cost + latency caps (default): d1-ultra (Liquid AI) enters at #1, Mercury Decide leaves the top five | API board, Capability, cost off + latency 2x (Florian's settings): d1-ultra (Liquid AI) enters at #2, Jev 1.13.0 (TypeSafe AI) leaves the top five | API board, Capability, official cost cap + no latency cap: d1-ultra (Liquid AI) enters at #1, SPX-CD Flash leaves the top five. Ranked rows: mixed 193 -> 194, open weights 162 (unchanged), API 31 -> 32. No other row enters or leaves any top five.", "Publication of v1.6.21: on Florian's GO of 16 Oct 2026 in the laptop chat (\"do we have released all the new JevBench and ImageJevBench pages? If not, please go for it\"), with the merged-board top-five preview sent to him as a Telegram card first because d1-ultra changes top-five lists. The top-five preview (PNG sha256 b237ebbe44e0a74651f51db1435aff8aa951775ec8f22b11786ef97e170b5d85) was sent to Florian as Telegram card #17650. It offered a hold for 60 minutes; there was no answer, so it was published on his direct-message GO above at 2026-10-16T09:13Z (\"no answer within the 60-minute hold window\"; DECISIONS.md 2026-10-16 09:14Z). The card's own button was not pressed.", "liquid-d1-ultra: d1-ultra (Liquid AI). Measured 15 Oct 2026 on Florian's instruction (DECISIONS 15 Oct 2026 16:53Z: measure d1-ultra now) through the unchanged full-set runner run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `liquid` before dispatch, and none of this draw's items had been exposed to Liquid before (the v1.6.0 exposure of 2,705 items does not overlap). No paid order. 1,485 of 1,500 answered (HTTP 200: 1,485); 15 HTTP 422 refusals: in the API's own words those 15 prompts are \"over the model's input limit of 32768 tokens ... or 250000 bytes of text\", and they are scored as wrong answers in the full 1,500-item denominator (and count zero tokens). Two rate-limit stops (HTTP 429) during the run were resumed with a 2 s spacing between requests; the measurement report records that nothing was lost. Vendor-hosted api.liquid.ai/decisions/v1/systemone API, model `d1-ultra`, through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `liquid` before the first request (Florian GO 15 Oct 2026). Native probabilities on all three question types. 15 of 1,500 items exceed the model's 32,768-token input limit (HTTP 422) and are scored as failures. Liquid AI public price list, USD 0.042/M input, output free (fetched 16 Oct 2026): the models table of Liquid's public pricing page lists d1-ultra at USD 0.042 per 1M input tokens and USD 0.00 per 1M output tokens (saved copy sha256 89fb8ef2...); the same rate Liquid's API billed on the measurement's smoke call (48 input tokens -> USD 0.000002016). Counted as the manufacturer's standard launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops and SubQ Snap), through a scoped re-freeze of the frozen pricing module that adds this one key and nothing else, with a zero-difference proof for every other registry entry. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (638,436 input tokens over the 1,485 answered items, 0 output tokens), divided by the full 1,500-row basis the scorer uses; the 15 items refused with HTTP 422 count as zero tokens and as wrong answers. Not a bill and not a quote. Provider-reported usage: 638,436 input tokens over the 1,485 answered items and 0 output tokens; USD 0.017876208 per 1,000 decisions, 0.277x the official cost cap USD 0.064595, so the row is inside the official cost-capped views. Measured Intelligence 69.700, Calibration 91.254, Capability 80.477, Speed 89.000, Cost axis 62.432; composite A 76.105, B 74.732, C 76.105; on this draw rank 1 (A), 1 (B), 1 (C) and 3 by Capability, of 18 ranked rows. Latency on the official subset from Sandy (Germany), without API adjustment: p50 raw 0.2314 s, p95 raw 0.5442 s. Hosting implementation and deployed weights were not independently inspected. Raw sha256 f69bf80f48267017cd1fa506787850b81602ddfcea93a1451a45323b50ff045b; the scored copy is byte-identical.", "v1.6.22 ADDS 4 ROWS (Messier One v0.4 (messier-one-v0.4), Nemotron Deciffusion 8B v2 (pst2154) (nemotron-deciffusion-8b-v2), Nev-2B (EryriLabs) (nev-2b), Trio-Spark v1.3 (MachineFi) (trio-spark-v1.3)) to the 22 full-draw rows of v1.6.21 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so every carried row keeps identical point scores, axes, capability, cost and speed objects, and the listed-not-ranked rows (reflex-instinct-0.6b, nimble-v4, nimble-v4-max, sage-1.4.0) and the unranked A2 measurement keep their wording. Two of the new rows are RANKED (Messier One v0.4 (messier-one-v0.4), Nev-2B (EryriLabs) (nev-2b)); two are LISTED BUT NOT RANKED because the frozen pricing path gives them no price (Nemotron Deciffusion 8B v2 (pst2154) (nemotron-deciffusion-8b-v2), Trio-Spark v1.3 (MachineFi) (trio-spark-v1.3)): they show Intelligence, Calibration, Capability and Speed only, no Cost, no Composite and no rank in any view. LONG ITEMS: 15 items of this draw have very long states. From the scored raws of this release, they are refused outright (all 15, scored as wrong answers) by 11 rows: attest-31b-v1.0, julia-1, liquid-d1-ultra, nev-2b, pjev-2b, purplemist-mini-1.0, respan-span-02, sage-1.4.0, standard_one_3b_sh, trio-spark-v1.3, uniphore-4b-decider; phocinae-largha-150m-v1 refuses 3 of them; every other row answers them under its own recipe (on a truncated or compacted state, or read whole), as the row notes say. What moved on carried rows, counted exactly. RANKS: amos-1-large A 15 -> 17; B 15 -> 16; C 15 -> 16, amos-1-medium A 11 -> 12; B 11 -> 12; C 11 -> 12, julia-1 A 18 -> 20; B 18 -> 20; C 18 -> 20; capability 18 -> 20, krino-2b A 13 -> 14; B 13 -> 14; C 13 -> 14; capability 16 -> 18, metask_jev_rain_31b_trained_v3 A 6 -> 7, phocinae-largha-150m-v1 A 17 -> 19; B 17 -> 19; C 17 -> 19; capability 17 -> 19, pjev-2b A 12 -> 13; B 12 -> 13; C 12 -> 13; capability 12 -> 13, purplemist-mini-1.0 A 16 -> 18; B 16 -> 18; C 16 -> 18; capability 15 -> 17, respan-span-02 A 8 -> 9; B 8 -> 9, standard_one_3b_sh A 10 -> 11; B 10 -> 11; C 10 -> 11; capability 14 -> 15, subq-snap-preview A 7 -> 8; B 7 -> 8, uniphore-4b-decider A 9 -> 10; B 9 -> 10; C 9 -> 10; capability 13 -> 14, vllm-sr-d3 A 14 -> 16; B 14 -> 15; C 14 -> 15. Each moved rank shifts only by the newly ranked rows above it. Rows listed but not ranked take no place in any order. CONFIDENCE INTERVALS: as in v1.6.17, v1.6.18 and v1.6.21, new keys sort into the key-sorted roster, and the scorer seeds each row's bootstrap with the shared seed 16 plus the row's position in that roster, so a carried row whose position changed gets a different interval while no point estimate moves. Carried rows whose intervals moved: metask_jev_rain_12b_trained, metask_jev_rain_31b_trained, metask_jev_rain_31b_trained_v3, nimble-v4, nimble-v4-max, phocinae-largha-150m-v1, pjev-2b, purplemist-mini-1.0, reflex-instinct-0.6b, respan-span-02, sage-1.4.0, standard_one_3b_sh, subq-snap-preview, torchcast-decision-4b-v1.1, uniphore-4b-decider, vllm-sr-d3. DIAGNOSTICS: the API-basis median diagnostic is 7.703968975298743 (previous 7.703968975298743); carried v16.raw_on_A moves: none. Equating offsets are I 0.0681 / C 1.5418 (previous I 0.1005 / C 1.4616); the equating pool changes from 14 to 17 members (messier-one-v0.4, nemotron-deciffusion-8b-v2, nev-2b joined). These diagnostics are not part of any score, rank or axis on this release: the hosted-API row is a full-set row and is not equated, and the carried A2 measurement keeps its own equating and is not rescored. reflex-instinct-0.6b stays LISTED BUT NOT RANKED under #18761 and the unranked A2 measurement of v1.6.14 is carried; neither is changed. CORRECTION to the v1.6.20 note: the refreshed-preview Telegram card #17515 on which Florian's v1.6.20 approval was recorded had been sent to him on 14 Oct 2026 at 14:59Z (CoS deputy #19679); the v1.6.20 note did not state that time.", "Board effect: merged-live-board top-five comparison before/after v1.6.22 over the mixed, open-weights and API boards, composites A/B/C and Capability with official caps, no caps, cost off + latency 2x (Florian settings) and official cost cap + no latency cap: 0 of 21 top-five lists change. No row enters or leaves any top five. Best placement of the two new ranked rows on the merged board: messier-one-v0.4 #21 (open-weights board, composite A); nev-2b #71 (open-weights board, Capability, official cost cap + no latency cap). nemotron-deciffusion-8b-v2 and trio-spark-v1.3 are listed but not ranked and take no place in any ranking. Ranked rows: mixed 194 -> 196, open weights 162 -> 164, API 32 (unchanged).", "Publication of v1.6.22: on Florian's GO of 16 Oct 2026 12:50Z, a direct message in the laptop chat (\"I really want all of the models we have evaluated to be online (even if they land in #5) on both JevBench and ImageJevBench\"; DECISIONS.md 2026-10-16 12:50 UTC). No top-five list changes (see the board-effect note), so no top-five preview card was sent and none is claimed; the GO arrived as a direct message and no Telegram button was pressed.", "messier-one-v0.4: Messier One v0.4 (agentmessier-ai), a newer revision of the live messier-one-v0.2 row. Measured 14 Oct 2026 by the add-requests lane on the open draw under custody #19335 (one use), with the independent pre-score review before scoring. No paid order. 1,500 of 1,500 answered (HTTP 200: 1,500). LICENCE: weights CC BY-NC 4.0 (non-commercial; listing a measured score is evaluation use). Two readout facts, both the author's own and both documented by the author. (1) --noul-floor 0.801 is ON BY DEFAULT: a yes/no probability closer to 0.5 than the floor is reported at the floor, the decision never changes, and the author writes that this is because JevBench scores yes/no answers this way; the raw probability is not recoverable from a floored answer, so a separate floor-0 diagnostic pass over the 375 noul items only was run under custody #19341 and is reported as a sensitivity figure - it is NOT this row's score and never substitutes for it. (2) The author's Prompter TRUNCATES a state longer than 7,500 tokens to its beginning + '...[truncated]...' + its end: exactly 15 of the 1,500 items are affected, and they are the same 15 long items that other rows of this draw refuse outright, answer on truncated input or read whole (the rows that refuse them are listed in the v1.6.22 release note). Three different policies exist on this draw; each is the system's own recipe and is scored as such. An as-refused sensitivity figure (those 15 set invalid) is published next to this row's score. v0.4 also differs from the live v0.2 row in more than the floor: bidirectional attention, AA labels, the qboth layout, the gate and an 8,192-token window. Floor-0 sensitivity (custody #19341; a separate diagnostic pass over the 375 yes/no items with the floor off, substituted into this row offline): Capability 60.888 against the row's 62.444, Intelligence 48.282 against 52.025, yes/no Calibration 71.094 against 69.196: the floor is worth +1.56 Capability on this draw. It is a footnote only, never this row's score. As-refused sensitivity (those 15 long items scored as refusals): Capability 62.403 against the row's 62.444. SPEED: measured on one rented RTX A6000 48 GB, not the H100/RTX 5090 hardware most GPU rows of this draw used, so its Speed is not comparable to the live messier-one-v0.2 row. Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX A6000 48 GB - different hardware from the H100/RTX 5090 rows of this draw and from the RTX 3090 the author quotes. Started by the author's OWN ./serve_vllm.sh MODEL_DIR 8080 with its defaults, which is what the author insists on for v0.4: vLLM 0.30.0, --max-model-len 8192, chunked prefill and prefix caching OFF (this revision is non-causal), --max-logprobs 256, processed_logprobs, front end --max-state 7500 --max-total 8150 --gate 0.5 --noul-floor 0.801; transformers 5.18.0 exactly as pinned. One request at a time from the unchanged runner. Isolation: the server ran inside a network namespace with loopback only (unshare -n), the cut proven before the sealed input was staged and re-proven immediately before the first sealed request, with the synthetic smoke gate passed in between; a network namespace replaces docker --network none because the pod carried no local image (custody #19335, same property, proven the same way). The pod host kept its network for ssh. ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 dcfcecb0..., the module in force; its base-model references are unchanged since the dae86740... module this row was first priced with - the two later re-freezes each added one hosted-API launch-list key and nothing else): declared base Qwen/Qwen3.5-4B -> deepinfra:Qwen/Qwen3.5-4B USD 0.030000000000000002/M input, USD 0.15/M output, the same basis the live messier-one-v0.2 row carries. The author's server reports NO usage block of any kind, so the input tokens are counted OFFLINE with the author's own Prompter (prompt.py at tag v0.4) at the exact serving settings (--max-state 7500 --max-total 8150, AA labels, qboth layout) and cross-checked against vLLM's own prompt-token counters from the measured pass; output tokens are 0 (the engine is asked for one token restricted to the option labels and nothing is generated for the caller). The reference listing was deprecated after the 25 Sep 2026 cut-off and replaced by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3). Estimated, not charged, not a quote from the author or any host. Offline-counted input tokens (the author's server reports no usage): 1,105,885, matching vLLM's own prompt-token counter delta; 0 output tokens. USD 0.022117700 per 1,000 decisions, 0.342x the official cost cap USD 0.064595, so the row is inside the official cost-capped views. Measured Intelligence 52.025, Calibration 72.862, Capability 62.444, Speed 89.671, Cost axis 59.658; composite A 65.725, B 62.436, C 49.415; on this draw rank 6 (A), 7 (B), 9 (C) and 12 by Capability, of 20 ranked rows. Latency on the official 124-item subset: p50 raw 0.0664 s, p95 raw 0.1157 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.2829 s, p95 0.3813 s. Latency figures quoted in the endpoint condition are over all answered items. Raw sha256 c7a7841a102e676804cfbb9a12fca5b86c8a3845ff3990996f3d20a00ecf10cb; the scored copy (offline token counts injected) has sha256 d2a0c4f797b9e607aebf26b613920a0573844b537aa0ea4aa288d677295a80eb.", "nemotron-deciffusion-8b-v2: Nemotron Deciffusion 8B v2 (pst2154), a follow-up of the live nemotron-diffusion-8b row. Measured 15 Oct 2026 by the add-requests lane on the open draw under custody #19428, with the independent pre-score review before dispatch. No paid order. 1,500 of 1,500 answered (HTTP 200: 1,500). LICENCE: NVIDIA Nemotron Open Model License. LONG ITEMS: it answers the 15 long items on about 10% of their state through its own BM25 compaction, where other rows of this draw refuse them outright (listed in the v1.6.22 release note). As-refused sensitivity (the 15 compacted items scored as refusals): Capability 68.851 against the row's 69.320, i.e. answering them on about a tenth of their state is worth 0.469 Capability points. UNPRICED: the frozen pricing path raises on its declared base nvidia/Nemotron-Labs-Diffusion-8B (no market-listing decision; question #19427 is open); no rate is invented and the predecessor row's carried per-decision price is not reused. Five facts travel with this row, four of them established by the independent pre-score review (PASS WITH FINDINGS, no P1), which ran the author's own compaction offline before anything was dispatched. (1) THE 15 LONG ITEMS ARE ANSWERED, NOT REFUSED, AND ON ABOUT A TENTH OF THEIR STATE. The server reduces any prompt over 8,192 tokens to the most relevant 8,192 tokens with BM25 over sentences (no model call, deterministic, verified byte-identical across two offline runs). On this draw exactly 15 items are affected: 78,830-84,436 raw tokens cut to 8,002-8,385, i.e. 9.5-10.6% of the state kept as 21-27 of 210-238 fixed 70-word chunks in their original order. The question and its options are rendered outside the state and are never touched. The response carries no flag that it happened, so the compacted items are identifiable only by their input_tokens (8,002-8,385 against a maximum of 1,923 for every other item). Other rows of this draw, nev-2b among them, refuse those same items outright (listed in the v1.6.22 release note), so those 15 cells are not like for like; an as-refused sensitivity is published next to this row. The author's own long-context evidence (BABILong, QuALITY) is prose question answering and covers nothing of this draw's numeric, log-like long states. (2) THE ROW IS UNPRICED, AND NOT BECAUSE OF ANYTHING THE AUTHOR DID. The frozen pricing module raises on the declared base nvidia/Nemotron-Labs-Diffusion-8B - that base has no market-listing decision in the frozen table and the dated snapshot carries no listing of the family - so no cost number can be formed without a decision that is not this measurement's to make. The row is therefore listed, measured and shown without a cost and without a rank, the pre-declared default; the question is on the record and the server reports both token fields, so a later pricing decision needs no re-measurement. The predecessor nemotron-diffusion-8b row's carried per-decision price belongs to a different draw and is deliberately not reused. (3) PRECISION AND LIMITS ARE THE AUTHOR'S OWN. BF16 (QUANTIZATION=none) is the author's documented switch and the precision of every headline number on the author's card; the container's default is FP8. MAX_MODEL_LEN=16384 is the value the author's own start.sh picks for a memory-constrained GPU, and the review proved it behaviour-neutral on this draw (prompt limits of 65,534 / 16,382 / 10,000 / 8,450 all produce byte-identical prompts). Nothing else was changed. The row ran from the image DIGEST, not the mutable tag, and the review verified that image layer by layer against vllm/vllm-openai:nightly-fe4df3df0eb7 plus exactly the four author layers of the public repository. (4) OUTPUT TOKENS ARE COUNTED DIFFERENTLY HERE. The server reports output_tokens equal to the number of scored decision positions (one per request for us) under output_tokens_basis \"scored_decision_positions\", with generated_text_tokens 0; it generates no text. Rows such as pjev-2b report 0 for the same physical behaviour. The difference is immaterial while the row is unpriced and must be restated if it is ever priced. (5) DECONTAMINATION. The author states that no benchmark test, validation or dev split was trained on, with a row-level licence and overlap check, and that every training row was checked against the public JevBench items; the review found nothing in the repository contradicting that, but notes that the public JevBench set was used as a model-selection gate and that the tasksource sub-sources are unaudited. The licence (NVIDIA Nemotron Open Model License) permits this evaluation. Evaluator-owned rented Lium GPU pod (United States; measurement receipts on file), 1x NVIDIA RTX A6000 48 GB - different hardware from the L40S and GB200 the author quotes and from the H100/RTX 5090 rows of this draw. Started from the author's OWN published container, by DIGEST ghcr.io/pst2154/nemotron-deciffusion@sha256:944c0af4967ebd1e461a3b1faa9d16db197288af24393b21c917617461f35276 (image config sha256:0a4bd2d039b5c2da9ff8b181232ee3714f6b38e31bd9a3100a29326068d6f802), in the author's own documented sandboxed form: docker run --gpus all --shm-size=8g --network none --read-only --tmpfs /tmp:rw,exec,size=16g --user 1000:1000 --cap-drop ALL --security-opt no-new-privileges -v :/models/checkpoint:ro, with QUANTIZATION=none (BF16) and MAX_MODEL_LEN=16384 and every other default of the image (ATTENTION=noncausal, COMPACT_OVER=8192, COMPACT_BUDGET=8192, MAX_BATCHED_TOKENS=65536, --no-enable-prefix-caching, --max-logprobs 1052, --logprobs-mode processed_logits, vLLM nightly fe4df3df0eb7 with the author's overlay). /health reported back quantization none, attention noncausal, max_prompt_tokens 16382. All ten checkpoint files verified against the author's own SHA256SUMS (sha256sum -c, 10/10 OK) BEFORE the container started. One request at a time from the unchanged runner, which reached the server through the container's own network namespace with nsenter - the container published no port and had no network at all. The cut was proven before the sealed input was staged (only lo, no routes, a name probe and a bare-IP probe both failed) and re-proven immediately before the first sealed request, with a synthetic smoke gate passed in between (custody #19428). 1,500/1,500 answered, HTTP 200 on every item, raw p50 105 ms / p95 180 ms / max 1.82 s. unpriced: the frozen pricing path has no market-listing decision for the declared base nvidia/Nemotron-Labs-Diffusion-8B and raises rather than estimating; the deputy's answer to #19427 decides whether an entry is added UNPRICED. The frozen pricing_v15 module RAISES on the declared base nvidia/Nemotron-Labs-Diffusion-8B ('base model lacks an explicit market-listing decision') and the dated snapshot carries no listing of that family, so this row carries no pricing_base and is listed, measured and shown WITHOUT a cost number and WITHOUT a rank - the pre-declared default (Sage 1.4, Celeris Option A). No rate is invented. The predecessor nemotron-diffusion-8b row's usd_per_1000_override is a v1.5.4 historical carry on a different draw and is deliberately NOT reused. Asked of the deputy on board #19427; the server reports both token fields, so a later pricing decision needs no re-measurement. Measured Intelligence 49.600, Calibration 89.039, Capability 69.320 and Speed 87.704 only; NOT RANKED: no Cost, no Composite, no rank in any view, including the Capability order. Where to look: the collapsed \"All data\" table and the detail view. Provider-reported usage: 992,887 input tokens and 1,500 output tokens on the server's scored-decision-positions basis (no text generated). Latency on the official 124-item subset: p50 raw 0.1015 s, p95 raw 0.1654 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.3529 s, p95 0.4807 s. Latency figures quoted in the endpoint condition are over all answered items. Raw sha256 1a0953d441ab462cbdfad12537ba10a7e858331414843d04e5e035d6878d2c0f; scored copy sha256 b2d3d5047bb9fa49141d4ce9e724a93934508bf9f0760f1788c1345e242d2af7.", "nev-2b: Nev-2B (EryriLabs), GGUF Q8_0. Measured 15 Oct 2026 by the add-requests lane on the open draw under custody #19428 (one use per weight set), with the independent pre-score review before dispatch. No paid order. 1,485 of 1,500 answered (HTTP 200: 1,485); 15 HTTP 502 context-overflow refusals (llama-server's own 400 exceed_context_size_error at the author's documented -c 8192), scored as wrong answers in the full 1,500-item denominator and counted as zero tokens, so the cost excludes those refused long prompts. ENGLISH-ONLY by the author's declaration; 23.3% of this draw (349 of 1,500 items) is not English. SPEED is Sandy's loaded shared CPU, not the model: not comparable to the GPU and API rows of this draw. Four facts travel with this row, three of them found or sharpened by the independent pre-score review (PASS WITH FINDINGS, no P1). (1) CONTEXT, not capability, costs this row 15 items: the author's layout cuts a state longer than 20,000 characters (nev.layout.MAX_CONTEXT_CHARS) and the author's card calls that \"about 5,000 tokens\", but this draw's 15 long items are dense, digit-heavy structured text at about 1.6 characters per token, so the cut still leaves 12,463-14,914 tokens and every one of them overflows the author's own documented -c 8192. llama-server answers HTTP 400 exceed_context_size_error - it does not silently truncate, which was verified on synthetic filler before any sealed item (measurement receipts on file) - and the shim relays it as HTTP 502. These are the same 15 items that other rows of this draw refuse outright, answer on truncated input (messier-one-v0.4 among them) or read whole (the rows that refuse them are listed in the v1.6.22 release note). Three policies exist on this draw; each is the system's own recipe. (2) THE READOUT IS CALIBRATED, and the calibration can move a decision. The per-file temperatures stored in the GGUF metadata (choice 0.9220, noul 0.9169, score 0.8460, verified against the repo's calibration_Q8_0.json with our own GGUF reader, byte-identical) and, for choice questions with exactly four options, a fitted letter-position prior of at most 2.2% spread, are both the author's own and are the route the author's published 0.810 public-item number was measured through. On choice the temperature can never change the winner and the prior can only flip near-ties; on yes/no a temperature below 1 sharpens, so a raw probability just inside JevBench's 0.2/0.8 abstention band can become a decision instead of an abstention - never the opposite decision; on score it moves the expected value toward the modal level in either direction. This is the same class as the disclosed --noul-floor 0.801 of h2o-lightning-4b and messier-one-v0.4, but a continuous calibration rather than a discontinuous floor, and it reaches only items within about 2% of the band edge. A no-calibration sensitivity is published next to the row; it is recoverable exactly from the returned probabilities and cost no further model call. (3) PROVENANCE. Nev is distilled from the full option distributions of Mapika/decider-4b v2 and crh225/Plumb-4B - models, not items - and both are rows on this board, so neither is independent evidence for this one. Its training mix lists \"jev-bench/*\" configs, which are Praveenrajus/jev-bench, a DIFFERENT benchmark from JevBench; the author holds out the public JevBench items by exact-state hash, and this draw's items are sealed, bespoke and never published (the review measured containment against the public items at mean 0.005, max 0.23, and custody #19428 agreed the risk is low). One training source, avbiswas/bev-decision-150K, carries an unknown licence on its own card. (4) THE AUTHOR DECLARES THE MODEL ENGLISH-ONLY and 23.3% of this draw (349 of 1,500 items) is not English; the multilingual cells are where this row should be read most carefully. The no-calibration sensitivity, computed offline from the same raw with no further model call, moves 15 of the 375 yes/no answers across JevBench's 0.2/0.8 abstention boundary and flips exactly one of the 750 choice answers - the near-tie the four-option letter prior can reach. It is a sensitivity and a footnote, never this row's score. No-calibration sensitivity (recomputed offline from the same raw, no further model call): Capability 50.811 against the row's 49.917. A footnote only, never this row's score. Sandy, the evaluator's own SHARED CPU host (AMD Ryzen 5 3600, 6 cores / 12 threads, Zen 2, AVX2, no AVX-512), NOT a dedicated machine and NOT a GPU: other jobs ran beside it throughout and the 1-minute load average was sampled every 60 s (measurement receipts on file) - over this row's window of 199 samples it ran 6.89 to 35.91, mean 13.06. The author's own two documented processes, unedited: llama-server -m nev-Q8_0.gguf -c 8192 --port 8080 --metrics (llama.cpp b11372, the one build the author names; --metrics is the only addition and changes no model behaviour), and the author's shim PYTHONPATH=python python -m nev.serve --backend llama-server:http://127.0.0.1:8080 --gguf nev-Q8_0.gguf on port 8090, with --backend-arg timeout=900 (the author's own documented flag, raised from the 300 s default because a loaded shared box can otherwise turn an answerable item into a false refusal). llama.cpp chose n_threads=6, its own default; the author passes none. One request at a time from the unchanged runner. Isolation: a user+network namespace (unshare -Urn, loopback only - Sandy permits no privileged unshare -n), and the author's two processes additionally inside bwrap with a cleared environment, a read-only root, tmpfs /home and /tmp and read-only binds of only the GGUF, the llama.cpp build, the venv, its interpreter and the author's own python/ tree, so the sealed item directory does not exist inside that sandbox; the runner stays outside it and the author's code never sees the file. All three cuts were PROVEN before the sealed input was read (only lo, no routes, a name probe and a bare-IP probe both failing, and an ls of the sealed tree failing inside the sandbox) and the network cut was re-proven immediately before the first sealed request, with a synthetic smoke gate passed in between (custody #19428). 1,485/1,500 answered, the 15 refusals all HTTP 502 relaying llama-server's own 400 exceed_context_size_error. Raw p50 7.02 s / p95 15.29 s / max 56.88 s - this is the shared box, not the model, and it is not comparable to the GPU and API rows of this draw. ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 dcfcecb0..., the module in force; its base-model references are unchanged since the dae86740... module this row was first priced with - the two later re-freezes each added one hosted-API launch-list key and nothing else): declared base Qwen/Qwen3.5-2B-Base -> deepinfra:Qwen/Qwen3.5-2B USD 0.02/M input, USD 0.1/M output. The author's shim reports usage {calls, permutations, backend} and NO token fields at all, so the input tokens are counted OFFLINE with the author's own prompt code (nev.layout.single_prompts over hev.decider_layout.render_row) at the exact serving settings, and cross-checked against llama-server's own prompt-token counter delta from the measured pass; output tokens are 0, because one readout position per call is restricted to the option letters and nothing is returned as text (the pjev-2b convention and the 9 Oct CoS rule for native option-probability models). Estimated, not charged, and not a quote from the author. Offline-counted input tokens (the author's shim reports none): 685,652 over the 1,485 answered items, cross-checked against llama-server's own counter; 0 output tokens. USD 0.009142027 per 1,000 decisions, 0.142x the official cost cap USD 0.064595, so the row is inside the official cost-capped views. Measured Intelligence 20.232, Calibration 79.602, Capability 49.917, Speed 54.829, Cost axis 71.169; composite A 6.947, B 5.697, C 4.824; on this draw rank 15 (A), 17 (B), 17 (C) and 16 by Capability, of 20 ranked rows. Latency on the official 124-item subset: p50 raw 6.3431 s, p95 raw 12.7389 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 12.8362 s, p95 25.6278 s. Latency figures quoted in the endpoint condition are over all answered items. Raw sha256 76b58967b5133653a5530966338efde6c8b32f854fd66da707040dc2195df4f3; the scored copy (offline token counts injected) has sha256 3ca90aef11ed254956211d443887fb8cd0a3577f2bbf995f9667416dd5ccd22b.", "trio-spark-v1.3: Trio-Spark v1.3 (MachineFi), hosted API. Measured 15 Oct 2026 by the add-requests lane under custody #19732 (one hosted-API use) and deputy #19740, with the submitter's own API key; the full-set exposure gate logged all 1,200 sealed items to provider `machinefi` before dispatch. No paid order. 1,398 of 1,500 answered (HTTP 200: 1,398); 102 HTTP 422 refusals, scored as wrong answers in the full 1,500-item denominator: 58 because the question has 9 options and Trio-Spark supports at most 8, 43 over MachineFi's own documented 1,024-token input cap, 1 \"cannot be evaluated\". Latency is raw from Sandy with no API adjustment. UNPRICED: no eligible tariff under the frozen pricing path. Vendor-hosted `platform.machinefi.com/v1/systemone` API, model `trio-spark-v1.3`; latency measured from Sandy (Germany) without API adjustment — the submitter explicitly asked for it to be timed as a hosted endpoint with the network included (over all 1,398 answered requests p50 1.346 s, p95 1.855 s wall; the Speed axis uses the official per-item latency subset; loadavg samples for the whole run are in the measurement receipts on file). Full S1200+P300 draw of `v1.6-regular-20261012` through the unchanged `run_api_full_v161.py` (sha256 `eceea17e…`) and the unchanged `jevbench.adapters.typesafe` (sha256 `780f8794…`); the full-set exposure gate logged all 1,200 sealed items to provider `machinefi` before the first request (custody #19732; this draw's sealed items were already past the two-provider retirement threshold, so the admission cost the reserve nothing). Native System One answer fields on all three question types, nothing generated — with one honesty note the independent review asked for (P2-3): MachineFi's public docs describe only `choice`, so `noul` and `score` are documented here by observation, not by their documentation. The adapter's `noul` → P(yes) orientation is not assumed either: on the 231 PUBLIC items this mapping scores 80.6 % (54 of 67 noul items) against 19.4 % if it were inverted, which settles the direction without touching a sealed item (`output_tokens` 0 on every answered row, 573,254 input tokens over the full set). Hosting implementation and deployed weights were not independently inspected; no weights or training code are published. Regular-queue request `bh-submit:216579c8` (ref BBAEFC3A), measured with the submitter's own API key. 102 of 1,500 requests returned HTTP 422 `request_refused`, for three reasons given in MachineFi's own error messages: 58 because the question has 9 options and Trio-Spark supports at most 8 (\"Question decision has 9 options; Trio-Spark supports at most 8.\" - every 9-option item of this draw), 43 with \"Input exceeds the supported token limit.\" (MachineFi's documented cap of 1,024 model input tokens; answered prompts reach at most 955 billed input tokens), and 1 with \"The decision input cannot be evaluated.\" By type: 76 choice, 17 score, 9 noul. The runner never re-sends a failed item and no request was truncated or shortened, so all 102 are scored as wrong in the full 1,500 denominator. No further requests failed for other reasons. The refusal count was pre-declared at 27–193 before the run, expecting token-limit refusals only; the 8-option limit was not foreseen there, and the add-requests hand-over attributed all 102 to the token cap, which the raw's own error messages correct. The pre-data text is §4 of the pre-registration at sha256 `e4340cdb…`, published on the agent board at 22:45Z and 22:55Z — before the dispatch began at 23:03:30Z — and preserved byte-exactly in the measurement receipts on file; the file itself was amended twice afterwards and now hashes `2fbc0ceb…`. Measured, but the frozen pricing path gives it no eligible price: MachineFi publishes a usage tariff for Trio-Spark (USD 0.042 per 1M billed input tokens, output not billed), yet `trio-spark-v1.3` is in neither frozen tariff set (`AGE_QUALIFIED_API`, `LAUNCH_LIST_PRICE_API`) and the system declares no base model and releases no weights, so there is no base-model market reference either — the frozen module (unchanged on this point since the 32117d75... module it was first checked against) returns \"no eligible 30-day list price or base-model estimate\" (verified by executing it; measurement receipts on file). Without a cost axis there is no composite, so the row shows Intelligence, Calibration and Capability and is not ranked, the same outcome as `sage-1.4.0` and as `subq-snap-preview` before CoS deputy decision #19174. The pricing question is open and can be decided in a later release without re-measuring anything. UNPRICED under the frozen pricing_v15 path (sha256 dcfcecb06525ce4a5ece37400ea261b97d2fe5d41155f3cd02b03cac86572835, the module in force since v1.6.21; also under its predecessor 32117d754104b3ead8bebdc9da5f8e0116a7550f4316cfc6874fbe6234c2d72b, against which it was first checked): MachineFi publishes a usage tariff for Trio-Spark of USD 0.042 per 1M billed input tokens with no output charge (docs/visual.md and the README of machinefi/trio-spark), but `trio-spark-v1.3` is in neither AGE_QUALIFIED_API nor LAUNCH_LIST_PRICE_API, and the system declares no base model and releases no weights, so there is no base-model market reference to fall back on. The frozen path therefore returns no eligible price (the same outcome as sage-1.4.0, and as subq-snap-preview before CoS deputy #19174). Measured Intelligence 28.409, Calibration 85.643, Capability 57.026 and Speed 76.467 only; NOT RANKED: no Cost, no Composite, no rank in any view, including the Capability order. Where to look: the collapsed \"All data\" table and the detail view. Provider-reported usage: 573,254 input tokens over the 1,398 answered items and 0 output tokens. Latency on the official 124-item subset: p50 raw 1.2838 s, p95 raw 1.7570 s, without API adjustment. Latency figures quoted in the endpoint condition are over all answered items. Hosting implementation and deployed weights were not independently inspected. Raw sha256 19cdb9cf78bac0f72b96e4f6af0ae3c76d392b5b959870b83b5ed132238244bf; scored copy sha256 23db012ec50f2876526d8f52225585c0b2b49fb2cbb6ebbc5c13ddfdfbe844c9.", "v1.6.23 PRICES 3 ROWS (Bespoke Nimble-v4 (Bespoke Labs) (nimble-v4), Bespoke Nimble-v4-max (Bespoke Labs) (nimble-v4-max), Sage 1.4 (Levanto Labs) (sage-1.4.0)) that v1.6.22 listed, measured and not ranked because the frozen pricing path gave them no price. Their vendors publish list prices (Levanto: USD 0.05/M input, USD 10/M output; Bespoke: nimble-v4 USD 0.04/M and nimble-v4-max USD 0.09/M input, output free), admitted under interpretation I-1 by a scoped re-freeze of the pricing module that adds exactly these three keys (zero difference for every other registry entry). No model was called again: each row keeps its measured responses, Intelligence, Calibration, Capability and Speed, and gains a Cost axis, composites and ranks. nimble-v4-max is over the official cost cap (disclosed) and is ranked outside the official cost-capped views; Sage 1.4's median latency (1.74 s raw) is above the official latency cap, so it is ranked on the composites but sits outside the default Capability view. Every other row keeps identical point scores, axes, capability, cost, speed and intervals; reflex-instinct-0.6b, nemotron-deciffusion-8b-v2, trio-spark-v1.3 and the unranked A2 measurement keep their listed-not-ranked wording. Sage 1.3.0 keeps its v1.6.1 row and rank; Sage 1.4 is Levanto's current version. What moved on carried rows, counted exactly. RANKS: amos-1-large A 17 -> 20; B 16 -> 19; C 16 -> 19; capability 5 -> 6, amos-1-medium A 12 -> 15; B 12 -> 15; C 12 -> 15; capability 11 -> 14, attest-31b-v1.0 A 3 -> 4; B 3 -> 4; C 2 -> 3; capability 8 -> 10, julia-1 A 20 -> 23; B 20 -> 23; C 20 -> 23; capability 20 -> 23, krino-2b A 14 -> 17; B 14 -> 17; C 14 -> 17; capability 18 -> 21, messier-one-v0.4 A 6 -> 7; B 7 -> 8; C 9 -> 10; capability 12 -> 15, metask_jev_rain_12b_trained A 4 -> 5; B 2 -> 3; C 3 -> 4; capability 7 -> 8, metask_jev_rain_31b_trained A 5 -> 6; B 4 -> 5; C 4 -> 5; capability 6 -> 7, metask_jev_rain_31b_trained_v3 A 7 -> 8; B 6 -> 7; C 5 -> 6; capability 4 -> 5, nev-2b A 15 -> 18; B 17 -> 20; C 17 -> 20; capability 16 -> 19, phocinae-largha-150m-v1 A 19 -> 22; B 19 -> 22; C 19 -> 22; capability 19 -> 22, pjev-2b A 13 -> 16; B 13 -> 16; C 13 -> 16; capability 13 -> 16, purplemist-mini-1.0 A 18 -> 21; B 18 -> 21; C 18 -> 21; capability 17 -> 20, respan-span-02 A 9 -> 10; B 9 -> 10; C 8 -> 9; capability 9 -> 11, standard_one_3b_sh A 11 -> 14; B 11 -> 14; C 11 -> 14; capability 15 -> 18, subq-snap-preview A 8 -> 9; B 8 -> 9; C 6 -> 7, torchcast-decision-4b-v1.1 B 5 -> 6; C 7 -> 8; capability 10 -> 12, uniphore-4b-decider A 10 -> 13; B 10 -> 13; C 10 -> 13; capability 14 -> 17, vllm-sr-d3 A 16 -> 19; B 15 -> 18; C 15 -> 18. Each moved rank shifts only by the newly ranked rows above it. CONFIDENCE INTERVALS: no key joins or leaves the roster, so every bootstrap seed and every carried interval is unchanged. Carried rows whose intervals moved: none. DIAGNOSTICS: the API-basis median diagnostic is 7.703968975298743 (unchanged); equating offsets I 0.0681 / C 1.5418 and the equating pool are unchanged; carried v16.raw_on_A moves: none. reflex-instinct-0.6b stays LISTED BUT NOT RANKED under #18761 and the unranked A2 measurement of v1.6.14 is carried; neither is changed.", "Board effect: merged-live-board top-five comparison before/after v1.6.23 over the mixed, open-weights and API boards, composites A/B/C and Capability with official caps, no caps, cost off + latency 2x (Florian settings) and official cost cap + no latency cap: 2 of 21 top-five lists change. API board, Capability, cost off + latency 2x (Florian's settings): Bespoke Nimble-v4-max (Bespoke Labs) (nimble-v4-max) enters at #3; Sage 1.3.0 (Levanto Labs) leaves the top five | API board, Capability, official cost cap + no latency cap: Sage 1.4 (Levanto Labs) (sage-1.4.0) enters at #5; d1 (Liquid AI) leaves the top five. Best placement of the three repriced rows on the merged board: nimble-v4 #11 (API board, Capability, official cost + latency caps (default)); nimble-v4-max #3 (API board, Capability, cost off + latency 2x (Florian's settings)); sage-1.4.0 #5 (API board, Capability, official cost cap + no latency cap). Ranked rows: mixed 196 -> 199, API 32 -> 35, open weights 164 (unchanged).", "Publication of v1.6.23: on Florian's priority instruction of 16 Oct 2026 ~18:00 Berlin, relayed by the chief of staff (\"we'll queue it into the evaluation with high priority (and actually do it please)\"; agent board #20217 (CoS decision)), and his GO of 16 Oct 2026 12:50Z, a direct message in the laptop chat (\"I really want all of the models we have evaluated to be online (even if they land in #5) on both JevBench and ImageJevBench\"; DECISIONS.md 2026-10-16 12:50 UTC). Florian selected \"Release (recommended)\" on Telegram card #17708 (2026-10-16T17:58:19Z), the top-five preview card with preview PNG sha256 53ec5cae5f02ee93e4a58076691f6dd306aaeb7b9eebe6fe63d05bc0296959a0.", "nimble-v4 PRICED in v1.6.23: Bespoke Nimble-v4 (Bespoke Labs), measured in v1.6.18 (last measured 2026-10-14); no model call in this release, same raw scoring copy (sha256 9df0f0e4919deccb8448004ece256f7ba76f9160308b37378405add632f0f978). Public list price read from docs.bespokelabs.ai/nimble/pricing on 2026-10-16 (saved copy sha256 03b0626140b8fb0a80f48754742bd4d553d5632ff78ce5790770a12e2bd58562): USD 0.04/M input, USD 0/M output, counted under interpretation I-1 through the v1.6.23 re-freeze of the pricing module (sha256 533d80f0..., predecessor dcfcecb0...). Provider-reported tokens: 1,935,351 input and 3,000 output over the full 1,500-row basis. USD 0.051609360 per 1,000 decisions, 0.799x the official cost cap USD 0.064595, inside the official cost cap. Unchanged measured Intelligence 41.405, Calibration 86.634, Capability 64.020, Speed 88.399; new Cost axis 48.618; composite A 38.379, B 35.343, C 26.652; on this draw rank 11 (A), 12 (B), 12 (C) and 13 by Capability, of 23 ranked rows. The 3,000 provider-reported output tokens are free under the tariff.", "nimble-v4-max PRICED in v1.6.23: Bespoke Nimble-v4-max (Bespoke Labs), measured in v1.6.18 (last measured 2026-10-14); no model call in this release, same raw scoring copy (sha256 dddfc639b98a07599170850f3b71702c976891f3a67a1b52ba2101a0e30a1505). Public list price read from docs.bespokelabs.ai/nimble/pricing on 2026-10-16 (saved copy sha256 03b0626140b8fb0a80f48754742bd4d553d5632ff78ce5790770a12e2bd58562): USD 0.09/M input, USD 0/M output, counted under interpretation I-1 through the v1.6.23 re-freeze of the pricing module (sha256 533d80f0..., predecessor dcfcecb0...). Provider-reported tokens: 1,935,351 input and 3,000 output over the full 1,500-row basis. USD 0.116121060 per 1,000 decisions, 1.798x the official cost cap USD 0.064595, OVER the official cap, so it is ranked but falls outside the official cost-capped views (disclosed in its cost basis). Unchanged measured Intelligence 68.185, Calibration 91.250, Capability 79.717, Speed 91.451; new Cost axis 38.053; composite A 36.869, B 37.365, C 36.869; on this draw rank 12 (A), 11 (B), 11 (C) and 4 by Capability, of 23 ranked rows. The 3,000 provider-reported output tokens are free under the tariff. OVER THE OFFICIAL COST CAP, DISCLOSED: the official-caps views exclude it by construction; it is ranked on the composites and in the cost-relaxed views.", "sage-1.4.0 PRICED in v1.6.23: Sage 1.4 (Levanto Labs), measured in v1.6.17 (last measured 2026-10-13); no model call in this release, same raw scoring copy (sha256 e7a2733a8f6b41729be92076dd5f067c442ef5f917330a2941263d095821763c). Public list price read from levanto.ai/pricing on 2026-10-16T15:57:17Z (saved copy sha256 1e98749c9601710ceae68bca591707535b4dde0987da82309055f73352860eb0): USD 0.05/M input, USD 10/M output, counted under interpretation I-1 through the v1.6.23 re-freeze of the pricing module (sha256 533d80f0..., predecessor dcfcecb0...). Provider-reported tokens: 665,989 input and 0 output over the full 1,500-row basis. USD 0.022199633 per 1,000 decisions, 0.344x the official cost cap USD 0.064595, inside the official cost cap. Unchanged measured Intelligence 65.882, Calibration 83.665, Capability 74.773, Speed 68.800; new Cost axis 59.610; composite A 68.444, B 67.916, C 68.444; on this draw rank 3 (A), 2 (B), 2 (C) and 9 by Capability, of 23 ranked rows. The provider reported 0 output (reasoning) tokens on every answered item, so the USD 10/M output term contributes nothing as measured. Its 15 HTTP 400 and 2 HTTP 504 requests stay scored as wrong answers. LATENCY: p50 1.7411 s raw from Sandy, above the official latency cap, so the default Capability view (official caps) does not list it; the latency-relaxed views do. Successor of sage-1.3.0, whose v1.6.1 row stays on the board as the older version.", "v1.6.24 ADDS 8 ROWS (ArseneLupin v1.1 (arsenelupin-v1.1), Bongard-mini (AgentBull, T5Gemma-2 4B-4B) (bongard-mini), Decision 2.0 Lux 9B (decision2-lux-9b), Imajev-2B (mohit67890, Qwen3.5-2B LoRA) (imajev-2b), Imajev-9B (mohit67890, Qwen3.5-9B LoRA) (imajev-9b), kettei-chub (Sqwish) (kettei-chub), kettei-fat (Sqwish) (kettei-fat), Noma (noma)) to the 26 full-draw rows of v1.6.23 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so every carried row keeps identical point scores, axes, capability, cost and speed objects, and the listed-not-ranked rows (reflex-instinct-0.6b, nemotron-deciffusion-8b-v2, trio-spark-v1.3) and the unranked A2 measurement keep their wording. Six are previously evaluated systems RE-MEASURED on this draw (their earlier results were on retired protocols): five are RANKED, priced from their frozen base-model references (ArseneLupin v1.1 (arsenelupin-v1.1), Decision 2.0 Lux 9B (decision2-lux-9b), Imajev-2B (mohit67890, Qwen3.5-2B LoRA) (imajev-2b), Imajev-9B (mohit67890, Qwen3.5-9B LoRA) (imajev-9b), Noma (noma); imajev-9b over the official cost cap, disclosed), and one is LISTED BUT NOT RANKED because the frozen pricing path gives it no price (Bongard-mini (AgentBull, T5Gemma-2 4B-4B) (bongard-mini)): it shows Intelligence, Calibration, Capability and Speed only, no Cost, no Composite and no rank in any view. decision2-lux-9b and noma supersede their historical v1.5 measurements on the v1.5 supplement page. The two PAID fast-lane rows (kettei-chub (Sqwish) (kettei-chub), kettei-fat (Sqwish) (kettei-fat); Sqwish, hosted API) are RANKED from Sqwish's public launch list price through a scoped re-freeze of the pricing module. LONG ITEMS: 15 items of this draw have very long states. From the scored raws of this release, they are refused or fail outright (all 15, scored as wrong answers) for 18 rows: arsenelupin-v1.1, attest-31b-v1.0, bongard-mini, decision2-lux-9b, imajev-2b, imajev-9b, julia-1, kettei-chub, kettei-fat, liquid-d1-ultra, nev-2b, pjev-2b, purplemist-mini-1.0, respan-span-02, sage-1.4.0, standard_one_3b_sh, trio-spark-v1.3, uniphore-4b-decider; phocinae-largha-150m-v1 fails 3 of them; every other row answers them under its own recipe (on a truncated or compacted state, or read whole), as the row notes say. What moved on carried rows, counted exactly. RANKS: amos-1-large A 20 -> 27; B 19 -> 26; C 19 -> 25; capability 6 -> 7, amos-1-medium A 15 -> 18; B 15 -> 18; C 15 -> 18; capability 14 -> 16, attest-31b-v1.0 A 4 -> 6; B 4 -> 6; C 3 -> 5; capability 10 -> 12, julia-1 A 23 -> 30; B 23 -> 30; C 23 -> 30; capability 23 -> 30, krino-2b A 17 -> 23; B 17 -> 23; C 17 -> 23; capability 21 -> 28, liquid-d1-ultra A 1 -> 2; B 1 -> 2; C 1 -> 2; capability 3 -> 4, messier-one-v0.4 A 7 -> 9; B 8 -> 10; C 10 -> 12; capability 15 -> 17, metask_jev_rain_12b_trained A 5 -> 7; B 3 -> 5; C 4 -> 6; capability 8 -> 9, metask_jev_rain_31b_trained A 6 -> 8; B 5 -> 7; C 5 -> 7; capability 7 -> 8, metask_jev_rain_31b_trained_v3 A 8 -> 10; B 7 -> 9; C 6 -> 8; capability 5 -> 6, nev-2b A 18 -> 25; B 20 -> 27; C 20 -> 27; capability 19 -> 26, nimble-v4 A 11 -> 13; B 12 -> 14; C 12 -> 14; capability 13 -> 15, nimble-v4-max A 12 -> 14; B 11 -> 13; C 11 -> 13; capability 4 -> 5, phocinae-largha-150m-v1 A 22 -> 29; B 22 -> 29; C 22 -> 29; capability 22 -> 29, pjev-2b A 16 -> 20; B 16 -> 20; C 16 -> 20; capability 16 -> 20, purplemist-mini-1.0 A 21 -> 28; B 21 -> 28; C 21 -> 28; capability 20 -> 27, respan-span-02 A 10 -> 12; B 10 -> 12; C 9 -> 11; capability 11 -> 13, sage-1.4.0 A 3 -> 5; B 2 -> 4; C 2 -> 4; capability 9 -> 10, standard_one_3b_sh A 14 -> 17; B 14 -> 17; C 14 -> 17; capability 18 -> 22, subq-snap-preview A 9 -> 11; B 9 -> 11; C 7 -> 9; capability 2 -> 3, torchcast-decision-4b-v1.1 A 2 -> 4; B 6 -> 8; C 8 -> 10; capability 12 -> 14, uniphore-4b-decider A 13 -> 16; B 13 -> 16; C 13 -> 16; capability 17 -> 21, vllm-sr-d3 A 19 -> 26; B 18 -> 25; C 18 -> 24; capability 1 -> 2. Each moved rank shifts only by the newly ranked rows above it. Rows listed but not ranked take no place in any order. CONFIDENCE INTERVALS: as in v1.6.17, v1.6.18, v1.6.21 and v1.6.22, new keys sort into the key-sorted roster, and the scorer seeds each row's bootstrap with the shared seed 16 plus the row's position in that roster, so a carried row whose position changed gets a different interval while no point estimate moves. Carried rows whose intervals moved: attest-31b-v1.0, julia-1, krino-2b, liquid-d1-ultra, messier-one-v0.4, metask_jev_rain_12b_trained, metask_jev_rain_31b_trained, metask_jev_rain_31b_trained_v3, nemotron-deciffusion-8b-v2, nev-2b, nimble-v4, nimble-v4-max, phocinae-largha-150m-v1, pjev-2b, purplemist-mini-1.0, reflex-instinct-0.6b, respan-span-02, sage-1.4.0, standard_one_3b_sh, subq-snap-preview, torchcast-decision-4b-v1.1, trio-spark-v1.3, uniphore-4b-decider, vllm-sr-d3. DIAGNOSTICS: the API-basis median diagnostic is 8.162049591230426 (previous 7.703968975298743); carried v16.raw_on_A moves: standard_one_3b_sh: intelligence 28.7534 -> 28.8915, capability 52.8636 -> 52.9327. Equating offsets are I 0.0681 / C 2.5891 (previous I 0.0681 / C 1.5418); the equating pool changes from 17 to 23 members (arsenelupin-v1.1, bongard-mini, decision2-lux-9b, imajev-2b, imajev-9b, noma joined). These diagnostics are not part of any score, rank or axis on this release: the hosted-API rows are full-set rows and are not equated, and the carried A2 measurement keeps its own equating and is not rescored. reflex-instinct-0.6b stays LISTED BUT NOT RANKED under #18761 and the unranked A2 measurement of v1.6.14 is carried; neither is changed.", "Board effect: merged-live-board top-five comparison before/after v1.6.24 over the mixed, open-weights and API boards, composites A/B/C and Capability with official caps, no caps, cost off + latency 2x (Florian settings) and official cost cap + no latency cap: 12 of 21 top-five lists change. mixed board (open weights + API), composite A: kettei-fat (Sqwish) (kettei-fat) enters at #1, kettei-chub (Sqwish) (kettei-chub) enters at #4; H2O-Lightning-4B v1.1, Mercury Decide leave the top five | mixed board (open weights + API), composite B: kettei-fat (Sqwish) (kettei-fat) enters at #1; Mercury Decide leaves the top five | mixed board (open weights + API), composite C: kettei-fat (Sqwish) (kettei-fat) enters at #1, kettei-chub (Sqwish) (kettei-chub) enters at #4; H2O-Lightning-4B v1.1, Mercury Decide leave the top five | mixed board (open weights + API), Capability, official cost + latency caps (default): kettei-fat (Sqwish) (kettei-fat) enters at #1; decisio v0.8.0 on gemma-4-31B-it leaves the top five | mixed board (open weights + API), Capability, cost off + latency 2x (Florian's settings): kettei-fat (Sqwish) (kettei-fat) enters at #1; Jeff-1.0-Large leaves the top five | mixed board (open weights + API), Capability, official cost cap + no latency cap: kettei-fat (Sqwish) (kettei-fat) enters at #1; decisio v0.8.0 on gemma-4-31B-it leaves the top five | API board, composite A: kettei-fat (Sqwish) (kettei-fat) enters at #1, kettei-chub (Sqwish) (kettei-chub) enters at #4; Mercury Decide, Jev 1.13.0 (TypeSafe AI) leave the top five | API board, composite B: kettei-fat (Sqwish) (kettei-fat) enters at #1, kettei-chub (Sqwish) (kettei-chub) enters at #5; wity-1 (Wity, reasoning auto), d1 (Liquid AI) leave the top five | API board, composite C: kettei-fat (Sqwish) (kettei-fat) enters at #1, kettei-chub (Sqwish) (kettei-chub) enters at #4; Mercury Decide, Jev 1.13.0 (TypeSafe AI) leave the top five | API board, Capability, official cost + latency caps (default): kettei-fat (Sqwish) (kettei-fat) enters at #1; SPX-CD Flash leaves the top five | API board, Capability, cost off + latency 2x (Florian's settings): kettei-fat (Sqwish) (kettei-fat) enters at #1; SPX-CD Pro leaves the top five | API board, Capability, official cost cap + no latency cap: kettei-fat (Sqwish) (kettei-fat) enters at #1; Sage 1.4 (Levanto Labs) leaves the top five. Top-five changes caused by: kettei-chub, kettei-fat (paid fast-lane rows only). Best placement of the new ranked rows on the merged board: arsenelupin-v1.1 #65 (open-weights board, Capability, official cost + latency caps (default)); decision2-lux-9b #45 (open-weights board, Capability, official cost + latency caps (default)); imajev-2b #70 (open-weights board, Capability, official cost + latency caps (default)); imajev-9b #57 (open-weights board, Capability, cost off + latency 2x (Florian's settings)); kettei-chub #4 (mixed board (open weights + API), composite A); kettei-fat #1 (mixed board (open weights + API), composite A); noma #68 (open-weights board, Capability, official cost + latency caps (default)). Ranked rows: mixed 199 -> 206, open weights 164 -> 169, API 35 -> 37.", "Publication of v1.6.24: the six re-measured rows on Florian's GO of 16 Oct 2026 12:50Z, a direct message in the laptop chat (\"I really want all of the models we have evaluated to be online (even if they land in #5) on both JevBench and ImageJevBench\"; DECISIONS.md 2026-10-16 12:50 UTC); the two paid fast-lane rows under his standing fast-lane rule (DECISIONS.md 29 Sep 2026 ~00:25 Berlin (Florian)). Every top-five change comes from the two paid fast-lane rows (kettei-fat, kettei-chub; public orders a87f712c and 6e10efe6); no other new row enters or leaves any top five. Under Florian's standing fast-lane rule (\"fast lane runs fully autonomously end-to-end incl. publication, customer mail and an @airesearch12 post when the top 5 change; paid runs no longer need his top-5 preview\"; DECISIONS.md 29 Sep 2026 ~00:25 Berlin (Florian)) a paid release that changes a top five publishes without a separate preview, as v1.6.13 did for the paid attest-31b-v1.0 row; so no top-five preview card was sent and none is claimed, and no Telegram button was pressed.", "arsenelupin-v1.1: ArseneLupin v1.1 (IBOU-SEARCH, Qwen3.5-4B LoRA). RE-MEASURED on this draw in the re-measure campaign (Batch A; Florian 16 Oct 2026 \"all evaluated models online\"), measured 16 Oct 2026 under custody #20150 (one self-hosted use) with GPU per deputy #20098, after an independent pre-dispatch review and with an independent post-run score review. No paid order and no author contact. 1,485 of 1,500 answered (HTTP 200: 1,485, HTTP 422: 15). 15 refusals on the long items (HTTP 413 from the author's server, mapped to 422), scored as wrong answers. The server reports prompt_tokens 0, so the input tokens are counted offline with the author's tokenizer and shared-prefix method. SPEED: measured on one rented RTX 5090 32 GB, not the RTX PRO 6000 / H100 of the earlier runs, so its Speed is as measured on this card and not comparable to the earlier measurement; probabilities may differ in the last digits. Evaluated pin IBOU-SEARCH/ArseneLupin-v1.1@411a6aefab87a4c155372c2a0a20777c2f5e1317 (code: code/ folder of IBOU-SEARCH/ArseneLupin-v1.1 @ 411a6aefab87a4c155372c2a0a20777c2f5e1317; evaluator wire-name proxy al_wire_proxy.py (sha ee5467f4..., verbatim from the earlier measurement)). LICENCE: Apache-2.0. Served by the author's own System One server behind the evaluator's wire-name proxy (noul p_yes -> noul, score list -> index map, HTTP 413 -> 422; no probability changed). The 15 long items exceed the author's max_branch_tokens and are refused (413 -> 422). The author's shipped max_batch_tokens was kept on the 32 GB card. Training data are undisclosed on the card, so overlap cannot be screened. Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's `python -m arsenelupin serve` (engine shared_prefix, bf16, calibration T 2.45) behind the wire-name proxy. ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-4B -> deepinfra:Qwen/Qwen3.5-4B USD 0.030000000000000002/M input, USD 0.15/M output (frozen at the 25 Sep cut-off, I-3). Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens: the server reports prompt_tokens 0 (not a measurement), so input is counted OFFLINE with the author's tokenizer and shared-prefix method (the method of the earlier measurement, measurement receipts on file; 1,011,369 over 1,485), output 0 under the 9 Oct 2026 rule (forward passes only). USD 0.020227380 per 1,000 decisions, 0.313x the official cost cap USD 0.064595, inside the official cost-capped views. Measured Intelligence 27.956, Calibration 82.226, Capability 55.091, Speed 89.695, Cost axis 60.822; composite A 16.558, B 14.045, C 11.499; on this draw rank 19 (A), 19 (B), 19 (C) and 23 by Capability, of 30 ranked rows. Latency on the official 124-item subset: p50 raw 0.0845 s, p95 raw 0.0932 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.3189 s, p95 0.3363 s. Raw sha256 59d02a3d7bf85d11445e4f983fb12f545aff618d22c3691b3e48c27d91c54b36; scored copy sha256 8c0bde3bc6faf59b768be089f2f1bcbbed5ed307739607e15f7edacdd47045f2.", "bongard-mini: Bongard-mini (AgentBull, T5Gemma-2 4B-4B). RE-MEASURED on this draw in the re-measure campaign (Batch A; Florian 16 Oct 2026 \"all evaluated models online\"), measured 16 Oct 2026 under custody #20150 (one self-hosted use) with GPU per deputy #20098, after an independent pre-dispatch review and with an independent post-run score review. No paid order and no author contact. 1,485 of 1,500 answered (HTTP 200: 1,485, HTTP 400: 15). 15 HTTP 400 refusals on the long items (over the author's 64,000-token request budget), scored as wrong answers. UNPRICED: the frozen pricing path raises on its declared base google/t5gemma-2-4b-4b (no market-listing decision); no rate is invented. SPEED: measured on one rented RTX 5090 32 GB, not the RTX PRO 6000 / H100 of the earlier runs, so its Speed is as measured on this card and not comparable to the earlier measurement; probabilities may differ in the last digits. Evaluated pin AgentBull/bongard-mini@c81c5b883a5983669be84be98658eac940489613 (code: AgentBull/bongard @ 3e6e6030a953dfe8d30d93aca2d0ab92b87b4a5a (bongard serve, option rotation off)). LICENCE: Gemma terms of use. Full post-train of google/t5gemma-2-4b-4b plus a judgment head (7.5B parameters, encoder-decoder). Pinned to c81c5b88 (not Hub HEAD 88bb70a7: layout changed, weights file identical). The 15 long items exceed the author's max_request_tokens 64,000 and are refused (HTTP 400). Gemma terms of use apply; the backbone repo is gated. Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's own `bongard serve` with their temperatures.json, default token budgets (64,000 request / 32,768 sequence), bf16. UNPRICED. The frozen pricing_v15 module (sha256 dcfcecb0...2835, and every later re-freeze, which only add hosted-API launch-list keys) RAISES on the declared base google/t5gemma-2-4b-4b ('base model lacks an explicit market-listing decision'); no price is invented here. Axes only: Intelligence, Calibration and Speed are computed; Cost, composite and rank are void. Tokens as reported by the author's server (input 786,538, output_tokens 0). unpriced: the frozen pricing path has no market-listing decision for the declared base google/t5gemma-2-4b-4b and raises rather than estimating; one listing decision would make the row rankable. Measured Intelligence 25.214, Calibration 84.358, Capability 54.786 and Speed 90.583 only; NOT RANKED: no Cost, no Composite, no rank in any view, including the Capability order. Where to look: the collapsed \"All data\" table and the detail view. Latency on the official 124-item subset: p50 raw 0.0695 s, p95 raw 0.0762 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.2891 s, p95 0.3025 s. Raw sha256 76b9f06e3b1d4ae8b2cd91d4730e44dc3e5453aafee51a96ecaf03e6a7fe9aae; scored copy sha256 29db8fd9c8e338aeadebaf154e56d5a6d09d1ba460c10171e17b2858278e82ae.", "decision2-lux-9b: Decision 2.0 Lux 9B (vLLM-SR, Qwen3.5-9B lineage). RE-MEASURED on this draw in the re-measure campaign (Batch A; Florian 16 Oct 2026 \"all evaluated models online\"), measured 16 Oct 2026 under custody #20150 (one self-hosted use) with GPU per deputy #20098, after an independent pre-dispatch review and with an independent post-run score review. No paid order and no author contact. It supersedes the historical v1.5 measurement of decision2-lux-9b shown on the v1.5 supplement page (/jev-models/v1.5.7; 1,624-item retired protocol, never ranked on the v1.6 boards); that page stays as the record of the old protocol. 1,485 of 1,500 answered (HTTP 200: 1,500). Its 15 failures are HTTP 200 responses whose answer could not be parsed (on the 15 long items), not refusals; they are scored as wrong answers in the full 1,500-item denominator. The authors' manifests name the public 231 items among the scored sets (the public items are part of this draw's open set). Speed is a lower bound (no fla / causal-conv1d kernels on this stack). SPEED: measured on one rented RTX 5090 32 GB, not the RTX PRO 6000 / H100 of the earlier runs, so its Speed is as measured on this card and not comparable to the earlier measurement; probabilities may differ in the last digits. Evaluated pin vllm-sr/Decision-2.0-Lux-9B@78bf3c03d9147aeb30b641edfe0e30ed04887ca5 (code: authors' packaged runtime decision2.api.Decision2.system_one behind the evaluator's loopback shim, verbatim from the earlier v1.5 measurement). LICENCE: Apache-2.0. Full fine-tune of vllm-sr/Decision-1.0-Lux-9B. The author's manifests name the public 231 items among the scored sets. The 15 long items came back as HTTP 200 'answer parse failed' (not the shim's 422 as predicted) and are counted as failed answers. Authors' runtime with share_context=False, bf16_resident=True, graphs and kernels off; no fla / causal-conv1d, so Speed is a lower bound. Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Authors' own in-process runtime behind the evaluator's loopback shim (verbatim from the earlier v1.5 measurement), serial requests. ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: lineage base Qwen/Qwen3.5-9B (the card's base vllm-sr/Decision-1.0-Lux-9B is itself a fine-tune and RAISES in the module; Qwen3.5-9B lineage verified by config comparison in the earlier v1.5 measurement) -> qwen/qwen3.5-9b USD 0.09999999999999999/M input, USD 0.15/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens as reported by the author's runtime (output_tokens 0, nothing generated). USD 0.051756600 per 1,000 decisions, 0.801x the official cost cap USD 0.064595, inside the official cost-capped views. Measured Intelligence 35.823, Calibration 87.476, Capability 61.650, Speed 89.933, Cost axis 48.581; composite A 27.281, B 24.482, C 18.945; on this draw rank 15 (A), 15 (B), 15 (C) and 19 by Capability, of 30 ranked rows. Latency on the official 124-item subset: p50 raw 0.0683 s, p95 raw 0.1021 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.2867 s, p95 0.3543 s. Raw sha256 e13fb2c75d4c182366e26b5205fe27f2deabaefdd80602468eb8912ed08807d9; scored copy sha256 96a345a3859577fb5ff28b1a1009e6fe1620a2a0634fcb340ec711435d3fc14c.", "imajev-2b: Imajev-2B (mohit67890, Qwen3.5-2B LoRA). RE-MEASURED on this draw in the re-measure campaign (Batch A; Florian 16 Oct 2026 \"all evaluated models online\"), measured 16 Oct 2026 under custody #20150 (one self-hosted use) with GPU per deputy #20098, after an independent pre-dispatch review and with an independent post-run score review. No paid order and no author contact. 1,485 of 1,500 answered (HTTP 200: 1,485, HTTP 422: 15). 15 HTTP 422 refusals on the long items, scored as wrong answers. The registered token basis is the SUM of the four option-order passes. INPUT-BASIS SENSITIVITY (footnote only): on the server-reported basis the cost would be USD 0.010618 per 1,000 decisions; the registered basis is the 4-pass sum. The dependency set was reconstructed (the old freeze is lost). SPEED: measured on one rented RTX 5090 32 GB, not the RTX PRO 6000 / H100 of the earlier runs, so its Speed is as measured on this card and not comparable to the earlier measurement; probabilities may differ in the last digits. Evaluated pin mohit67890/imajev-2b@66e4b8c808bae45ca98bed5edd2de6323bdb0576 (code: https://github.com/mohit67890/imajev @ 4469df3c61fa643deeeff60a64aeffbdf34a48b7 (author playground server, torch backend, --rotations 4, shipped calibration.json)). LICENCE: Apache-2.0 (adapter; base Apache-2.0). Disclosures: (1) INPUT BASIS: the cost basis is the SUM of every pass the system computes (4 cyclic option orders x every question field, booleans 3), counted offline with the author's own prompt code and tokenizer (measurement receipts on file), the precedent of an earlier rotation recount; the server's own usage.input_tokens is the longest single pass (796,317 over the answered items, 1/3.78 of the sum) and is reported only as a sensitivity. (2) Output tokens 0 under the 9 Oct 2026 rule: native option logits, nothing generated (verified in the author's torch path). (3) The 15 long items are refused with HTTP 422 (request validation, state > 32,768 bytes). (4) The dependency set was reconstructed 'as of 24 Sep 2026' because the old freeze is lost (UNVERIFIED). (5) The model card quotes JevBench ranks of its own; they are not placements from this release. (6) Probabilities may differ in the last digits from the earlier runs (different card). The 15 long items of this draw are refused by the system and count as wrong in every axis (full 1,500 denominator). Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's own scripts/playground/server.py --backend torch, 4 option orders, 4,096-token request limit. ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-2B -> deepinfra:Qwen/Qwen3.5-2B USD 0.02/M input, USD 0.1/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens: input = 4-pass sum counted offline (see disclosures), output 0. USD 0.040140227 per 1,000 decisions, 0.621x the official cost cap USD 0.064595, inside the official cost-capped views. Measured Intelligence 21.161, Calibration 86.777, Capability 53.969, Speed 81.553, Cost axis 51.893; composite A 7.933, B 6.510, C 5.509; on this draw rank 24 (A), 24 (B), 26 (C) and 25 by Capability, of 30 ranked rows. Latency on the official 124-item subset: p50 raw 0.2894 s, p95 raw 0.4047 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.7289 s, p95 0.9595 s. Raw sha256 39022d0cb2fad4ce56c5cac0960996a3c1bc4ab6264ceb7b28857092928548f5; scored copy sha256 6259931d6f68ff5d75091f8c24e101c2d1b8fc48dbd0cc086a904c48563ea4da.", "imajev-9b: Imajev-9B (mohit67890, Qwen3.5-9B LoRA). RE-MEASURED on this draw in the re-measure campaign (Batch A; Florian 16 Oct 2026 \"all evaluated models online\"), measured 16 Oct 2026 under custody #20150 (one self-hosted use) with GPU per deputy #20098, after an independent pre-dispatch review and with an independent post-run score review. No paid order and no author contact. 1,485 of 1,500 answered (HTTP 200: 1,485, HTTP 422: 15). 15 HTTP 422 refusals on the long items (the author's server caps the request state), scored as wrong answers. OVER THE OFFICIAL COST CAP, DISCLOSED: the registered token basis is the SUM of the four option-order passes the system computes per question, which puts it at 3.11x the cap; it is ranked and outside the official cost-capped views. INPUT-BASIS SENSITIVITY (footnote only, never this row's cost): with the server-reported input of the longest single pass (796,317 tokens) instead of the registered 4-pass sum, the cost would be USD 0.053088 per 1,000 decisions (0.822x the cap, inside it) and composite A 27.969 in a separate what-if scoring of the same raws; it would enter no top five either way. The dependency set was reconstructed (the old freeze is lost). SPEED: measured on one rented RTX 5090 32 GB, not the RTX PRO 6000 / H100 of the earlier runs, so its Speed is as measured on this card and not comparable to the earlier measurement; probabilities may differ in the last digits. Evaluated pin mohit67890/imajev-9b@7301600591665b941d324b1a297e097b56724f9b (code: https://github.com/mohit67890/imajev @ 4469df3c61fa643deeeff60a64aeffbdf34a48b7 (author playground server, torch backend, --rotations 4, shipped calibration.json)). LICENCE: Apache-2.0 (adapter; base Apache-2.0). Disclosures: (1) INPUT BASIS: the cost basis is the SUM of every pass the system computes (4 cyclic option orders x every question field, booleans 3), counted offline with the author's own prompt code and tokenizer (measurement receipts on file), the precedent of an earlier rotation recount; the server's own usage.input_tokens is the longest single pass (796,317 over the answered items, 1/3.78 of the sum) and is reported only as a sensitivity. (2) Output tokens 0 under the 9 Oct 2026 rule: native option logits, nothing generated (verified in the author's torch path). (3) The 15 long items are refused with HTTP 422 (request validation, state > 32,768 bytes). (4) The dependency set was reconstructed 'as of 24 Sep 2026' because the old freeze is lost (UNVERIFIED). (5) The model card quotes JevBench ranks of its own; they are not placements from this release. (6) Probabilities may differ in the last digits from the earlier runs (different card). The 15 long items of this draw are refused by the system and count as wrong in every axis (full 1,500 denominator). Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's own scripts/playground/server.py --backend torch, 4 option orders, 4,096-token request limit. ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-9B -> qwen/qwen3.5-9b USD 0.09999999999999999/M input, USD 0.15/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens: input = 4-pass sum counted offline (see disclosures), output 0. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.200701 per 1,000 decisions is 3.11x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 precedent) and falls outside the official-caps board views by construction. USD 0.200701133 per 1,000 decisions, 3.107x the official cost cap USD 0.064595, OVER the official cap, so the row is ranked but falls outside the official cost-capped views (disclosed in its cost basis). Measured Intelligence 36.733, Calibration 88.109, Capability 62.421, Speed 79.488, Cost axis 30.924; composite A 9.891, B 9.323, C 6.868; on this draw rank 22 (A), 22 (B), 22 (C) and 18 by Capability, of 30 ranked rows. Latency on the official 124-item subset: p50 raw 0.3831 s, p95 raw 0.5389 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.9163 s, p95 1.2278 s. Raw sha256 ebf9ea8cab0bbe34973080663f0f4a8851fcd310f35a1345e1ed269e1906c4b5; scored copy sha256 90edc9548e818ddc6eb645f0e73f1446355b33b95e764462a966ef3d47b3d92e.", "kettei-chub: kettei-chub (Sqwish), the middle kettei size, hosted API. Paid public fast-lane order 6e10efe6, measured 16 Oct 2026 under custody #20263 (one hosted-API use) and deputy #20305/#20306, with the customer's own API key; the full-set exposure gate logged all 1,200 sealed items to provider `sqwish` before dispatch. 1,485 of 1,500 answered (HTTP 200: 1,485); 15 HTTP 422 context_too_long refusals (the 15 long items exceed the documented 65,536-token input limit), scored as wrong answers in the full 1,500-item denominator and counted as zero tokens. Every row names kettei-chub as the answering model: no fallback size answered. Latency is raw from Sandy with no API adjustment. PRICED from Sqwish's public launch list price (USD 0.04/M input, no output billed; saved copy sha256 698a29e5434a5e30146c6c91c118b3aec5f5f6425cb58cec34682ed640d7895c) under interpretation I-1 through the v1.6.24 re-freeze of the pricing module (sha256 3948e3f1..., predecessor 533d80f0...). COST-CEILING SENSITIVITY (footnote only, never this row's cost): the unchanged runner wrote a conservative ceiling of 65,536 input tokens (the model's documented input limit) into each of the 15 refused rows; charged, it would make the figure USD 0.043564 per 1,000 decisions (0.674x the official cost cap). Refused requests count as zero tokens on this draw, as d1-ultra's identical 15 refusals do, so the scoring copy drops that one field on those 15 rows only. Vendor-hosted api.sqwish.ai/v1/systemone API, model `kettei-chub` (Sqwish docs: kettei-chub v1.0, released 14 Oct 2026), through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `sqwish` before the first request (custody #20263, deputy #20305/#20306). Native probabilities on all three question types. The runner recorded kettei-chub as the answering model on all 1,500 rows (the adapter reads the response's model field), so no row was answered by a fallback size. 15 of 1,500 items exceed the model's documented 65,536-token input limit (HTTP 422 context_too_long) and are scored as failures. Paid public fast-lane order 6e10efe6, run with the customer's own API key; hosting implementation and deployed weights were not independently inspected. Sqwish public price list, USD 0.04/M input, output not billed (docs.sqwish.ai/guides/pricing, fetched 16 Oct 2026 16:17Z, saved copy sha256 698a29e5...): kettei chub at USD 0.04 per million input tokens for decisions; \"The models generate no output tokens, and none are billed\"; \"standard list prices, the same since launch on 14 October 2026\". Counted as the manufacturer's standard launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops, SubQ Snap, d1-ultra, Sage 1.4 and Nimble), through a scoped re-freeze of the frozen pricing module that adds the two kettei keys and nothing else, with a zero-difference proof for every other registry entry. The once-per-inbox USD 5.00 starter credit is a trial credit and is not used. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (650,605 input tokens over the 1,485 answered items, 0 output tokens), divided by the full 1,500-row basis the scorer uses; the 15 items refused with HTTP 422 count as zero tokens and as wrong answers, as every refused request on this draw does (d1-ultra's identical 15 refusals included). The unchanged runner also wrote a conservative cost ceiling into those 15 refused rows (65,536 input tokens each, the model's input limit); the scoring copy drops that one field on those 15 rows only. With the ceiling charged, the figure would be USD 0.043564 per 1,000 decisions (0.674x the official cost cap); that is a sensitivity, not this row's cost. Not a bill and not a quote. Provider-reported usage: 650,605 input tokens over the 1,485 answered items and 0 output tokens. USD 0.017349467 per 1,000 decisions, 0.269x the official cost cap USD 0.064595, inside the official cost-capped views. Measured Intelligence 62.183, Calibration 84.103, Capability 73.143, Speed 95.092, Cost axis 62.821; composite A 73.521, B 70.934, C 73.521; on this draw rank 3 (A), 3 (B), 3 (C) and 11 by Capability, of 30 ranked rows. Latency on the official 124-item subset: p50 raw 0.1667 s, p95 raw 0.1858 s, without API adjustment. Hosting implementation and deployed weights were not independently inspected. Raw sha256 a6f5eaac7938b9cc59e7a397cfa28037160c3f6ce538c1ef0734f1d956c2be52; scored copy sha256 c6ce25fe51576843951a2209ac6232be4dd36ed6500a540fbc1d39b4272a31e8.", "kettei-fat: kettei-fat (Sqwish), the largest kettei size, hosted API. Paid public fast-lane order a87f712c, measured 16 Oct 2026 under custody #20263 (one hosted-API use) and deputy #20305/#20306, with the customer's own API key; the full-set exposure gate logged all 1,200 sealed items to provider `sqwish` before dispatch. 1,485 of 1,500 answered (HTTP 200: 1,485); 15 HTTP 422 context_too_long refusals (the 15 long items exceed the documented 65,536-token input limit), scored as wrong answers in the full 1,500-item denominator and counted as zero tokens. Every row names kettei-fat as the answering model: no fallback size answered. Latency is raw from Sandy with no API adjustment. PRICED from Sqwish's public launch list price (USD 0.06/M input, no output billed; saved copy sha256 698a29e5434a5e30146c6c91c118b3aec5f5f6425cb58cec34682ed640d7895c) under interpretation I-1 through the v1.6.24 re-freeze of the pricing module (sha256 3948e3f1..., predecessor 533d80f0...). COST-CEILING SENSITIVITY (footnote only, never this row's cost): the unchanged runner wrote a conservative ceiling of 65,536 input tokens (the model's documented input limit) into each of the 15 refused rows; charged, it would make the figure USD 0.065346 per 1,000 decisions (1.012x the official cost cap). Refused requests count as zero tokens on this draw, as d1-ultra's identical 15 refusals do, so the scoring copy drops that one field on those 15 rows only. Vendor-hosted api.sqwish.ai/v1/systemone API, model `kettei-fat` (Sqwish docs: kettei-fat v1.0, released 14 Oct 2026), through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `sqwish` before the first request (custody #20263, deputy #20305/#20306). Native probabilities on all three question types. The runner recorded kettei-fat as the answering model on all 1,500 rows (the adapter reads the response's model field), so no row was answered by a fallback size. 15 of 1,500 items exceed the model's documented 65,536-token input limit (HTTP 422 context_too_long) and are scored as failures. Paid public fast-lane order a87f712c, run with the customer's own API key; hosting implementation and deployed weights were not independently inspected. Sqwish public price list, USD 0.06/M input, output not billed (docs.sqwish.ai/guides/pricing, fetched 16 Oct 2026 16:17Z, saved copy sha256 698a29e5...): kettei fat at USD 0.06 per million input tokens for decisions; \"The models generate no output tokens, and none are billed\"; \"standard list prices, the same since launch on 14 October 2026\". Counted as the manufacturer's standard launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops, SubQ Snap, d1-ultra, Sage 1.4 and Nimble), through a scoped re-freeze of the frozen pricing module that adds the two kettei keys and nothing else, with a zero-difference proof for every other registry entry. The once-per-inbox USD 5.00 starter credit is a trial credit and is not used. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (650,605 input tokens over the 1,485 answered items, 0 output tokens), divided by the full 1,500-row basis the scorer uses; the 15 items refused with HTTP 422 count as zero tokens and as wrong answers, as every refused request on this draw does (d1-ultra's identical 15 refusals included). The unchanged runner also wrote a conservative cost ceiling into those 15 refused rows (65,536 input tokens each, the model's input limit); the scoring copy drops that one field on those 15 rows only. With the ceiling charged, the figure would be USD 0.065346 per 1,000 decisions (1.012x the official cost cap); that is a sensitivity, not this row's cost. Not a bill and not a quote. Provider-reported usage: 650,605 input tokens over the 1,485 answered items and 0 output tokens. USD 0.026024200 per 1,000 decisions, 0.403x the official cost cap USD 0.064595, inside the official cost-capped views. Measured Intelligence 79.839, Calibration 92.781, Capability 86.310, Speed 93.191, Cost axis 57.539; composite A 77.801, B 78.200, C 77.801; on this draw rank 1 (A), 1 (B), 1 (C) and 1 by Capability, of 30 ranked rows. Latency on the official 124-item subset: p50 raw 0.2001 s, p95 raw 0.2397 s, without API adjustment. Hosting implementation and deployed weights were not independently inspected. Raw sha256 a01bf93701251d3f085440d6aa327821d80613c865c0d717a397ca2a153e924d; scored copy sha256 3d2949448dbd7db8f422ea1ca8d3535174ce4b69901cce215925361e8f69de08.", "noma: Noma (Blackdrome AI Labs, Qwen3.5-4B-Base derivative). RE-MEASURED on this draw in the re-measure campaign (Batch A; Florian 16 Oct 2026 \"all evaluated models online\"), measured 16 Oct 2026 under custody #20150 (one self-hosted use) with GPU per deputy #20098, after an independent pre-dispatch review and with an independent post-run score review. No paid order and no author contact. It supersedes the historical v1.5 measurement of noma shown on the v1.5 supplement page (/jev-models/v1.5.7; 1,624-item retired protocol, never ranked on the v1.6 boards); that page stays as the record of the old protocol. 1,500 of 1,500 answered (HTTP 200: 1,500). No refusal: the author's serializer TRUNCATES long states to a 3,584-token prefix budget and answers every item, so the 15 long items are answered on a truncated state. SPEED: measured on one rented RTX 5090 32 GB, not the RTX PRO 6000 / H100 of the earlier runs, so its Speed is as measured on this card and not comparable to the earlier measurement; probabilities may differ in the last digits. Evaluated pin BlackdromeAILabs/noma@fac0429688a137ce451a27ad34d05e9fd47fb624 (code: blackdromeai-labs/noma @ aad38472bab4226ba99f5ddbe32b424d843f09a6 (noma serve, fast path on)). LICENCE: MPL-2.0 (weights and code). The author's serializer truncates long states to a 3,584-token prefix budget (head 2/3 + tail 1/3) and answers every item, so all 1,500 items are answered (no refusal). Weights and code MPL-2.0. The author's own identity check scored 176/231 on the public items. causal-conv1d has no wheel for this stack, so Speed is a lower bound. noma is installed non-editable from the pinned source. Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's own `noma serve` with the default fast path (128-token buckets + CUDA graphs). ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-4B-Base -> deepinfra:Qwen/Qwen3.5-4B USD 0.030000000000000002/M input, USD 0.15/M output (frozen at the 25 Sep cut-off, I-3). Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens as reported by the author's server (output_tokens 0, nothing generated). USD 0.019615120 per 1,000 decisions, 0.304x the official cost cap USD 0.064595, inside the official cost-capped views. Measured Intelligence 25.051, Calibration 83.637, Capability 54.344, Speed 93.342, Cost axis 61.222; composite A 12.723, B 10.562, C 8.835; on this draw rank 21 (A), 21 (B), 21 (C) and 24 by Capability, of 30 ranked rows. Latency on the official 124-item subset: p50 raw 0.0255 s, p95 raw 0.0402 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.2010 s, p95 0.2304 s. Raw sha256 958af0bc05f19bab24b4975813f0b2b40209b48f0981e03baf00370ce857c05c; scored copy sha256 664b7db62501f00141742b24c3ec593dcd9ec68f0cbf36acce3e3162a2690e2d.", "v1.6.25 ADDS 4 ROWS (Celeris-1 Decision (Celeris) (celeris-1-decision), Jebadiah 9B v2.1 (jebadiah-9b-v2-1), StartLux-Decision v1.1 27B AutoThink (startlux-decision-v1.1-27b-autothink), StartLux-Decision v1.1 27B Instant (startlux-decision-v1.1-27b-instant)) to the 34 full-draw rows of v1.6.24 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so every carried row keeps identical point scores, axes, capability, cost and speed objects, and the listed-not-ranked rows (reflex-instinct-0.6b, nemotron-deciffusion-8b-v2, trio-spark-v1.3, bongard-mini) and the unranked A2 measurement keep their wording. Jebadiah 9B v2.1 (jebadiah-9b-v2-1) is a new add-request row (self-hosted on the author's own server), RANKED and priced from its frozen base-model reference. StartLux-Decision v1.1 27B AutoThink (startlux-decision-v1.1-27b-autothink), StartLux-Decision v1.1 27B Instant (startlux-decision-v1.1-27b-instant) were added on Florian's own request (self-hosted on the author's own server, weights CC BY-NC 4.0, non-commercial), priced from their frozen base-model reference and RANKED OVER THE OFFICIAL COST CAP (disclosed), so they fall outside the official cost-capped views by construction; JevBench had no StartLux row before. The PAID fast-lane row Celeris-1 Decision (Celeris) (celeris-1-decision) (Celeris, hosted API) is RANKED from Celeris's public list price through a scoped re-freeze of the pricing module. LONG ITEMS: 15 items of this draw have very long states. From the scored raws of this release, they are refused or fail outright (all 15, scored as wrong answers) for 21 rows: arsenelupin-v1.1, attest-31b-v1.0, bongard-mini, decision2-lux-9b, imajev-2b, imajev-9b, jebadiah-9b-v2-1, julia-1, kettei-chub, kettei-fat, liquid-d1-ultra, nev-2b, pjev-2b, purplemist-mini-1.0, respan-span-02, sage-1.4.0, standard_one_3b_sh, startlux-decision-v1.1-27b-autothink, startlux-decision-v1.1-27b-instant, trio-spark-v1.3, uniphore-4b-decider; phocinae-largha-150m-v1 fails 3 of them; every other row answers them under its own recipe (on a truncated or compacted state, or read whole), as the row notes say. What moved on carried rows, counted exactly. RANKS: amos-1-large A 27 -> 31; B 26 -> 30; C 25 -> 29; capability 7 -> 9, amos-1-medium A 18 -> 21; B 18 -> 21; C 18 -> 21; capability 16 -> 20, arsenelupin-v1.1 A 19 -> 22; B 19 -> 22; C 19 -> 22; capability 23 -> 27, attest-31b-v1.0 capability 12 -> 15, decision2-lux-9b A 15 -> 17; B 15 -> 17; C 15 -> 18; capability 19 -> 23, imajev-2b A 24 -> 28; B 24 -> 28; C 26 -> 30; capability 25 -> 29, imajev-9b A 22 -> 26; B 22 -> 26; C 22 -> 26; capability 18 -> 22, julia-1 A 30 -> 34; B 30 -> 34; C 30 -> 34; capability 30 -> 34, kettei-chub capability 11 -> 14, krino-2b A 23 -> 27; B 23 -> 27; C 23 -> 27; capability 28 -> 32, liquid-d1-ultra capability 4 -> 5, messier-one-v0.4 C 12 -> 13; capability 17 -> 21, metask_jev_rain_12b_trained capability 9 -> 11, metask_jev_rain_31b_trained capability 8 -> 10, metask_jev_rain_31b_trained_v3 capability 6 -> 8, nev-2b A 25 -> 29; B 27 -> 31; C 27 -> 31; capability 26 -> 30, nimble-v4 A 13 -> 14; B 14 -> 15; C 14 -> 15; capability 15 -> 18, nimble-v4-max A 14 -> 15; B 13 -> 14; C 13 -> 14; capability 5 -> 7, noma A 21 -> 24; B 21 -> 25; C 21 -> 25; capability 24 -> 28, phocinae-largha-150m-v1 A 29 -> 33; B 29 -> 33; C 29 -> 33; capability 29 -> 33, pjev-2b A 20 -> 23; B 20 -> 23; C 20 -> 24; capability 20 -> 24, purplemist-mini-1.0 A 28 -> 32; B 28 -> 32; C 28 -> 32; capability 27 -> 31, respan-span-02 A 12 -> 13; B 12 -> 13; C 11 -> 12; capability 13 -> 16, sage-1.4.0 capability 10 -> 12, standard_one_3b_sh A 17 -> 20; B 17 -> 20; C 17 -> 20; capability 22 -> 26, subq-snap-preview A 11 -> 12; B 11 -> 12; C 9 -> 10; capability 3 -> 4, torchcast-decision-4b-v1.1 C 10 -> 11; capability 14 -> 17, uniphore-4b-decider A 16 -> 18; B 16 -> 18; C 16 -> 19; capability 21 -> 25, vllm-sr-d3 A 26 -> 30; B 25 -> 29; C 24 -> 28. Each moved rank shifts only by the newly ranked rows above it. Rows listed but not ranked take no place in any order. CONFIDENCE INTERVALS: as in v1.6.17, v1.6.18, v1.6.21, v1.6.22 and v1.6.24, new keys sort into the key-sorted roster, and the scorer seeds each row's bootstrap with the shared seed 16 plus the row's position in that roster, so a carried row whose position changed gets a different interval while no point estimate moves. Carried rows whose intervals moved: decision2-lux-9b, imajev-2b, imajev-9b, julia-1, kettei-chub, kettei-fat, krino-2b, liquid-d1-ultra, messier-one-v0.4, metask_jev_rain_12b_trained, metask_jev_rain_31b_trained, metask_jev_rain_31b_trained_v3, nemotron-deciffusion-8b-v2, nev-2b, nimble-v4, nimble-v4-max, noma, phocinae-largha-150m-v1, pjev-2b, purplemist-mini-1.0, reflex-instinct-0.6b, respan-span-02, sage-1.4.0, standard_one_3b_sh, subq-snap-preview, torchcast-decision-4b-v1.1, trio-spark-v1.3, uniphore-4b-decider, vllm-sr-d3. DIAGNOSTICS: the API-basis median diagnostic is 8.5949815619838 (previous 8.162049591230426); carried v16.raw_on_A moves: standard_one_3b_sh: intelligence 28.8915 -> 29.0221, capability 52.9327 -> 52.9980. Equating offsets are I -0.0395 / C 2.7537 (previous I 0.0681 / C 2.5891); the equating pool changes from 23 to 26 members (jebadiah-9b-v2-1, startlux-decision-v1.1-27b-autothink, startlux-decision-v1.1-27b-instant joined). These diagnostics are not part of any score, rank or axis on this release: the hosted-API rows are full-set rows and are not equated, and the carried A2 measurement keeps its own equating and is not rescored. reflex-instinct-0.6b stays LISTED BUT NOT RANKED under #18761 and the unranked A2 measurement of v1.6.14 is carried; neither is changed.", "Board effect: merged-live-board top-five comparison before/after v1.6.25 over the mixed, open-weights and API boards, composites A/B/C and Capability with official caps, no caps, cost off + latency 2x (Florian settings) and official cost cap + no latency cap: 3 of 21 top-five lists change. mixed board (open weights + API), Capability, cost off + latency 2x (Florian's settings): StartLux-Decision v1.1 27B Instant (startlux-decision-v1.1-27b-instant) enters at #4; torchcast-decision-27b leaves the top five | open-weights board, Capability, no caps: StartLux-Decision v1.1 27B Instant (startlux-decision-v1.1-27b-instant) enters at #3; Jeff-1.0-Large leaves the top five | open-weights board, Capability, cost off + latency 2x (Florian's settings): StartLux-Decision v1.1 27B Instant (startlux-decision-v1.1-27b-instant) enters at #3; Jeff-1.0-Large leaves the top five. Top-five changes caused by: startlux-decision-v1.1-27b-instant (non-paid: startlux-decision-v1.1-27b-instant). Best placement of the new ranked rows on the merged board: celeris-1-decision #8 (API board, Capability, official cost + latency caps (default)); jebadiah-9b-v2-1 #36 (open-weights board, Capability, official cost + latency caps (default)); startlux-decision-v1.1-27b-autothink #7 (open-weights board, Capability, no caps); startlux-decision-v1.1-27b-instant #3 (open-weights board, Capability, no caps). Ranked rows: mixed 206 -> 210, open weights 169 -> 172, API 37 -> 38.", "Publication of v1.6.25: the add-request row on Florian's GO of 16 Oct 2026 12:50Z, a direct message in the laptop chat (\"I really want all of the models we have evaluated to be online (even if they land in #5) on both JevBench and ImageJevBench\"; DECISIONS.md 2026-10-16 12:50 UTC); the two StartLux rows on his own request of 16 Oct 2026 ~19:00Z (\"if we don't have it on jevbench, add it\"; Florian asks ledger A144 (board #20441)) and his GPU approval (the button \"Both rows (recommended)\" on the GPU request, card #17713; that card was a GPU request, not a top-five preview); the paid fast-lane row under his standing fast-lane rule (DECISIONS.md 29 Sep 2026 ~00:25 Berlin (Florian)). Florian selected \"Release (recommended)\" on Telegram card #17730 (2026-10-16T23:56:58Z), the top-five preview card with preview PNG sha256 1b397377c5c9cf6832f59db5d29d2d0a2d60236757bc717a4360fb8c64860b73.", "celeris-1-decision: Celeris-1 Decision (Celeris), hosted API. Paid public fast-lane order 70c4455b, measured 16 Oct 2026 under custody #20414 (one hosted-API use, under #19242), with the customer's own API key; the full-set exposure gate logged all 1,200 sealed items to provider `celeris` (no proxy) before dispatch. 1,500 of 1,500 answered (HTTP 200: 1,500), including the 15 long items, which fit its documented 131,072-token context window. Every row names celeris-1-decision as the answering model. Celeris returns every probability at two-decimal precision (332 returned probabilities are exactly 0); they are scored as returned. Latency is raw from Sandy with no API adjustment. PRICED from Celeris's public list price (USD 0.04/M input, output free; saved copies sha256 c3ed0fdc8cbb2bffc9743f73c8be33d3ad0ae12188f4f8fcf6f2586256a94f2b and ff4b658d61fbfb053109196c09480afce85a896b2c0661f41acb9506a26b65de, the docs page byte-identical to the copy saved on 14 Oct 2026) under interpretation I-1 through the v1.6.25 re-freeze of the pricing module (sha256 32556e54..., predecessor 3948e3f1...). The 14 Oct 2026 ruling had listed this model unpriced with no pricing change while no key was available (it was not measured then); it is now a paid order run with the customer's key. TOKEN COUNTING: Celeris counts input tokens its own way; per its docs, input_tokens covers everything the model reads for the request. On the same 1,500 items it reports 1,968,382 input tokens, 3.03x the 650,605 that Sqwish reports for kettei-fat, so its Cost axis reflects Celeris's own count at its own tariff. Vendor-hosted inference.celeris.ai/celeris-1-decision System One API (POST /v1/systemone), model `celeris-1-decision`, through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `celeris` (no proxy) before the first request (custody #20414 under #19242). Native probabilities on all three question types; Celeris returns every probability at two-decimal precision (1,500 of 1,500 rows; 332 returned probabilities are exactly 0), scored as returned. The runner recorded celeris-1-decision as the answering model on all 1,500 rows (the adapter reads the response's model field). All 1,500 items answered with HTTP 200, including the 15 long items, which fit the documented 131,072-token context window (largest reported input 81,719 tokens). Paid public fast-lane order 70c4455b, run with the customer's own API key; hosting implementation and deployed weights were not independently inspected. Celeris public price, USD 0.04/M input, output free (celeris.ai/pricing, fetched 16 Oct 2026 19:27Z, saved copy sha256 c3ed0fdc...; docs.celeris.ai/pricing rates table, saved copy sha256 ff4b658d..., byte-identical to the copy saved on 14 Oct 2026): \"Celeris-1 Decision $0.04 / M input tokens, Free output tokens\"; prepaid per-token billing with no subscription or commitment. Counted as the manufacturer's standard launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops, SubQ Snap, d1-ultra, Sage 1.4, Nimble and kettei), through a scoped re-freeze of the frozen pricing module that adds this one key and nothing else, with a zero-difference proof for every other registry entry. Workspace activation at Celeris is capacity-limited (an activation waitlist, as for SubQ Snap); free or promotional credit is not used. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (1,968,382 input tokens over the 1,500 answered items, 0 output tokens), divided by the full 1,500-row basis. Celeris counts input tokens its own way: per its docs, input_tokens \"covers everything the model reads for the request, which is more than your state and questions alone\"; on the same 1,500 items it reports about 3.0x the input tokens kettei-fat reports, so the cost reflects Celeris's own count. Not a bill and not a quote. Provider-reported usage: 1,968,382 input tokens over the 1,500 answered items and 0 output tokens. USD 0.052490187 per 1,000 decisions, 0.813x the official cost cap USD 0.064595, inside the official cost-capped views. Measured Intelligence 58.890, Calibration 88.355, Capability 73.622, Speed 86.972, Cost axis 48.398; composite A 61.988, B 60.494, C 59.716; on this draw rank 11 (A), 11 (B), 9 (C) and 13 by Capability, of 34 ranked rows. Latency on the official 124-item subset: p50 raw 0.4326 s, p95 raw 0.4642 s, without API adjustment. Hosting implementation and deployed weights were not independently inspected. Raw sha256 67104fd1ec5fb5cdc3a44f34b9b2e6df02c1696c7063585c14cd3736d349664d (the raw is the scored copy).", "jebadiah-9b-v2-1: Jebadiah 9B v2.1 (Frontier Infra, Qwen3.5-9B LoRA). NEW add-request row (GitHub issue #220), measured 16 Oct 2026 under custody #20323 (one self-hosted use) with GPU per deputy #20315, after an independent pre-score review (gate 3: the snapshot without its scripts/ folder is the submitted system) and an independent re-review of the run package; every pre-registered prediction matched (1,485 answered, the 15 refused items, 750,303 input and 1,485 output tokens). No paid order and no author contact. It is a different model from the published jebadiah-27b row (v1.6.1 headline draw), which stays as published. 1,485 of 1,500 answered (HTTP 200: 1,485, HTTP 422: 15); the 15 HTTP 422 refusals are the 15 long items (the author's 8,192-token prompt cap), scored as wrong answers in the full 1,500-item denominator. SPEED: measured on one rented RTX 5090 32 GB, not the author's RTX PRO 6000, so its Speed is a lower bound on this card. Evaluated pin frontier-infra/jebadiah-9b-v2-1@c5e49fab7ec8e9923e1f907d85911a53fb8e9c32 (code: getainode/jebadiah @ 74a0cd92965529e90a411532e9887b544c931b2d (the author's own standalone jebadiah-serve; bundled model_scripts ainode_prompt_verbatim.py 61e81179, jebadiah_prompt.py 35edd3fe, jebadiah_model.py 2c5c90dc)). LICENCE: Apache-2.0 (weights and base). Disclosures: (1) SERVING: the pinned model snapshot was downloaded WITHOUT its scripts/ directory, because v2.1 ships two newer copies (jebadiah_prompt.py c0e74561..., jebadiah_model.py 97567c45...) than the author's own server bundles (35edd3fe..., 2c5c90dc...), which makes the server's strict byte-for-byte prompt-contract check refuse to start. The server therefore rendered and scored with its own bundled copies, while all four substantive contract checks stayed armed and passed at load time (/health.prompt_contract.ok true, scripts_match {}, problems []): renderer source hash d2660ebe..., chat-template hash a4aee8af..., 68 single-token labels, thinking off. The only differences are training-side additions - a re-truncation loop for over-length prompts, which this server refuses instead of cutting, and an autocast option that is off by default and never used by the server - and two independent reviews confirmed they cannot change any answered item (48 renderer cases identical, option_logits bitwise equal, the real engine raising ContractError with scripts/ present and on each mutated contract field). --allow-contract-mismatch was NOT used. (2) 15 of 1,500 items are refused with HTTP 422 at the author's default 8,192-token prompt cap: by their design an over-long prompt is refused, never cut. Those items render to 76,358-80,516 tokens; the longest answered prompt is 1,858 tokens. All 15 refusals, and their exact token counts, were predicted offline before the run. (3) usage.output_tokens is 1 per answered request (the server returns len(answers)), identical to the live jebadiah-27b row; it is what the cost axis counts. (4) THE AUTHOR'S CARD REPORTS A HIGHER NUMBER UNDER A DIFFERENT RECIPE: a public-only Decision Index of 51.53 for these same weights under their 'T1e' recipe (the payload repeated inside the prompt plus a confidence-gated reverse-order second read, about 3.8x the input tokens and 2.4x the median latency), against 47.20 for the standard read shipped in the repository. The standard read is the submitted system and is what this row measures. (5) Calibration uses the repository's own temperatures.json (applied_target train, score_fit hard): choice 1.0467 and noul 1.0284 are the train fits, score 0.6805 is the hard-label fit; the adapter sends no calibration field. (6) The author's own standalone script truncates over-length states at 2,048 tokens while the served route refuses them. (7) The server pin and GitHub HEAD b1d5a273 are identical in the serving path. Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA GeForce RTX 5090 32 GB (sm_120, driver 595.71.05), Romania (EU) - different hardware from the RTX PRO 6000 the author measures on, so Speed is a lower bound on this card and not comparable to their 49.9 ms median. The author's OWN standalone server `jebadiah-serve --model --device cuda` from getainode/jebadiah @ 74a0cd92965529e90a411532e9887b544c931b2d (the commit behind the published jebadiah-27b row, independently code-reviewed PASS), installed from the author's own frozen uv.lock (`uv sync --frozen --no-dev --extra cuda`, torch 2.14.0+cu130, transformers 5.17.0, flash-linear-attention 0.5.2), all author defaults (dtype auto = bf16, batch-size 8, --max-prompt-tokens 8192, strict prompt-contract check ON, the repo's own temperatures.json, fp32 candidate logits), JEBADIAH_API_KEY unset. Isolation: server started inside a network namespace with loopback only (unshare -n), the cut proven before the sealed input was staged and re-proven immediately before the first sealed request, synthetic smoke gate passed in between; runner unchanged (run_v15.py, 0 resumes, rc 0); one use of input-selfhosted-1500 (custody #20323, deputy #20315). Pod removed by exact id and verified gone four ways. Published Speed uses the standing x2 + 0.15 s self-hosted adjustment; raw p50 0.0502 s / p95 0.0837 s. ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-9B -> qwen/qwen3.5-9b USD 0.09999999999999999/M input, USD 0.15/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 533d80f0...d960e5, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens as reported by the author's own server (750,303 input over the 1,485 answered items, output_tokens 1 per answered request). USD 0.050168700 per 1,000 decisions, 0.777x the official cost cap USD 0.064595, inside the official cost-capped views. Measured Intelligence 37.150, Calibration 89.677, Capability 63.413, Speed 91.400, Cost axis 48.987; composite A 30.532, B 27.502, C 21.203; on this draw rank 16 (A), 16 (B), 17 (C) and 19 by Capability, of 34 ranked rows. Latency on the official 124-item subset: p50 raw 0.0431 s, p95 raw 0.0783 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.2363 s, p95 0.3066 s. Raw sha256 61ae20039465eaeb34784d41a6d8ebfac50891f59d5d15aa3d797443a55eb490 (the raw is the scored copy).", "startlux-decision-v1.1-27b-autothink: StartLux-Decision v1.1 27B AutoThink (StartLux Labs, Qwen3.8-27B fine-tune, gate 0.6). NEW row added on Florian's own request (16 Oct 2026), measured 16 Oct 2026 under custody #20395 (one self-hosted use of this draw per row; both rows on one rented H200 pod, GPU approved by Florian) after an independent code review of the author package, an independent pre-run review and an independent pre-score review; every pre-registered prediction matched (1,485 answered, the 15 refused items, every per-item input-token count). No paid order and no author contact. CONTEXT: JevBench had no StartLux row before; the published torchcast-decision-27b row is a fine-tune of StartLux-Decision-27B v1. AutoThink is measured at the author's SERVER DEFAULT gate 0.6; the model card's published AutoThink results use gate 0.9, which was NOT measured. 1,485 of 1,500 answered (HTTP 200: 1,485, HTTP 422: 15); the 15 HTTP 422 refusals are the 15 long items (over the 65,536-token context of vLLM 0.19.1), scored as wrong answers in the full 1,500-item denominator. LICENCE (NC): weights CC BY-NC 4.0, non-commercial; commercial use needs a separate licence from StartLux Labs; listing a measured score is evaluation use. Evaluated pin startlux-models/StartLux-Decision-v1.1-27B-AutoThink@dd62d5cd84a483d75bdd548880bbf08bdd3f67a4 (code: StartLuxLabs/StartLux-Decision @ cb1d11f1ac9e9b43b95ef3a6468fa29b607c40fd (Apache-2.0; the author's own `startlux-decide serve --launch-vllm`, startlux-decide 1.1.0; hash-locked vLLM 0.19.1 stack, lock sha256 4c00e831...)). Disclosures: (L) LICENCE: the weights are CC BY-NC 4.0 (non-commercial; commercial use needs a separate licence from StartLux Labs); the inference package is Apache-2.0. (S) SERVING: the author's own documented command `startlux-decide serve --model --launch-vllm --gpus 0 --port 8090` on vLLM 0.19.1 (the version of the author's published scores; launcher context 65,536 tokens, prefix caching and chunked prefill off), all author defaults, bound to 127.0.0.1 inside a network namespace with the network cut and proven before any item. The server's default `none_check` is on: every choice question carries an internal extra 'None of the above' option; returned probabilities cover only the listed options (renormalised) and the answer is always a listed option. (R) 15 of 1,500 items are refused with HTTP 422 because their prompts (76,353-80,511 tokens) exceed the 65,536-token context of vLLM 0.19.1; the card says prompts over 65,536 tokens need vLLM 0.31. Refused, never cut; all 15 ids and every per-item input-token count were predicted offline before the run and matched exactly. (U) usage.input_tokens counts the first read-out prompt of each question only. (H) Hardware: one NVIDIA H200 141 GB (the author's own card type), Lium pod in Finland; the standing self-hosted latency adjustment (x2 + 0.15 s) applies as for every self-hosted row. (D) The Decision Index figures on the card (72.83 Instant / 73.37 AutoThink, public subset) are the author's own runs and, by their statement, reflect model selection; JevBench measures its own sealed set. (A) AUTOTHINK: measured at the author's SERVER DEFAULT gate 0.6 (think_mode auto, thought <= 4,096 tokens, temperature 0.6 / top-p 0.95 / top-k 20, final answer 0.25 x first read-out + 0.75 x second read-out). The card's published AutoThink results use gate 0.9; that setting was NOT measured (it was not affordable within this run's GPU cap: about 273 thought tokens per question). 279 of 1,485 answered questions (18.8%) thought, 72,063 thought tokens in total (48.5 per answered question); usage.output_tokens counts exactly these thought tokens and the cost axis prices them at the base output rate. Thoughts are sampled (seeded from the prompt hash), so a thinking question can answer differently on another run. Latency includes the thought time (raw p95 5.4 s, max 66.9 s). Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA H200 141 GB (sm_90, driver 580.173.02), Finland (EU). The author's own server on vLLM 0.19.1, offline, loopback, one request at a time. ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: base Qwen/Qwen3.8-27B -> qwen/qwen3.8-27b USD 0.42/M input, USD 3.0/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 3948e3f1...ea4d0, the module this row was first priced with (the later re-freeze only adds a hosted-API launch-list key), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). The model cards do not name the Qwen version; Qwen/Qwen3.8-27B is the base the Decision Index metadata gives for the sibling Instant variant (AutoThink itself is not on the Decision Index), shares the same NOTICE, consistent with the NOTICE (modified Alibaba Cloud Apache-2.0 model) and with the config (Qwen3_5ForConditionalGeneration, 64 layers, hidden 5120), the same reference as the live torchcast-decision-27b row (a fine-tune of StartLux-Decision-27B v1). OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.350763 per 1,000 decisions is 5.43x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 and v1.6.24 imajev-9b precedent) and falls outside the official-caps board views by construction. USD 0.350763200 per 1,000 decisions, 5.430x the official cost cap USD 0.064595, OVER the official cap, so the row is ranked but falls outside the official cost-capped views (disclosed in its cost basis). Measured Intelligence 68.964, Calibration 91.731, Capability 80.347, Speed 75.052, Cost axis 23.650; composite A 11.047, B 11.712, C 11.047; on this draw rank 25 (A), 24 (B), 23 (C) and 6 by Capability, of 34 ranked rows. Latency on the official 124-item subset: p50 raw 0.0556 s, p95 raw 5.9052 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.2612 s, p95 11.9605 s. Raw sha256 8fc7145c01b6854821d0458e56cbf4b67941b3cfa25b732efc317e448695e654 (the raw is the scored copy).", "startlux-decision-v1.1-27b-instant: StartLux-Decision v1.1 27B Instant (StartLux Labs, Qwen3.8-27B fine-tune). NEW row added on Florian's own request (16 Oct 2026), measured 16 Oct 2026 under custody #20395 (one self-hosted use of this draw per row; both rows on one rented H200 pod, GPU approved by Florian) after an independent code review of the author package, an independent pre-run review and an independent pre-score review; every pre-registered prediction matched (1,485 answered, the 15 refused items, every per-item input-token count). No paid order and no author contact. CONTEXT: JevBench had no StartLux row before; the published torchcast-decision-27b row is a fine-tune of StartLux-Decision-27B v1. Instant: one read-out per question, no thinking. 1,485 of 1,500 answered (HTTP 200: 1,485, HTTP 422: 15); the 15 HTTP 422 refusals are the 15 long items (over the 65,536-token context of vLLM 0.19.1), scored as wrong answers in the full 1,500-item denominator. LICENCE (NC): weights CC BY-NC 4.0, non-commercial; commercial use needs a separate licence from StartLux Labs; listing a measured score is evaluation use. Evaluated pin startlux-models/StartLux-Decision-v1.1-27B-Instant@32e7d35d74db615c4a059181efbf21c5421f3561 (code: StartLuxLabs/StartLux-Decision @ cb1d11f1ac9e9b43b95ef3a6468fa29b607c40fd (Apache-2.0; the author's own `startlux-decide serve --launch-vllm`, startlux-decide 1.1.0; hash-locked vLLM 0.19.1 stack, lock sha256 4c00e831...)). Disclosures: (L) LICENCE: the weights are CC BY-NC 4.0 (non-commercial; commercial use needs a separate licence from StartLux Labs); the inference package is Apache-2.0. (S) SERVING: the author's own documented command `startlux-decide serve --model --launch-vllm --gpus 0 --port 8090` on vLLM 0.19.1 (the version of the author's published scores; launcher context 65,536 tokens, prefix caching and chunked prefill off), all author defaults, bound to 127.0.0.1 inside a network namespace with the network cut and proven before any item. The server's default `none_check` is on: every choice question carries an internal extra 'None of the above' option; returned probabilities cover only the listed options (renormalised) and the answer is always a listed option. (R) 15 of 1,500 items are refused with HTTP 422 because their prompts (76,353-80,511 tokens) exceed the 65,536-token context of vLLM 0.19.1; the card says prompts over 65,536 tokens need vLLM 0.31. Refused, never cut; all 15 ids and every per-item input-token count were predicted offline before the run and matched exactly. (U) usage.input_tokens counts the first read-out prompt of each question only. (H) Hardware: one NVIDIA H200 141 GB (the author's own card type), Lium pod in Finland; the standing self-hosted latency adjustment (x2 + 0.15 s) applies as for every self-hosted row. (D) The Decision Index figures on the card (72.83 Instant / 73.37 AutoThink, public subset) are the author's own runs and, by their statement, reflect model selection; JevBench measures its own sealed set. (I) INSTANT: one read-out per question, no thinking (think_mode never, usage.output_tokens 0). The author's vLLM plugin makes the prompt attention prefix-bidirectional; the run verified 'prefix-bidirectional prompt attention ON' in the vLLM log. This is the variant listed on the Jev Decision Index (engine startlux-decision-v1.1-27b-instant); that listing is the author-side run, JevBench measures its own sealed set. Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA H200 141 GB (sm_90, driver 580.173.02), Finland (EU). The author's own server on vLLM 0.19.1, offline, loopback, one request at a time. ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: base Qwen/Qwen3.8-27B -> qwen/qwen3.8-27b USD 0.42/M input, USD 3.0/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 3948e3f1...ea4d0, the module this row was first priced with (the later re-freeze only adds a hosted-API launch-list key), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). The model cards do not name the Qwen version; Qwen/Qwen3.8-27B is the base the Decision Index metadata gives for this model, consistent with the NOTICE (modified Alibaba Cloud Apache-2.0 model) and with the config (Qwen3_5ForConditionalGeneration, 64 layers, hidden 5120), the same reference as the live torchcast-decision-27b row (a fine-tune of StartLux-Decision-27B v1). OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.206637 per 1,000 decisions is 3.20x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 and v1.6.24 imajev-9b precedent) and falls outside the official-caps board views by construction. USD 0.206637200 per 1,000 decisions, 3.199x the official cost cap USD 0.064595, OVER the official cap, so the row is ranked but falls outside the official cost-capped views (disclosed in its cost basis). Measured Intelligence 73.194, Calibration 91.921, Capability 82.558, Speed 90.623, Cost axis 30.544; composite A 21.850, B 22.760, C 21.850; on this draw rank 19 (A), 19 (B), 16 (C) and 3 by Capability, of 34 ranked rows. Latency on the official 124-item subset: p50 raw 0.0551 s, p95 raw 0.0914 s; published with the standard self-hosted x2 + 0.15 s adjustment (assumption, not measured): p50 0.2603 s, p95 0.3329 s. Raw sha256 18930e0fc47504edc6f2b699cd754d794b2ac0805e1b90d4fa15ff1e2d677a4b (the raw is the scored copy)." ], "n_ranked": 34, "not_measured": [], "noul_method": { "applied": "O1S", "native_method_source_sha256": "119e59486ab3300b972fbcbadb0fe1dceceb124287607375aa61c66063c607d0" }, "options": { "A": { "intelligence_floor": 50.0, "weights": { "calibration": 25, "cost": 25, "intelligence": 25, "speed": 25 } }, "B": { "intelligence_floor": 50.0, "weights": { "calibration": 20, "cost": 20, "intelligence": 40, "speed": 20 } }, "C": { "intelligence_floor": 60.0, "weights": { "calibration": 25, "cost": 25, "intelligence": 25, "speed": 25 } } }, "protocol": "jevbench::v1.6", "provisional": false, "revision": "v1.6.25", "roster_count": 39, "run_kind": "regular-queue", "sample": { "open": 300, "published_open": 300, "sealed": 1200, "total": 1500 }, "sealed_share_of_intelligence": 0.5, "source_note": "39 rows on the regular draw v1.6-regular-20261012 (seeded rotation-ledger draw of 1,200 sealed items + the 300-item epoch public set, drawn 2026-10-12T01:32:59Z): the 34 full-draw rows of v1.6.24, the four rows added in v1.6.25 (celeris-1-decision, jebadiah-9b-v2-1, startlux-decision-v1.1-27b-autothink, startlux-decision-v1.1-27b-instant), and the unranked A2 measurement carried from v1.6.14 unchanged. ONE score_v16.py invocation over the 38 full-draw rows, unmodified frozen path (sha256 02132271...), JEV_NOUL_METHOD=O1S, B=1000, seed 16, --g-med-fixed 9.839526770355711, the median frozen at v1.6.10, so every carried row keeps every point score; the 34 carried raw scoring copies are byte-identical to the v1.6.24 proof and the carried registry is the v1.6.24 one; the four new raws are their own scoring copies. The A2 row is not rescored here. jebadiah-9b-v2-1 and the two StartLux rows ran through the official v1.6 self-hosted path on the frozen input (sha256 dfd36720...); celeris-1-decision through the unchanged run_api_full_v161.py after a full-set exposure gate. Pricing module: the v1.6.25 re-freeze (sha256 32556e54..., predecessor 3948e3f1...), which adds exactly celeris-1-decision to its launch-list-price set.", "source_sha256": "9be38c87a6807abea30a0137b981f74c79b1ec247a6ea630a55cd5ebf0418fb6", "status": "published", "systems": [ { "addendum": null, "api_flag": true, "api_subset": false, "author": "Scira (zaid@scira.ai)", "axes": { "calibration": 88.99486051802106, "cost": 17.95976602146007, "intelligence": 69.72341288828827, "speed": 83.61644312651299 }, "calibration": { "parts": { "choice": { "ece_hard": 0.056815051399762694, "ece_score": 88.63698972004747, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 88.63698972004747 }, "noul": { "abstention_rate": 0.15200000000000002, "brier": 0.06134542488590291, "ece": 0.061674600501855234, "n": 375, "score": 87.66507989962895 }, "score": { "interval90_coverage": 0.9866666666666667, "interval90_width": 0.30288888888888893, "n": 375, "nrps": 0.054088924234169763, "score": 90.6825119343868, "top_ece": 0.06613041853904722 } }, "score": 88.99486051802106 }, "capability": 79.35913670315466, "composite_ci95": { "A": [ 5.45311287887791, 5.5715312549082645 ], "B": [ 5.86549219636293, 6.072075535488968 ], "C": [ 5.45311287887791, 5.5715312549082645 ] }, "cost": { "basis": "ESTIMATE (*est.), exact base-model market reference from the frozen 25 Sep 2026 snapshot (pricing_v15 sha256 dae86740...): qwen/qwen3.8-27b USD 0.42/M input, USD 3.0/M output, x the provider-reported input tokens; free keyless endpoint, nothing generated (output 0). Full 1,500 basis. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.542843 per 1,000 decisions is 8.40x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 precedent) and falls outside the official-caps board views by construction.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.5428432799999999 }, "display": "Amos 1 large (Scira)", "endpoint_condition": "Vendor-hosted api.amos.scira.ai /v1/systemone API, model `amos-1-large`; latency measured from Sandy (Germany) on the official per-item latency subset without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `scira` before the first request (CoS deputy #18975; every sealed item of this draw was already retired). Native probabilities on all three question types, no generated tokens. Hosting implementation and deployed weights were not independently inspected. Free, keyless; 1 s spacing for the documented 60 requests/minute.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 72.08563541362035, "I_sealed": 67.3611903629562, "base": 69.72341288828827, "excess": -5.115081719691553, "gap": 4.724445050664158, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 86.63494209808633, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 100.00000000000001, "hard": 86.78829820698331, "judge": 88.13447593935399, "standard": 77.39640016743407 } }, "open|noul": { "cc": 70.59903381642512, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 82.6086956521739, "judge": 39.99999999999999, "standard": 77.77777777777777 } }, "open|score": { "cc": 59.02293032634961, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 69.3524417552081, "hard": 68.76756728678528, "judge": 55.49798398673283, "standard": 39.65632020047416 } }, "sealed|choice": { "cc": 84.15652609682918, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 94.28208386277001, "hard": 79.70394356424192, "judge": 85.01469147894223, "standard": 86.71166420586374 } }, "sealed|noul": { "cc": 58.34810994461021, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 79.3103448275862, "hard": 49.09090909090909, "judge": 53.846153846153854, "standard": 73.13432835820895 } }, "sealed|score": { "cc": 59.578935047429226, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 79.12768431971863, "hard": 59.81698505291848, "judge": 55.43196057988098, "standard": 55.548922101628364 } } } }, "jevbench_score": 5.5363026097715755, "key": "amos-1-large", "last_measured_on": "2026-10-13", "licence": "Apache-2.0 open weights declared by the submitter (mukaddamzaid/amos-1-large); the measured object is Scira's hosted endpoint, whose deployed weights were not inspected", "listing": "ranked", "measured_in": "v1.6.17", "measurement_date_status": "measured", "model_pin": "hosted-api:api.amos.scira.ai/v1/systemone#amos-1-large", "noul_decisive": { "acc_among_decisive": 0.9654088050314465, "decisive": 318, "decisive_rate": 0.848, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 31, "ranked": true, "ranks": { "A": 31, "B": 30, "C": 29, "capability": 9 }, "repo": "https://huggingface.co/mukaddamzaid/amos-1-large", "scores": { "A": 5.5363026097715755, "B": 5.99760001090754, "C": 5.5363026097715755 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.5090118609950878, "p50_s_raw": 0.5090118609950878, "p95_s_adjusted": 0.8543341861019143, "p95_s_raw": 0.8543341861019143 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.9090909090909091, "cc": 85.70884085363811, "cc_by_type": { "choice": 83.33333333333333, "noul": 100.0, "score": 73.79318922758102 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 85.70884085363811 }, "calibration_gen": { "accuracy": 0.693069306930693, "cc": 53.4932926677426, "cc_by_type": { "choice": 74.54931071049842, "noul": 43.99999999999999, "score": 41.93056729272939 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 53.4932926677426 }, "composite_gen": { "accuracy": 0.9207920792079208, "cc": 86.67411659317217, "cc_by_type": { "choice": 96.7479674796748, "noul": 92.59259259259258, "score": 70.68178970724917 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 86.67411659317217 }, "fairness": { "accuracy": 0.9375, "cc": 83.2317845951076, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 89.69535378532278 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 83.2317845951076 }, "jaggedness": { "accuracy": 0.7222222222222222, "cc": 52.46040101820392, "cc_by_type": { "choice": 66.79499518768047, "noul": 29.729729729729737, "score": 60.85647813720154 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 52.46040101820392 }, "llm_router": { "accuracy": 0.9329073482428115, "cc": 84.01158887400364, "cc_by_type": { "choice": 95.24522321035484, "noul": 97.93814432989691, "score": 58.85139908175916 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 84.01158887400364 }, "long_state": { "accuracy": 0.6, "cc": 13.94993894062535, "cc_by_type": { "choice": 48.387096774193544, "noul": 0.0, "score": -6.5372799523174985 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 13.94993894062535 }, "ml_usecase": { "accuracy": 0.7037037037037037, "cc": 43.067969834272596, "cc_by_type": { "choice": 50.51546391752578, "noul": 25.0, "score": 53.68844558529201 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 43.067969834272596 }, "multilingual": { "accuracy": 0.7818181818181819, "cc": 62.019117736598936, "cc_by_type": { "choice": 81.41506358004565, "noul": 44.44444444444444, "score": 60.19784518530671 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 62.019117736598936 }, "negation_pair_gen": { "accuracy": 0.8953488372093024, "cc": 75.73214123542938, "cc_by_type": { "choice": 100.0, "noul": 66.66666666666667, "score": 60.529757039621465 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 75.73214123542938 }, "option_order": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 100.0 }, "perturb_pair": { "accuracy": 0.890625, "cc": 81.14468880284434, "cc_by_type": { "choice": 87.98798798798799, "noul": 100.0, "score": 55.44607842054504 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 81.14468880284434 }, "usecase": { "accuracy": 0.7863636363636364, "cc": 61.1838712254527, "cc_by_type": { "choice": 81.61764705882352, "noul": 41.66666666666667, "score": 60.267299950867894 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 61.1838712254527 } }, "lang": { "ar": { "accuracy": 0.75, "cc": 61.882864079814546, "cc_by_type": { "choice": 61.904761904761905, "noul": 100.0, "score": 23.743830334681725 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 61.882864079814546 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 0.7692307692307693, "cc": 52.73782751626438, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 24.880149215459824 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 52.73782751626438 }, "de": { "accuracy": 0.875, "cc": 83.90096934595631, "cc_by_type": { "choice": 76.10619469026548, "noul": 100.0, "score": 75.59671334760345 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 83.90096934595631 }, "el": { "accuracy": 0.6666666666666666, "cc": 53.05558601212194, "cc_by_type": { "choice": 66.4804469273743, "noul": 0.0, "score": 92.6863111089915 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 53.05558601212194 }, "en": { "accuracy": 0.8444830582102519, "cc": 71.92616617433795, "cc_by_type": { "choice": 86.38044446614026, "noul": 67.62589928057554, "score": 61.77215477629807 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 71.92616617433795 }, "es": { "accuracy": 0.8387096774193549, "cc": 70.72155656154153, "cc_by_type": { "choice": 90.2120717781403, "noul": 100.0, "score": 21.952597906484318 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 70.72155656154153 }, "fi": { "accuracy": 0.875, "cc": 74.873681863149, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 91.28771225611368 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 74.873681863149 }, "fr": { "accuracy": 0.8571428571428571, "cc": 81.05883440654436, "cc_by_type": { "choice": 57.64705882352941, "noul": 100.0, "score": 85.52944439610368 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 81.05883440654436 }, "hi": { "accuracy": 0.8095238095238095, "cc": 61.66610015218894, "cc_by_type": { "choice": 66.4804469273743, "noul": 50.0, "score": 68.51785352919251 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 61.66610015218894 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8666666666666667, "cc": 67.97700324005393, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": 17.96062780278067 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 67.97700324005393 }, "ja": { "accuracy": 0.68, "cc": 43.73893493579494, "cc_by_type": { "choice": 100.0, "noul": 19.999999999999996, "score": 11.216804807384817 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 43.73893493579494 }, "ko": { "accuracy": 0.6, "cc": 20.773298386484385, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": 62.319895159453154 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 20.773298386484385 }, "mixed": { "accuracy": 1.0, "cc": 97.24507216985027, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 91.73521650955081 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 97.24507216985027 }, "nl": { "accuracy": 0.5, "cc": -17.41688895535159, "cc_by_type": { "choice": 74.02597402597404, "noul": -100.0, "score": -26.276640892028812 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.8333333333333334, "cc": 75.0, "cc_by_type": { "choice": 100.0, "noul": 50.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 75.0 }, "pl": { "accuracy": 0.8461538461538461, "cc": 77.465615091364, "cc_by_type": { "choice": 92.382564536606, "noul": 80.0, "score": 60.01428073748598 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 77.465615091364 }, "pt": { "accuracy": 0.7857142857142857, "cc": 63.573232278872894, "cc_by_type": { "choice": 68.1791396582204, "noul": 60.00000000000001, "score": 62.54055717839828 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 63.573232278872894 }, "sv": { "accuracy": 0.6363636363636364, "cc": 56.68991453443647, "cc_by_type": { "choice": 48.93617021276595, "noul": 60.00000000000001, "score": 61.13357339054346 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 56.68991453443647 }, "tr": { "accuracy": 0.7692307692307693, "cc": 67.58806742727757, "cc_by_type": { "choice": 100.0, "noul": 50.0, "score": 52.764202281832695 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 67.58806742727757 }, "uk": { "accuracy": 0.8571428571428571, "cc": 80.91469064595238, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 75.00213645398617 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 80.91469064595238 }, "zh": { "accuracy": 0.9, "cc": 70.9322772713171, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 79.463498480618 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 70.9322772713171 } }, "type": { "choice": { "accuracy": 0.8853333333333333, "cc": 85.1891540213169, "cc_by_type": { "choice": 85.1891540213169 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 85.1891540213169 }, "noul": { "accuracy": 0.8186666666666667, "cc": 63.733333333333334, "cc_by_type": { "noul": 63.733333333333334 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 63.733333333333334 }, "score": { "accuracy": 0.7386666666666667, "cc": 60.63169182802718, "cc_by_type": { "score": 60.63169182802718 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 60.63169182802718 } }, "usecase": { "advertising": { "accuracy": 0.45454545454545453, "cc": 12.801234460928859, "cc_by_type": { "choice": 52.0, "noul": -50.0, "score": 36.40370338278658 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 12.801234460928859 }, "code_linting": { "accuracy": 0.9166666666666666, "cc": 83.81931093174488, "cc_by_type": { "choice": 81.36645962732919, "noul": 100.0, "score": 70.09147316790545 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 83.81931093174488 }, "customer_support": { "accuracy": 0.6428571428571429, "cc": 25.116387537117735, "cc_by_type": { "choice": 67.47967479674797, "noul": 0.0, "score": 7.869487814605236 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 25.116387537117735 }, "demand_forecasting": { "accuracy": 0.4117647058823529, "cc": -0.49442484717067475, "cc_by_type": { "choice": 18.128654970760234, "noul": 33.33333333333333, "score": -52.94526284560559 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.7647058823529411, "cc": 40.641138161384106, "cc_by_type": { "choice": 85.50724637681158, "noul": -33.333333333333336, "score": 69.74950144067407 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 40.641138161384106 }, "feature_extraction": { "accuracy": 0.6842105263157895, "cc": -4.760219093872943, "cc_by_type": { "choice": 74.84276729559748, "noul": 33.33333333333333, "score": -122.45675791054964 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.8333333333333334, "cc": 57.85491369773835, "cc_by_type": { "choice": 100.00000000000001, "noul": 14.28571428571428, "score": 59.27902680750074 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 57.85491369773835 }, "gaming": { "accuracy": 0.9090909090909091, "cc": 71.2806493361724, "cc_by_type": { "choice": 89.54703832752612, "noul": 50.0, "score": 74.29490968099105 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 71.2806493361724 }, "insurance_claims": { "accuracy": 0.5333333333333333, "cc": 16.02023316616262, "cc_by_type": { "choice": 37.5, "noul": -19.999999999999996, "score": 30.560699498487853 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 16.02023316616262 }, "knowledge_graphs": { "accuracy": 0.7272727272727273, "cc": 67.5820202385729, "cc_by_type": { "choice": 74.30406852248395, "noul": 66.66666666666667, "score": 61.77532552656804 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 67.5820202385729 }, "lead_generation": { "accuracy": 0.76, "cc": 62.07152141651769, "cc_by_type": { "choice": 88.3720930232558, "noul": 55.55555555555556, "score": 42.2869156707417 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 62.07152141651769 }, "legal_compliance": { "accuracy": 0.2, "cc": 0.9993600672424611, "cc_by_type": { "choice": -29.032258064516128, "noul": 33.33333333333333, "score": -1.3029950670898183 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.9993600672424611 }, "llm_guardrails": { "accuracy": 0.8518518518518519, "cc": 73.86169788565626, "cc_by_type": { "choice": 100.00000000000001, "noul": 60.00000000000001, "score": 61.58509365696874 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 73.86169788565626 }, "model_routing": { "accuracy": 0.9170506912442397, "cc": 81.67870057339692, "cc_by_type": { "choice": 95.32467532467533, "noul": 100.0, "score": 49.711426395515446 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 81.67870057339692 }, "moderation_trust_safety": { "accuracy": 0.7777777777777778, "cc": 19.51262058240074, "cc_by_type": { "choice": 88.48368522072937, "noul": -100.0, "score": 70.05417652647284 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 19.51262058240074 }, "recruiting": { "accuracy": 0.7777777777777778, "cc": 57.303088588281206, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 72.00211627552143 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 57.303088588281206 }, "risk_assessment": { "accuracy": 0.6666666666666666, "cc": 26.574834589572514, "cc_by_type": { "choice": 100.0, "noul": -50.0, "score": 29.724503768717547 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 26.574834589572514 }, "scientific_discovery": { "accuracy": 0.7391304347826086, "cc": 48.40292921906184, "cc_by_type": { "choice": 85.5072463768116, "noul": 0.0, "score": 59.701541280373924 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 48.40292921906184 }, "search_retrieval": { "accuracy": 1.0, "cc": 97.1544132796688, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 91.46323983900642 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 97.1544132796688 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 84.70410190085266, 89.27499648007384 ], "I": [ 64.83012939104476, 73.50027376113194 ], "capability": [ 75.526957020295, 80.7067065083684 ], "composite": { "A": [ 5.45311287887791, 5.5715312549082645 ], "B": [ 5.86549219636293, 6.072075535488968 ], "C": [ 5.45311287887791, 5.5715312549082645 ] }, "seed": 16 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 88.99486051802106, "capability": 79.35913670315466, "intelligence": 69.72341288828827 }, "per_type": { "choice": { "accuracy": 0.8853333333333333, "calibration": 88.63698972004747, "cc": 85.1891540213169, "cc_by_type": { "choice": 85.1891540213169 }, "cc_tiered": 85.39573409745776, "cc_tiered_open": 86.63494209808633, "cc_tiered_sealed": 84.15652609682918, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 85.1891540213169 }, "noul": { "accuracy": 0.8186666666666667, "calibration": 87.66507989962895, "cc": 63.733333333333334, "cc_by_type": { "noul": 63.733333333333334 }, "cc_tiered": 64.47357188051767, "cc_tiered_open": 70.59903381642512, "cc_tiered_sealed": 58.34810994461021, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 63.733333333333334 }, "score": { "accuracy": 0.7386666666666667, "calibration": 90.6825119343868, "cc": 60.63169182802718, "cc_by_type": { "score": 60.63169182802718 }, "cc_tiered": 59.30093268688942, "cc_tiered_open": 59.02293032634961, "cc_tiered_sealed": 59.578935047429226, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 60.63169182802718 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 4.721340447009444, "Emphasis on Accuracy 60:20:20": 5.829281470915487, "Emphasis on Cost 20:20:60": 3.3366111981752655, "Emphasis on Speed 20:60:20": 6.091626979152469, "Intelligence only": 8.995803852584828 } }, { "addendum": null, "api_flag": true, "api_subset": false, "author": "Scira (zaid@scira.ai)", "axes": { "calibration": 87.98659582926916, "cost": 36.657244733397086, "intelligence": 38.36362733252365, "speed": 86.48270625779318 }, "calibration": { "parts": { "choice": { "ece_hard": 0.06666585481947378, "ece_score": 86.66682903610524, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 86.66682903610524 }, "noul": { "abstention_rate": 0.43999999999999995, "brier": 0.12374828347808393, "ece": 0.06865945905447007, "n": 375, "score": 86.26810818910599 }, "score": { "interval90_coverage": 0.9413333333333334, "interval90_width": 0.4971111111111112, "n": 375, "nrps": 0.09858810096196241, "score": 91.02485026259626, "top_ece": 0.04045744689305625 } }, "score": 87.98659582926916 }, "capability": 63.175111580896406, "composite_ci95": { "A": [ 11.456107363335107, 21.7606419869365 ], "B": [ 10.40605460041017, 20.67120879171385 ], "C": [ 7.955630113427155, 15.111556935372572 ] }, "cost": { "basis": "ESTIMATE (*est.), exact base-model market reference from the frozen 25 Sep 2026 snapshot (pricing_v15 sha256 dae86740...): qwen/qwen3.5-9b USD 0.09999999999999999/M input, USD 0.15/M output, x the provider-reported input tokens; free keyless endpoint, nothing generated (output 0). Full 1,500 basis. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.129248 per 1,000 decisions is 2.00x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 precedent) and falls outside the official-caps board views by construction.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.12924839999999999 }, "display": "Amos 1 medium (Scira)", "endpoint_condition": "Vendor-hosted api.amos.scira.ai /v1/systemone API, model `amos-1-medium`; latency measured from Sandy (Germany) on the official per-item latency subset without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `scira` before the first request (CoS deputy #18975; every sealed item of this draw was already retired). Native probabilities on all three question types, no generated tokens. Hosting implementation and deployed weights were not independently inspected. Free, keyless; 1 s spacing for the documented 60 requests/minute.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 43.91406567257623, "I_sealed": 32.81318899247107, "base": 38.36362733252365, "excess": 1.2613499097494483, "gap": 11.10087668010516, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 69.87986229279102, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 84.43243243243244, "hard": 66.02705253224286, "judge": 80.22412656558998, "standard": 54.79280033486815 } }, "open|noul": { "cc": 22.332643202208423, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 57.14285714285714, "hard": 30.434782608695656, "judge": 0.0, "standard": 22.222222222222232 } }, "open|score": { "cc": 39.52969152272925, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 47.14080159637061, "hard": 44.43989097468418, "judge": 36.67618524977716, "standard": 30.18399699142684 } }, "sealed|choice": { "cc": 63.438944647462236, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 75.22236340533672, "hard": 63.05076905285068, "judge": 60.33300685602351, "standard": 62.982493144906144 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 31.034482758620683, "hard": -9.090909090909093, "judge": -16.92307692307692, "standard": 25.373134328358216 } }, "sealed|score": { "cc": 35.00062232995097, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 49.394405829279044, "hard": 31.30311505659157, "judge": 33.35206227687498, "standard": 37.67158520661971 } } } }, "jevbench_score": 16.594331365489452, "key": "amos-1-medium", "last_measured_on": "2026-10-13", "licence": "Apache-2.0 open weights declared by the submitter (mukaddamzaid/amos-1-medium); the measured object is Scira's hosted endpoint, whose deployed weights were not inspected", "listing": "ranked", "measured_in": "v1.6.17", "measurement_date_status": "measured", "model_pin": "hosted-api:api.amos.scira.ai/v1/systemone#amos-1-medium", "noul_decisive": { "acc_among_decisive": 0.9714285714285714, "decisive": 210, "decisive_rate": 0.56, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 21, "ranked": true, "ranks": { "A": 21, "B": 21, "C": 21, "capability": 20 }, "repo": "https://huggingface.co/mukaddamzaid/amos-1-medium", "scores": { "A": 16.594331365489452, "B": 15.459663751265634, "C": 11.523841226034342 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.44348932350840187, "p50_s_raw": 0.44348932350840187, "p95_s_adjusted": 0.5068112015054794, "p95_s_raw": 0.5068112015054794 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.8636363636363636, "cc": 70.72841647184556, "cc_by_type": { "choice": 83.33333333333333, "noul": 100.0, "score": 28.851916082203388 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 70.72841647184556 }, "calibration_gen": { "accuracy": 0.4752475247524752, "cc": 14.975090543066846, "cc_by_type": { "choice": 36.37327677624603, "noul": 4.0000000000000036, "score": 4.551994852954511 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 14.975090543066846 }, "composite_gen": { "accuracy": 0.49504950495049505, "cc": 6.091925072879623, "cc_by_type": { "choice": 67.47967479674797, "noul": -62.96296296296296, "score": 13.759063384853864 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 6.091925072879623 }, "fairness": { "accuracy": 0.9375, "cc": 91.49611264605214, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 74.48833793815639 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 91.49611264605214 }, "jaggedness": { "accuracy": 0.5111111111111111, "cc": 12.733145931714104, "cc_by_type": { "choice": 45.13955726660251, "noul": -40.54054054054054, "score": 33.60042106908033 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 12.733145931714104 }, "llm_router": { "accuracy": 0.7827476038338658, "cc": 63.07665363153567, "cc_by_type": { "choice": 71.47133926212908, "noul": 62.88659793814433, "score": 54.87202369433361 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 63.07665363153567 }, "long_state": { "accuracy": 0.5333333333333333, "cc": 17.509851646759817, "cc_by_type": { "choice": 35.483870967741936, "noul": 0.0, "score": 17.045683972537518 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 17.509851646759817 }, "ml_usecase": { "accuracy": 0.4074074074074074, "cc": -13.788806870177382, "cc_by_type": { "choice": 50.51546391752578, "noul": -75.0, "score": -16.881884528057924 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.6618181818181819, "cc": 40.822452938700465, "cc_by_type": { "choice": 70.6553635474405, "noul": 5.555555555555558, "score": 46.256439713105344 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 40.822452938700465 }, "negation_pair_gen": { "accuracy": 0.5, "cc": -6.27999295377069, "cc_by_type": { "choice": 73.33333333333334, "noul": -80.0, "score": -12.173312194645414 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.0 }, "option_order": { "accuracy": 0.8541666666666666, "cc": 81.35818908122504, "cc_by_type": { "choice": 81.35818908122504 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 81.35818908122504 }, "perturb_pair": { "accuracy": 0.78125, "cc": 72.02153229907263, "cc_by_type": { "choice": 66.96696696696698, "noul": 100.0, "score": 49.09762993025094 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 72.02153229907263 }, "usecase": { "accuracy": 0.5954545454545455, "cc": 33.58098123168537, "cc_by_type": { "choice": 63.23529411764707, "noul": 0.0, "score": 37.50764957740904 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 33.58098123168537 } }, "lang": { "ar": { "accuracy": 0.375, "cc": -5.625522909037805, "cc_by_type": { "choice": 42.857142857142854, "noul": 0.0, "score": -59.73371158425627 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.8571428571428571, "cc": 89.47368421052632, "cc_by_type": { "choice": 78.94736842105263, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 89.47368421052632 }, "da": { "accuracy": 0.7692307692307693, "cc": 49.37440980616833, "cc_by_type": { "choice": 79.26267281105991, "noul": 33.33333333333333, "score": 35.52722327411174 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 49.37440980616833 }, "de": { "accuracy": 0.90625, "cc": 86.26833998280311, "cc_by_type": { "choice": 84.07079646017698, "noul": 100.0, "score": 74.73422348823236 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 86.26833998280311 }, "el": { "accuracy": 0.4, "cc": 14.87006399662579, "cc_by_type": { "choice": 49.720670391061454, "noul": -100.0, "score": 94.88952159881592 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 14.87006399662579 }, "en": { "accuracy": 0.634231103388358, "cc": 35.36588322148937, "cc_by_type": { "choice": 63.754408659889414, "noul": 6.474820143884896, "score": 35.8684208606938 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 35.36588322148937 }, "es": { "accuracy": 0.7741935483870968, "cc": 62.123511307964144, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": -13.629466076107576 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 62.123511307964144 }, "fi": { "accuracy": 0.625, "cc": 36.950170300317374, "cc_by_type": { "choice": 67.21311475409836, "noul": -33.333333333333336, "score": 76.97072948018709 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 36.950170300317374 }, "fr": { "accuracy": 0.7857142857142857, "cc": 46.162580842492474, "cc_by_type": { "choice": 57.64705882352941, "noul": 0.0, "score": 80.84068370394802 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 46.162580842492474 }, "hi": { "accuracy": 0.5714285714285714, "cc": 15.044394910666796, "cc_by_type": { "choice": 83.24022346368716, "noul": -25.0, "score": -13.107038731686771 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 15.044394910666796 }, "id": { "accuracy": 0.5, "cc": -22.222222222222225, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8, "cc": 59.21561631194971, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": -8.32353298153199 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 59.21561631194971 }, "ja": { "accuracy": 0.68, "cc": 40.035094113356514, "cc_by_type": { "choice": 71.40587768069896, "noul": 19.999999999999996, "score": 28.699404659370586 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 40.035094113356514 }, "ko": { "accuracy": 0.5, "cc": -0.99009225371654, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0, "score": 41.47416768329484 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 75.0593334229456, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 91.84466693550348 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 75.0593334229456 }, "nl": { "accuracy": 0.4, "cc": -6.718549297485519, "cc_by_type": { "choice": 48.05194805194805, "noul": -100.0, "score": 31.79240405559539 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.0, "cc": -64.51612903225806, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.6666666666666666, "cc": 45.495999792566494, "cc_by_type": { "choice": 84.76512907321202, "noul": 19.999999999999996, "score": 31.72287030448745 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 45.495999792566494 }, "pt": { "accuracy": 0.7857142857142857, "cc": 70.80789114038846, "cc_by_type": { "choice": 46.96523276370065, "noul": 100.0, "score": 65.45844065746472 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 70.80789114038846 }, "sv": { "accuracy": 0.36363636363636365, "cc": 17.008028029074485, "cc_by_type": { "choice": 23.404255319148945, "noul": -19.999999999999996, "score": 47.61982876807451 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 17.008028029074485 }, "tr": { "accuracy": 0.7692307692307693, "cc": 63.07036542482134, "cc_by_type": { "choice": 78.26086956521739, "noul": 100.0, "score": 10.950226709246625 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 63.07036542482134 }, "uk": { "accuracy": 0.7142857142857143, "cc": 75.65635733546749, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 59.22713652253151 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 75.65635733546749 }, "zh": { "accuracy": 0.6, "cc": 1.6406301152936227, "cc_by_type": { "choice": 74.57627118644068, "noul": -33.333333333333336, "score": -36.32104750722647 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 1.6406301152936227 } }, "type": { "choice": { "accuracy": 0.732, "cc": 65.38395300331044, "cc_by_type": { "choice": 65.38395300331044 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 65.38395300331044 }, "noul": { "accuracy": 0.544, "cc": 8.800000000000008, "cc_by_type": { "noul": 8.800000000000008 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 8.800000000000008 }, "score": { "accuracy": 0.5653333333333334, "cc": 36.795063482577376, "cc_by_type": { "score": 36.795063482577376 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 36.795063482577376 } }, "usecase": { "advertising": { "accuracy": 0.3181818181818182, "cc": -11.612698505906499, "cc_by_type": { "choice": 20.0, "noul": -50.0, "score": -4.838095517719498 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.5416666666666666, "cc": 24.522153858600834, "cc_by_type": { "choice": 81.36645962732919, "noul": 11.111111111111116, "score": -18.911109162637807 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 24.522153858600834 }, "customer_support": { "accuracy": 0.5714285714285714, "cc": 30.995651582372062, "cc_by_type": { "choice": 34.959349593495936, "noul": 50.0, "score": 8.027605153620243 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 30.995651582372062 }, "demand_forecasting": { "accuracy": 0.4117647058823529, "cc": -21.304913464984477, "cc_by_type": { "choice": 41.5204678362573, "noul": -100.0, "score": -5.435208231210731 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5294117647058824, "cc": 19.91667593401915, "cc_by_type": { "choice": 56.52173913043478, "noul": -33.333333333333336, "score": 36.56162200495601 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 19.91667593401915 }, "feature_extraction": { "accuracy": 0.47368421052631576, "cc": -23.035431154993848, "cc_by_type": { "choice": 49.685534591194966, "noul": -33.333333333333336, "score": -85.45849472284317 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5333333333333333, "cc": 21.961594726546508, "cc_by_type": { "choice": 70.58823529411765, "noul": -42.85714285714286, "score": 38.15369174266473 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 21.961594726546508 }, "gaming": { "accuracy": 0.45454545454545453, "cc": -2.6028106451585344, "cc_by_type": { "choice": 47.73519163763067, "noul": -100.0, "score": 44.456376426893726 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.4, "cc": 14.729500546085612, "cc_by_type": { "choice": 58.33333333333333, "noul": -60.0, "score": 45.85516830492351 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 14.729500546085612 }, "knowledge_graphs": { "accuracy": 0.5, "cc": 18.37776298144904, "cc_by_type": { "choice": 22.912205567451824, "noul": 0.0, "score": 32.2210833768953 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 18.37776298144904 }, "lead_generation": { "accuracy": 0.64, "cc": 37.43981895623384, "cc_by_type": { "choice": 100.0, "noul": -11.111111111111116, "score": 23.430567979812633 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 37.43981895623384 }, "legal_compliance": { "accuracy": 0.3, "cc": -26.159304338357146, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0, "score": -13.96178398281338 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.5555555555555556, "cc": 26.337856947168092, "cc_by_type": { "choice": 65.4510556621881, "noul": -19.999999999999996, "score": 33.56251517931618 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 26.337856947168092 }, "model_routing": { "accuracy": 0.8341013824884793, "cc": 69.75791357046573, "cc_by_type": { "choice": 80.12987012987014, "noul": 87.87878787878789, "score": 41.265082702739164 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 69.75791357046573 }, "moderation_trust_safety": { "accuracy": 0.7222222222222222, "cc": 12.28249541563279, "cc_by_type": { "choice": 88.48368522072937, "noul": -100.0, "score": 48.363801026168986 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 12.28249541563279 }, "recruiting": { "accuracy": 0.7777777777777778, "cc": 62.49576394692955, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 54.15395850745532 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 62.49576394692955 }, "risk_assessment": { "accuracy": 0.7222222222222222, "cc": 41.45276807709171, "cc_by_type": { "choice": 85.22167487684729, "noul": 0.0, "score": 39.13662935442785 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 41.45276807709171 }, "scientific_discovery": { "accuracy": 0.4782608695652174, "cc": 10.829423826169853, "cc_by_type": { "choice": 27.536231884057973, "noul": -33.333333333333336, "score": 38.28537292778492 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 10.829423826169853 }, "search_retrieval": { "accuracy": 0.875, "cc": 77.41282379352732, "cc_by_type": { "choice": 100.0, "noul": 50.0, "score": 82.23847138058198 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 77.41282379352732 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 82.14451225239841, 88.63865202401091 ], "I": [ 32.87771623313232, 43.16183347366633 ], "capability": [ 58.690357425326795, 64.77827562501643 ], "composite": { "A": [ 11.456107363335107, 21.7606419869365 ], "B": [ 10.40605460041017, 20.67120879171385 ], "C": [ 7.955630113427155, 15.111556935372572 ] }, "seed": 17 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 87.98659582926916, "capability": 63.175111580896406, "intelligence": 38.36362733252365 }, "per_type": { "choice": { "accuracy": 0.732, "calibration": 86.66682903610524, "cc": 65.38395300331044, "cc_by_type": { "choice": 65.38395300331044 }, "cc_tiered": 66.65940347012662, "cc_tiered_open": 69.87986229279102, "cc_tiered_sealed": 63.438944647462236, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 65.38395300331044 }, "noul": { "accuracy": 0.544, "calibration": 86.26810818910599, "cc": 8.800000000000008, "cc_by_type": { "noul": 8.800000000000008 }, "cc_tiered": 11.166321601104212, "cc_tiered_open": 22.332643202208423, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 8.800000000000008 }, "score": { "accuracy": 0.5653333333333334, "calibration": 91.02485026259626, "cc": 36.795063482577376, "cc_by_type": { "score": 36.795063482577376 }, "cc_tiered": 37.26515692634011, "cc_tiered_open": 39.52969152272925, "cc_tiered_sealed": 35.00062232995097, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 36.795063482577376 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 14.624957720018413, "Emphasis on Accuracy 60:20:20": 13.517858975949368, "Emphasis on Cost 20:20:60": 13.243271225162754, "Emphasis on Speed 20:60:20": 17.971860545006717, "Intelligence only": 12.13943883806462 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "IBOU-SEARCH", "axes": { "calibration": 82.22627427476863, "cost": 60.82181099868142, "intelligence": 27.956374646954437, "speed": 89.69506927060962 }, "calibration": { "parts": { "choice": { "ece_hard": 0.13190589669297953, "ece_score": 73.6188206614041, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 73.6188206614041 }, "noul": { "abstention_rate": 0.581769436997319, "brier": 0.1619943265239912, "ece": 0.0674130307496136, "n": 373, "score": 86.51739385007728 }, "score": { "interval90_coverage": 0.9919354838709677, "interval90_width": 0.7842741935483872, "n": 372, "nrps": 0.1214577533167964, "score": 86.54260831282451, "top_ece": 0.07384504021335675 } }, "score": 82.22627427476863 }, "capability": 55.09132446086153, "composite_ci95": { "A": [ 12.638068496393121, 21.00186132087051 ], "B": [ 10.553027361763364, 18.124389061736544 ], "C": [ 8.776436455828554, 14.584625917271186 ] }, "cost": { "basis": "ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-4B -> deepinfra:Qwen/Qwen3.5-4B USD 0.030000000000000002/M input, USD 0.15/M output (frozen at the 25 Sep cut-off, I-3). Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens: the server reports prompt_tokens 0 (not a measurement), so input is counted OFFLINE with the author's tokenizer and shared-prefix method (the method of the earlier measurement, measurement receipts on file; 1,011,369 over 1,485), output 0 under the 9 Oct 2026 rule (forward passes only).", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.02022738 }, "display": "ArseneLupin v1.1 (IBOU-SEARCH, Qwen3.5-4B LoRA)", "endpoint_condition": "Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's `python -m arsenelupin serve` (engine shared_prefix, bf16, calibration T 2.45) behind the wire-name proxy.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 33.63845928917361, "I_sealed": 22.274290004735263, "base": 27.956374646954437, "excess": 1.524642514082636, "gap": 11.364169284438347, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 66.71937273976232, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 76.64864864864865, "hard": 54.70273670965713, "judge": 72.31377719182598, "standard": 77.39640016743407 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 71.42857142857142, "hard": 4.347826086956519, "judge": -40.0, "standard": -33.333333333333336 } }, "open|score": { "cc": 34.196005127758525, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 36.46334077824246, "hard": 22.992987523752227, "judge": 43.70520186247461, "standard": 41.20457740845502 } }, "sealed|choice": { "cc": 47.96190215319236, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 56.16264294790343, "hard": 44.836359431016525, "judge": 48.87365328109698, "standard": 48.74499050833158 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 10.344827586206895, "hard": -30.909090909090907, "judge": -53.84615384615385, "standard": -22.388059701492537 } }, "sealed|score": { "cc": 18.860967861013435, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 32.91947880531089, "hard": 16.11073859508191, "judge": 16.587666801933665, "standard": 20.742122509347404 } } } }, "jevbench_score": 16.558026636431048, "key": "arsenelupin-v1.1", "last_measured_on": "2026-10-16", "licence": "Apache-2.0", "listing": "ranked", "measured_in": "v1.6.24", "measurement_date_status": "measured", "model_pin": "IBOU-SEARCH/ArseneLupin-v1.1@411a6aefab87a4c155372c2a0a20777c2f5e1317", "noul_decisive": { "acc_among_decisive": 0.9423076923076923, "decisive": 156, "decisive_rate": 0.41823056300268097, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": 22, "ranked": true, "ranks": { "A": 22, "B": 22, "C": 22, "capability": 27 }, "repo": "https://huggingface.co/IBOU-SEARCH/ArseneLupin-v1.1", "scores": { "A": 16.558026636431048, "B": 14.045200350350513, "C": 11.498629608632674 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.3189419380272739, "p50_s_raw": 0.08447096901363693, "p95_s_adjusted": 0.3363422646856634, "p95_s_raw": 0.0931711323428317 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.6818181818181818, "cc": 50.51872953772545, "cc_by_type": { "choice": 50.0, "noul": 100.0, "score": 1.5561886131763458 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 50.51872953772545 }, "calibration_gen": { "accuracy": 0.38613861386138615, "cc": -1.2802984282284164, "cc_by_type": { "choice": 20.466595970307537, "noul": -28.000000000000004, "score": 3.692508745007217 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.504950495049505, "cc": 9.769891325776209, "cc_by_type": { "choice": 57.72357723577236, "noul": -62.96296296296296, "score": 34.54905970451922 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 9.769891325776209 }, "fairness": { "accuracy": 0.6875, "cc": 54.72513077918578, "cc_by_type": { "choice": 100.0, "noul": 19.999999999999996, "score": 44.17539233755736 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 54.72513077918578 }, "jaggedness": { "accuracy": 0.3055555555555556, "cc": -19.677183728618825, "cc_by_type": { "choice": 13.378248315688179, "noul": -67.56756756756756, "score": -4.842231933977081 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.6932907348242812, "cc": 46.00272325098312, "cc_by_type": { "choice": 69.09395086730652, "noul": 34.020618556701024, "score": 34.89360032894183 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 46.00272325098312 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.3333333333333333, "cc": -12.636857704693645, "cc_by_type": { "choice": 13.402061855670107, "noul": -75.0, "score": 23.687365030248962 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.5563636363636364, "cc": 19.72366546133326, "cc_by_type": { "choice": 56.9611998695794, "noul": -36.111111111111114, "score": 38.3209076255315 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 19.72366546133326 }, "negation_pair_gen": { "accuracy": 0.36046511627906974, "cc": -11.788936369121076, "cc_by_type": { "choice": 33.333333333333336, "noul": -73.33333333333334, "score": 4.633190892636774 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.0 }, "option_order": { "accuracy": 0.8125, "cc": 76.03195739014647, "cc_by_type": { "choice": 76.03195739014647 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 76.03195739014647 }, "perturb_pair": { "accuracy": 0.796875, "cc": 63.02665837969059, "cc_by_type": { "choice": 75.97597597597598, "noul": 60.00000000000001, "score": 53.10399916309577 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 63.02665837969059 }, "usecase": { "accuracy": 0.45, "cc": 8.571271059852021, "cc_by_type": { "choice": 50.98039215686277, "noul": -45.833333333333336, "score": 20.566754356026628 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 8.571271059852021 } }, "lang": { "ar": { "accuracy": 0.375, "cc": 13.512897780745016, "cc_by_type": { "choice": -14.285714285714286, "noul": 50.0, "score": 4.824407627949334 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 13.512897780745016 }, "cs": { "accuracy": 0.42857142857142855, "cc": 57.89473684210526, "cc_by_type": { "choice": 15.789473684210522, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 57.89473684210526 }, "da": { "accuracy": 0.6153846153846154, "cc": 4.233223448197042, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": 12.69967034459113 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 4.233223448197042 }, "de": { "accuracy": 0.6875, "cc": 39.83897361366874, "cc_by_type": { "choice": 76.10619469026548, "noul": 0.0, "score": 43.41072615074075 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 39.83897361366874 }, "el": { "accuracy": 0.3333333333333333, "cc": 4.953090501411556, "cc_by_type": { "choice": 32.960893854748605, "noul": -100.0, "score": 81.89837764948606 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 4.953090501411556 }, "en": { "accuracy": 0.5178105994787141, "cc": 15.967276196171342, "cc_by_type": { "choice": 50.3545233765758, "noul": -20.863309352517987, "score": 18.410614564456218 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 15.967276196171342 }, "es": { "accuracy": 0.8387096774193549, "cc": 61.117636296678036, "cc_by_type": { "choice": 90.2120717781403, "noul": 75.0, "score": 18.140837111893816 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 61.117636296678036 }, "fi": { "accuracy": 0.25, "cc": 2.2535777645684334, "cc_by_type": { "choice": 1.6393442622950836, "noul": -33.333333333333336, "score": 38.45472236474355 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 2.2535777645684334 }, "fr": { "accuracy": 0.5714285714285714, "cc": 29.96031603152154, "cc_by_type": { "choice": 36.47058823529412, "noul": 0.0, "score": 53.410359859270486 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 29.96031603152154 }, "hi": { "accuracy": 0.5714285714285714, "cc": 20.903606766234788, "cc_by_type": { "choice": 83.24022346368716, "noul": -75.0, "score": 54.470596835017204 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 20.903606766234788 }, "id": { "accuracy": 0.5, "cc": -22.222222222222225, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.7333333333333333, "cc": 43.96472119872376, "cc_by_type": { "choice": 85.97038191738113, "noul": 33.33333333333333, "score": 12.590448345456796 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 43.96472119872376 }, "ja": { "accuracy": 0.44, "cc": 7.47713527194653, "cc_by_type": { "choice": 14.2176330420969, "noul": 0.0, "score": 8.21377277374269 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 7.47713527194653 }, "ko": { "accuracy": 0.4, "cc": -10.20949871018135, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0, "score": 13.815948313900407 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 93.5496427429219, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 80.64892822876573 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 93.5496427429219 }, "nl": { "accuracy": 0.4, "cc": -13.055905252050016, "cc_by_type": { "choice": 22.077922077922082, "noul": -100.0, "score": 38.75436216592787 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.16666666666666666, "cc": -32.25806451612903, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.5641025641025641, "cc": 19.62379626604363, "cc_by_type": { "choice": 69.53025814642403, "noul": -40.0, "score": 29.34113065170686 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 19.62379626604363 }, "pt": { "accuracy": 0.5714285714285714, "cc": 22.01660632547086, "cc_by_type": { "choice": 46.96523276370065, "noul": -19.999999999999996, "score": 39.08458621271193 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 22.01660632547086 }, "sv": { "accuracy": 0.18181818181818182, "cc": -7.000724261111401, "cc_by_type": { "choice": -2.127659574468081, "noul": -60.0, "score": 41.12548679113388 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.46153846153846156, "cc": 3.4888991168227768, "cc_by_type": { "choice": 34.78260869565218, "noul": -50.0, "score": 25.68408865481615 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 3.4888991168227768 }, "uk": { "accuracy": 0.8571428571428571, "cc": 83.92575718462467, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 51.777271553874016 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 83.92575718462467 }, "zh": { "accuracy": 0.6, "cc": 9.15759259136832, "cc_by_type": { "choice": 74.57627118644068, "noul": -33.333333333333336, "score": -13.77016007900238 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 9.15759259136832 } }, "type": { "choice": { "accuracy": 0.6253333333333333, "cc": 51.60642186034941, "cc_by_type": { "choice": 51.60642186034941 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 51.60642186034941 }, "noul": { "accuracy": 0.392, "cc": -21.599999999999998, "cc_by_type": { "noul": -21.599999999999998 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.4693333333333333, "cc": 22.200266391018797, "cc_by_type": { "score": 22.200266391018797 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 22.200266391018797 } }, "usecase": { "advertising": { "accuracy": 0.3181818181818182, "cc": -15.830990825213652, "cc_by_type": { "choice": 52.0, "noul": -75.0, "score": -24.49297247564095 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.375, "cc": 1.7168943804986962, "cc_by_type": { "choice": 25.465838509316768, "noul": -33.333333333333336, "score": 13.018177965512656 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 1.7168943804986962 }, "customer_support": { "accuracy": 0.7142857142857143, "cc": 28.7536967232783, "cc_by_type": { "choice": 67.47967479674797, "noul": 0.0, "score": 18.78141537308693 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 28.7536967232783 }, "demand_forecasting": { "accuracy": 0.17647058823529413, "cc": -25.640628784241386, "cc_by_type": { "choice": -5.263157894736844, "noul": -100.0, "score": 28.34127154201269 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5294117647058824, "cc": 11.906917556526437, "cc_by_type": { "choice": 42.02898550724638, "noul": -33.333333333333336, "score": 27.025100495666265 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 11.906917556526437 }, "feature_extraction": { "accuracy": 0.3684210526315789, "cc": -28.821393224240854, "cc_by_type": { "choice": 24.528301886792455, "noul": -100.0, "score": -10.99248155951502 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5, "cc": 20.561134748485653, "cc_by_type": { "choice": 70.58823529411765, "noul": -42.85714285714286, "score": 33.95231180848217 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 20.561134748485653 }, "gaming": { "accuracy": 0.2727272727272727, "cc": -17.75792051210721, "cc_by_type": { "choice": 26.82926829268293, "noul": -100.0, "score": 19.896970170995452 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.26666666666666666, "cc": 5.325356796383858, "cc_by_type": { "choice": 37.5, "noul": -60.0, "score": 38.47607038915157 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 5.325356796383858 }, "knowledge_graphs": { "accuracy": 0.22727272727272727, "cc": -12.492206719922391, "cc_by_type": { "choice": -2.7837259100642404, "noul": -33.333333333333336, "score": -1.3595609163695954 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.52, "cc": 20.266776521258546, "cc_by_type": { "choice": 88.3720930232558, "noul": -33.333333333333336, "score": 5.761569873853178 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 20.266776521258546 }, "legal_compliance": { "accuracy": 0.3, "cc": -20.09225148226946, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0, "score": 4.239374585449685 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.48148148148148145, "cc": 5.74928641158261, "cc_by_type": { "choice": 53.93474088291747, "noul": -60.0, "score": 23.313118351830365 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 5.74928641158261 }, "model_routing": { "accuracy": 0.7004608294930875, "cc": 45.029199454317904, "cc_by_type": { "choice": 76.62337662337664, "noul": 39.393939393939405, "score": 19.07028234563767 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 45.029199454317904 }, "moderation_trust_safety": { "accuracy": 0.5555555555555556, "cc": 1.9206201017423343, "cc_by_type": { "choice": 53.93474088291747, "noul": -100.0, "score": 51.82711942230953 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 1.9206201017423343 }, "recruiting": { "accuracy": 0.5555555555555556, "cc": 20.15666397470658, "cc_by_type": { "choice": 66.57381615598885, "noul": -33.333333333333336, "score": 27.229509101464224 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 20.15666397470658 }, "risk_assessment": { "accuracy": 0.4444444444444444, "cc": 11.531424940662966, "cc_by_type": { "choice": 70.44334975369459, "noul": -50.0, "score": 14.150925068294306 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 11.531424940662966 }, "scientific_discovery": { "accuracy": 0.30434782608695654, "cc": -17.435804548794813, "cc_by_type": { "choice": 27.536231884057973, "noul": -100.0, "score": 20.15635446955759 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.6666666666666666, "cc": 15.043192447509194, "cc_by_type": { "choice": 78.64768683274022, "noul": -100.0, "score": 66.48189050978736 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 15.043192447509194 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 77.05749158556878, 84.35911732206152 ], "I": [ 25.130908153793317, 30.813538172271908 ], "capability": [ 52.12469490249783, 56.47423186619373 ], "composite": { "A": [ 12.638068496393121, 21.00186132087051 ], "B": [ 10.553027361763364, 18.124389061736544 ], "C": [ 8.776436455828554, 14.584625917271186 ] }, "seed": 18 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 82.22627427476863, "capability": 55.09132446086153, "intelligence": 27.956374646954437 }, "per_type": { "choice": { "accuracy": 0.6253333333333333, "calibration": 73.6188206614041, "cc": 51.60642186034941, "cc_by_type": { "choice": 51.60642186034941 }, "cc_tiered": 57.340637446477345, "cc_tiered_open": 66.71937273976232, "cc_tiered_sealed": 47.96190215319236, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 51.60642186034941 }, "noul": { "accuracy": 0.392, "calibration": 86.51739385007728, "cc": -21.599999999999998, "cc_by_type": { "noul": -21.599999999999998 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.4693333333333333, "calibration": 86.54260831282451, "cc": 22.200266391018797, "cc_by_type": { "score": 22.200266391018797 }, "cc_tiered": 26.52848649438598, "cc_tiered_open": 34.196005127758525, "cc_tiered_sealed": 18.860967861013435, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 22.200266391018797 } }, "raw_on_A": { "calibration": 75.10885434845505, "capability": 52.09577580877465, "intelligence": 29.082697269094247 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 14.802164601429327, "Emphasis on Accuracy 60:20:20": 11.587198338419961, "Emphasis on Cost 20:20:60": 16.241312971660097, "Emphasis on Speed 20:60:20": 18.248326776097045, "Intelligence only": 8.739821181204116 } }, { "addendum": null, "api_flag": true, "api_subset": false, "author": "attest", "axes": { "calibration": 73.84793106824938, "cost": 50.86035455384382, "intelligence": 65.23849472031306, "speed": 97.10993413996468 }, "calibration": { "parts": { "choice": { "ece_hard": 0.08913950574898584, "ece_score": 82.17209885020283, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 82.17209885020283 }, "noul": { "abstention_rate": 0.02680965147453085, "brier": 0.12374104570386348, "ece": 0.1233374127983535, "n": 373, "score": 75.3325174403293 }, "score": { "interval90_coverage": 0.7016129032258065, "interval90_width": 0.0378584229390681, "n": 372, "nrps": 0.10002561109774619, "score": 64.03917691421601, "top_ece": 0.3095954253089667 } }, "score": 73.84793106824938 }, "capability": 69.54321289428123, "composite_ci95": { "A": [ 65.63036158689977, 69.61532365956334 ], "B": [ 64.31522564445854, 69.6364881740045 ], "C": [ 64.50783042337476, 69.61532365956334 ] }, "cost": { "basis": "ESTIMATE, v1.5-M2 base-model reference rule: vendor-hosted API, customer list price USD 0.048/M input and USD 0.00/M output effective 2026-10-11 has been in force fewer than 30 days. Base model google/gemma-4-31b-it reference on OpenRouter USD 0.09/M input / USD 0.34/M output, snapshot fetched 2026-10-12T17:55:05Z (SHA-256 81eae2f3f94bf299c8d4499260f842c357b7a869ce82dfa1741ad6a89cf57014), same base-model estimate precedent as metask_jev_rain_31b_trained. Applied to measured response usage; failed HTTP 500 usage/billing may be unreported. No output generation; reported output tokens 0. Estimated, not a vendor bill.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.043449839999999997 }, "display": "attest-31B-v1.0", "endpoint_condition": "Paid public fast-lane order 07d13315. Vendor-hosted attest.engineering /v1/systemone API; latency measured from Sandy (Germany), using the official per-item latency subset without API adjustment. Full S1200+P300 draw, not API A300 supplement; no API equating. 15 of 1,500 requests returned HTTP 500 after 8 retry invocations and are scored as wrong in the full denominator. Public model documentation declares frozen google/gemma-4-31B-it @842da3794eaa0b77d5f08bae87a17459d91ff475, unmodified bf16, no adapter or quantization; single forward pass reading option-label logits, no generated tokens, one fitted temperature (value not documented). Hosting implementation and deployed weights were not independently inspected. Hosted API access is not an open-weights release by the vendor; base weights are under the Gemma 4 licence. ESTIMATE, v1.5-M2 base-model reference rule: vendor-hosted API, customer list price USD 0.048/M input and USD 0.00/M output effective 2026-10-11 has been in force fewer than 30 days. Base model google/gemma-4-31b-it reference on OpenRouter USD 0.09/M input / USD 0.34/M output, snapshot fetched 2026-10-12T17:55:05Z (SHA-256 81eae2f3f94bf299c8d4499260f842c357b7a869ce82dfa1741ad6a89cf57014), same base-model estimate precedent as metask_jev_rain_31b_trained. Applied to measured response usage; failed HTTP 500 usage/billing may be unreported. No output generation; reported output tokens 0. Estimated, not a vendor bill.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 64.92424666524875, "I_sealed": 65.55274277537737, "base": 65.23849472031306, "excess": -10.468022880484334, "gap": -0.6284961101286228, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 73.17162102453656, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 88.32432432432431, "hard": 67.9144385026738, "judge": 72.31377719182598, "standard": 77.39640016743407 } }, "open|noul": { "cc": 68.67632850241546, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 73.91304347826086, "judge": 60.00000000000001, "standard": 55.55555555555556 } }, "open|score": { "cc": 52.92479046879421, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 63.941722778327325, "hard": 60.749677220164536, "judge": 51.85551193839042, "standard": 33.370468606892665 } }, "sealed|choice": { "cc": 74.62818230752828, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 80.94027954256671, "hard": 71.89776801202729, "judge": 74.43682664054849, "standard": 77.2199957814807 } }, "sealed|noul": { "cc": 68.21259029184918, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 86.20689655172413, "hard": 58.18181818181818, "judge": 72.30769230769232, "standard": 73.13432835820895 } }, "sealed|score": { "cc": 53.817455726754645, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 78.07383467048999, "hard": 52.90470314143051, "judge": 47.46936317941805, "standard": 53.03691024654009 } } } }, "jevbench_score": 67.9936749770638, "key": "attest-31b-v1.0", "last_measured_on": "2026-10-12", "licence": "Hosted API; declared base weights under the Gemma 4 licence (no vendor weights release)", "listing": "ranked", "measured_in": "v1.6.13", "measurement_date_status": "measured", "model_pin": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475", "noul_decisive": { "acc_among_decisive": 0.8787878787878788, "decisive": 363, "decisive_rate": 0.9731903485254692, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": 6, "ranked": true, "ranks": { "A": 6, "B": 6, "C": 5, "capability": 15 }, "repo": "https://attest.engineering/model", "scores": { "A": 67.9936749770638, "B": 67.42417758535593, "C": 67.9936749770638 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.10787388649987406, "p50_s_raw": 0.10787388649987406, "p95_s_adjusted": 0.18033925039962923, "p95_s_raw": 0.18033925039962923 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.8636363636363636, "cc": 91.66666666649981, "cc_by_type": { "choice": 75.0, "noul": 100.0, "score": 99.99999999949942 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 91.66666666649981 }, "calibration_gen": { "accuracy": 0.7227722772277227, "cc": 65.68205659085977, "cc_by_type": { "choice": 61.82396606574761, "noul": 76.0, "score": 59.22220370683169 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 65.68205659085977 }, "composite_gen": { "accuracy": 0.8217821782178217, "cc": 71.7068789691002, "cc_by_type": { "choice": 93.4959349593496, "noul": 62.962962962962955, "score": 58.66173898498804 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 71.7068789691002 }, "fairness": { "accuracy": 0.9375, "cc": 90.47619047605701, "cc_by_type": { "choice": 71.42857142857143, "noul": 100.0, "score": 99.99999999959958 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 90.47619047605701 }, "jaggedness": { "accuracy": 0.5722222222222222, "cc": 41.84478952801769, "cc_by_type": { "choice": 45.13955726660251, "noul": 56.75675675675676, "score": 23.63805456069382 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 41.84478952801769 }, "llm_router": { "accuracy": 0.8977635782747604, "cc": 83.50801061677168, "cc_by_type": { "choice": 88.90552082416131, "noul": 93.81443298969072, "score": 67.804078036463 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 83.50801061677168 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.7407407407407407, "cc": 60.78261745433469, "cc_by_type": { "choice": 50.51546391752578, "noul": 75.0, "score": 56.83238844547831 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 60.78261745433469 }, "multilingual": { "accuracy": 0.7563636363636363, "cc": 63.88819552264045, "cc_by_type": { "choice": 71.63351809585916, "noul": 61.111111111111114, "score": 58.91995736095108 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 63.88819552264045 }, "negation_pair_gen": { "accuracy": 0.7441860465116279, "cc": 41.72809466728673, "cc_by_type": { "choice": 93.33333333333333, "noul": 6.666666666666665, "score": 25.184284001860192 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 41.72809466728673 }, "option_order": { "accuracy": 0.8958333333333334, "cc": 86.68442077230361, "cc_by_type": { "choice": 86.68442077230361 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 86.68442077230361 }, "perturb_pair": { "accuracy": 0.875, "cc": 85.01995309361071, "cc_by_type": { "choice": 81.98198198198199, "noul": 100.0, "score": 73.07787729885015 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 85.01995309361071 }, "usecase": { "accuracy": 0.7772727272727272, "cc": 69.59874631177811, "cc_by_type": { "choice": 76.7156862745098, "noul": 75.0, "score": 57.080552660824544 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 69.59874631177811 } }, "lang": { "ar": { "accuracy": 0.5625, "cc": 25.657000371915586, "cc_by_type": { "choice": 42.857142857142854, "noul": 50.0, "score": -15.886141741396088 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 25.657000371915586 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 1.0, "cc": 99.83499124305172, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 99.50497372915514 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 99.83499124305172 }, "de": { "accuracy": 0.875, "cc": 88.7927420089141, "cc_by_type": { "choice": 76.10619469026548, "noul": 100.0, "score": 90.27203133647683 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 88.7927420089141 }, "el": { "accuracy": 0.7333333333333333, "cc": 66.60459342006904, "cc_by_type": { "choice": 66.4804469273743, "noul": 33.33333333333333, "score": 99.9999999994995 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 66.60459342006904 }, "en": { "accuracy": 0.7793223284100782, "cc": 66.85046932152197, "cc_by_type": { "choice": 75.39693193883403, "noul": 70.50359712230217, "score": 54.650878903429714 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 66.85046932152197 }, "es": { "accuracy": 0.8064516129032258, "cc": 66.69113187317208, "cc_by_type": { "choice": 80.42414355628058, "noul": 100.0, "score": 19.649252063235643 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 66.69113187317208 }, "fi": { "accuracy": 0.75, "cc": 66.84881602901042, "cc_by_type": { "choice": 67.21311475409836, "noul": 33.33333333333333, "score": 99.99999999959955 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 66.84881602901042 }, "fr": { "accuracy": 0.7142857142857143, "cc": 75.35603715160619, "cc_by_type": { "choice": 57.64705882352941, "noul": 100.0, "score": 68.42105263128916 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 75.35603715160619 }, "hi": { "accuracy": 0.9047619047619048, "cc": 81.9134078211331, "cc_by_type": { "choice": 83.24022346368716, "noul": 100.0, "score": 62.499999999712166 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 81.9134078211331 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8, "cc": 53.22313405090798, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": -26.300979764657196 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 53.22313405090798 }, "ja": { "accuracy": 0.68, "cc": 45.07973339782123, "cc_by_type": { "choice": 71.40587768069896, "noul": 60.00000000000001, "score": 3.833322512764714 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 45.07973339782123 }, "ko": { "accuracy": 0.7, "cc": 46.296506956290735, "cc_by_type": { "choice": 55.55555555555555, "noul": 0.0, "score": 83.33396531331667 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 46.296506956290735 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 77.77777777761094, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 99.9999999994995 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 77.77777777761094 }, "nl": { "accuracy": 0.6, "cc": 43.862463409737074, "cc_by_type": { "choice": 48.05194805194805, "noul": 33.33333333333333, "score": 50.202108843929864 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 43.862463409737074 }, "no": { "accuracy": 0.5, "cc": 10.483870967741936, "cc_by_type": { "choice": -29.032258064516128, "noul": 50.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 10.483870967741936 }, "pl": { "accuracy": 0.7435897435897436, "cc": 58.11281326204074, "cc_by_type": { "choice": 84.76512907321202, "noul": 39.99999999999999, "score": 49.5733107129102 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 58.11281326204074 }, "pt": { "accuracy": 0.75, "cc": 73.9630456140903, "cc_by_type": { "choice": 57.57218621096051, "noul": 100.0, "score": 64.3169506313104 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 73.9630456140903 }, "sv": { "accuracy": 0.6363636363636364, "cc": 52.97872340417189, "cc_by_type": { "choice": 48.93617021276595, "noul": 60.00000000000001, "score": 49.99999999974971 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 52.97872340417189 }, "tr": { "accuracy": 0.9230769230769231, "cc": 88.24218637816098, "cc_by_type": { "choice": 78.26086956521739, "noul": 100.0, "score": 86.46568956926559 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 88.24218637816098 }, "uk": { "accuracy": 0.7142857142857143, "cc": 72.53664033508191, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 49.86798552137475 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 72.53664033508191 }, "zh": { "accuracy": 0.9, "cc": 66.66666666666664, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": -4.440892098500626e-14 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 66.66666666666664 } }, "type": { "choice": { "accuracy": 0.804, "cc": 74.68378652480914, "cc_by_type": { "choice": 74.68378652480914 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 74.68378652480914 }, "noul": { "accuracy": 0.8506666666666667, "cc": 70.13333333333334, "cc_by_type": { "noul": 70.13333333333334 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 70.13333333333334 }, "score": { "accuracy": 0.6506666666666666, "cc": 55.14020270807265, "cc_by_type": { "score": 55.14020270807265 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 55.14020270807265 } }, "usecase": { "advertising": { "accuracy": 0.6363636363636364, "cc": 41.05013190841991, "cc_by_type": { "choice": 84.0, "noul": 0.0, "score": 39.15039572525971 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 41.05013190841991 }, "code_linting": { "accuracy": 0.7916666666666666, "cc": 69.18221735790132, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 7.546652073703974 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 69.18221735790132 }, "customer_support": { "accuracy": 0.7142857142857143, "cc": 33.73983739844072, "cc_by_type": { "choice": 51.21951219512195, "noul": 100.0, "score": -49.999999999799805 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 33.73983739844072 }, "demand_forecasting": { "accuracy": 0.35294117647058826, "cc": 40.20929030056567, "cc_by_type": { "choice": -5.263157894736844, "noul": 100.0, "score": 25.891028796433858 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 40.20929030056567 }, "ecommerce_marketplaces": { "accuracy": 0.7647058823529411, "cc": 59.627914855635204, "cc_by_type": { "choice": 71.0144927536232, "noul": 33.33333333333333, "score": 74.53591847994909 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 59.627914855635204 }, "feature_extraction": { "accuracy": 0.7894736842105263, "cc": 81.67133102077554, "cc_by_type": { "choice": 49.685534591194966, "noul": 100.0, "score": 95.32845847113167 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 81.67133102077554 }, "financial_crime": { "accuracy": 0.6333333333333333, "cc": 51.05695837350043, "cc_by_type": { "choice": 70.58823529411765, "noul": 42.85714285714286, "score": 39.72549696924077 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 51.05695837350043 }, "gaming": { "accuracy": 0.6363636363636364, "cc": 31.06744966418366, "cc_by_type": { "choice": 58.18815331010453, "noul": -50.0, "score": 85.01419568244644 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 31.06744966418366 }, "insurance_claims": { "accuracy": 0.6, "cc": 45.83333333332221, "cc_by_type": { "choice": 37.5, "noul": 100.0, "score": -3.3373304120232206e-11 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 45.83333333332221 }, "knowledge_graphs": { "accuracy": 0.8181818181818182, "cc": 79.31347738618848, "cc_by_type": { "choice": 74.30406852248395, "noul": 100.0, "score": 63.636363636081526 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 79.31347738618848 }, "lead_generation": { "accuracy": 0.8, "cc": 72.4290843339358, "cc_by_type": { "choice": 88.3720930232558, "noul": 77.77777777777777, "score": 51.13738220077384 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 72.4290843339358 }, "legal_compliance": { "accuracy": 0.8, "cc": 74.35692248903656, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 87.58689649936777 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 74.35692248903656 }, "llm_guardrails": { "accuracy": 0.8888888888888888, "cc": 78.81343307279933, "cc_by_type": { "choice": 100.00000000000001, "noul": 60.00000000000001, "score": 76.44029921839794 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 78.81343307279933 }, "model_routing": { "accuracy": 0.8894009216589862, "cc": 82.42648273169006, "cc_by_type": { "choice": 90.64935064935067, "noul": 100.0, "score": 56.63009754571953 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 82.42648273169006 }, "moderation_trust_safety": { "accuracy": 0.9444444444444444, "cc": 92.05979877042455, "cc_by_type": { "choice": 100.00000000000001, "noul": 100.0, "score": 76.17939631127363 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 92.05979877042455 }, "recruiting": { "accuracy": 0.7222222222222222, "cc": 57.779434740179134, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 73.43115473121522 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 57.779434740179134 }, "risk_assessment": { "accuracy": 0.6666666666666666, "cc": 47.99092050610208, "cc_by_type": { "choice": 70.44334975369459, "noul": 50.0, "score": 23.52941176461164 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 47.99092050610208 }, "scientific_discovery": { "accuracy": 0.7391304347826086, "cc": 59.19668873382481, "cc_by_type": { "choice": 71.01449275362319, "noul": 33.33333333333333, "score": 73.24224011451793 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 59.19668873382481 }, "search_retrieval": { "accuracy": 1.0, "cc": 99.99999999986125, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 99.99999999958375 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 99.99999999986125 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 69.4525546701294, 77.05018910372995 ], "I": [ 59.359757454986834, 69.86878066910623 ], "capability": [ 65.09462316887347, 72.93154551383675 ], "composite": { "A": [ 65.63036158689977, 69.61532365956334 ], "B": [ 64.31522564445854, 69.6364881740045 ], "C": [ 64.50783042337476, 69.61532365956334 ] }, "seed": 19 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 73.84793106824938, "capability": 69.54321289428123, "intelligence": 65.23849472031306 }, "per_type": { "choice": { "accuracy": 0.804, "calibration": 82.17209885020283, "cc": 74.68378652480914, "cc_by_type": { "choice": 74.68378652480914 }, "cc_tiered": 73.89990166603242, "cc_tiered_open": 73.17162102453656, "cc_tiered_sealed": 74.62818230752828, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 74.68378652480914 }, "noul": { "accuracy": 0.8506666666666667, "calibration": 75.3325174403293, "cc": 70.13333333333334, "cc_by_type": { "noul": 70.13333333333334 }, "cc_tiered": 68.44445939713232, "cc_tiered_open": 68.67632850241546, "cc_tiered_sealed": 68.21259029184918, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 70.13333333333334 }, "score": { "accuracy": 0.6506666666666666, "calibration": 64.03917691421601, "cc": 55.14020270807265, "cc_by_type": { "score": 55.14020270807265 }, "cc_tiered": 53.37112309777443, "cc_tiered_open": 52.92479046879421, "cc_tiered_sealed": 53.817455726754645, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 55.14020270807265 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 66.24320971525083, "Emphasis on Accuracy 60:20:20": 65.83763287132996, "Emphasis on Cost 20:20:60": 59.093957268612236, "Emphasis on Speed 20:60:20": 75.89225472047454, "Intelligence only": 65.23849472031306 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "AgentBull (@0xDing)", "axes": { "calibration": 84.35778065081391, "cost": null, "intelligence": 25.21389259164919, "speed": 90.58298788933249 }, "calibration": { "parts": { "choice": { "ece_hard": 0.07912071749672457, "ece_score": 84.17585650065509, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 84.17585650065509 }, "noul": { "abstention_rate": 0.7426273458445041, "brier": 0.16309961620902136, "ece": 0.09024781687300962, "n": 373, "score": 81.95043662539807 }, "score": { "interval90_coverage": 0.9731182795698925, "interval90_width": 0.7293906810035842, "n": 372, "nrps": 0.1309150904455136, "score": 86.9470488263886, "top_ece": 0.0650719665133573 } }, "score": 84.35778065081391 }, "capability": 54.78583662123155, "composite_ci95": { "A": null, "B": null, "C": null }, "cost": { "basis": "UNPRICED. The frozen pricing_v15 module (sha256 dcfcecb0...2835, and every later re-freeze, which only add hosted-API launch-list keys) RAISES on the declared base google/t5gemma-2-4b-4b ('base model lacks an explicit market-listing decision'); no price is invented here. Axes only: Intelligence, Calibration and Speed are computed; Cost, composite and rank are void. Tokens as reported by the author's server (input 786,538, output_tokens 0).", "common_basis": null, "kind": "unpriced", "usage_estimated_rows": 0, "usd_per_1000": null }, "display": "Bongard-mini (AgentBull, T5Gemma-2 4B-4B)", "endpoint_condition": "Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's own `bongard serve` with their temperatures.json, default token budgets (64,000 request / 32,768 sequence), bf16.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 29.129004250783858, "I_sealed": 21.29878093251452, "base": 25.21389259164919, "excess": -2.0093034520863746, "gap": 7.830223318269336, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 68.15046810038176, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 84.43243243243244, "hard": 69.80182447310474, "judge": 64.40342781806197, "standard": 62.32733361239012 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 0.0, "hard": -56.52173913043479, "judge": -50.0, "standard": -11.111111111111116 } }, "open|score": { "cc": 19.23654465196982, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 35.916731669865264, "hard": 9.296542606261049, "judge": 38.265686721521966, "standard": 2.2327421301114203 } }, "sealed|choice": { "cc": 47.46015197187779, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 61.88055908513342, "hard": 40.152654099687744, "judge": 47.11067580803136, "standard": 55.38915840539971 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": -31.034482758620683, "hard": -60.0, "judge": -72.3076923076923, "standard": -58.2089552238806 } }, "sealed|score": { "cc": 16.436190825665776, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 26.462361570919747, "hard": 11.017848733353297, "judge": 15.93709798260039, "standard": 23.008428902261812 } } } }, "jevbench_score": null, "key": "bongard-mini", "last_measured_on": "2026-10-16", "licence": "Gemma terms of use", "listing": "listed", "measured_in": "v1.6.24", "measurement_date_status": "measured", "model_pin": "AgentBull/bongard-mini@c81c5b883a5983669be84be98658eac940489613", "not_ranked_because": "unpriced: the frozen pricing path has no market-listing decision for the declared base google/t5gemma-2-4b-4b and raises rather than estimating; one listing decision would make the row rankable.", "noul_decisive": { "acc_among_decisive": 0.9375, "decisive": 96, "decisive_rate": 0.257372654155496, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": null, "ranked": false, "ranks": { "A": null, "B": null, "C": null, "capability": null }, "repo": "https://huggingface.co/AgentBull/bongard-mini", "scores": { "A": null, "B": null, "C": null }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.2890852989861742, "p50_s_raw": 0.06954264949308708, "p95_s_adjusted": 0.3024650537088746, "p95_s_raw": 0.07623252685443731 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.5, "cc": 0.4622404706696545, "cc_by_type": { "choice": 41.666666666666664, "noul": -50.0, "score": 9.7200547453423 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 0.4622404706696545 }, "calibration_gen": { "accuracy": 0.21782178217821782, "cc": -35.922342791778796, "cc_by_type": { "choice": 10.92258748674444, "noul": -100.0, "score": -18.689615862080842 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.5841584158415841, "cc": 19.530539881920774, "cc_by_type": { "choice": 80.48780487804878, "noul": -40.74074074074075, "score": 18.8445555084543 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 19.530539881920774 }, "fairness": { "accuracy": 0.5625, "cc": 21.117700941050316, "cc_by_type": { "choice": 100.0, "noul": -60.0, "score": 23.35310282315095 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 21.117700941050316 }, "jaggedness": { "accuracy": 0.26666666666666666, "cc": -28.736348312955844, "cc_by_type": { "choice": 16.265640038498567, "noul": -94.5945945945946, "score": -7.88009038277151 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.5942492012779552, "cc": 29.786361842610262, "cc_by_type": { "choice": 61.96178568283877, "noul": -11.340206185567014, "score": 38.73750603055903 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 29.786361842610262 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.37037037037037035, "cc": -20.572694854487796, "cc_by_type": { "choice": 25.773195876288657, "noul": -100.0, "score": 12.50871956024795 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.5272727272727272, "cc": 10.133806197798606, "cc_by_type": { "choice": 60.873818063254, "noul": -61.111111111111114, "score": 30.638711641252925 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 10.133806197798606 }, "negation_pair_gen": { "accuracy": 0.6046511627906976, "cc": 15.634962703773246, "cc_by_type": { "choice": 83.33333333333333, "noul": -53.333333333333336, "score": 16.90488811131974 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 15.634962703773246 }, "option_order": { "accuracy": 0.7291666666666666, "cc": 65.37949400798935, "cc_by_type": { "choice": 65.37949400798935 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 65.37949400798935 }, "perturb_pair": { "accuracy": 0.71875, "cc": 43.09664377241094, "cc_by_type": { "choice": 66.96696696696698, "noul": 19.999999999999996, "score": 42.32296435026585 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 43.09664377241094 }, "usecase": { "accuracy": 0.41818181818181815, "cc": 0.5246135780320955, "cc_by_type": { "choice": 46.07843137254903, "noul": -70.83333333333333, "score": 26.32874269488058 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 0.5246135780320955 } }, "lang": { "ar": { "accuracy": 0.375, "cc": -17.34084978485645, "cc_by_type": { "choice": 23.809523809523807, "noul": -50.0, "score": -25.83207316409315 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.5714285714285714, "cc": -21.05263157894737, "cc_by_type": { "choice": 57.89473684210526, "noul": -100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 0.0 }, "da": { "accuracy": 0.8461538461538461, "cc": 39.147237797438464, "cc_by_type": { "choice": 100.0, "noul": -33.333333333333336, "score": 50.77504672564872 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 39.147237797438464 }, "de": { "accuracy": 0.65625, "cc": 43.40220333009313, "cc_by_type": { "choice": 60.176991150442475, "noul": 25.0, "score": 45.02961883983691 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 43.40220333009313 }, "el": { "accuracy": 0.5333333333333333, "cc": 19.633041242538255, "cc_by_type": { "choice": 83.24022346368716, "noul": -100.0, "score": 75.6589002639276 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 19.633041242538255 }, "en": { "accuracy": 0.473501303214596, "cc": 3.7280108956536004, "cc_by_type": { "choice": 49.91518287548356, "noul": -53.23741007194245, "score": 14.506259883419691 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 3.7280108956536004 }, "es": { "accuracy": 0.7096774193548387, "cc": 34.3819807991804, "cc_by_type": { "choice": 90.2120717781403, "noul": 0.0, "score": 12.933870619400889 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 34.3819807991804 }, "fi": { "accuracy": 0.25, "cc": 0.3453151180197314, "cc_by_type": { "choice": 34.42622950819673, "noul": -100.0, "score": 66.60971584586247 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 0.3453151180197314 }, "fr": { "accuracy": 0.7142857142857143, "cc": 38.936757527017214, "cc_by_type": { "choice": 78.82352941176471, "noul": 0.0, "score": 37.98674316928694 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 38.936757527017214 }, "hi": { "accuracy": 0.38095238095238093, "cc": -13.009001358504877, "cc_by_type": { "choice": 49.720670391061454, "noul": -100.0, "score": 11.252325533423912 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 0.0 }, "id": { "accuracy": 0.5, "cc": -22.222222222222225, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8666666666666667, "cc": 43.06232479155725, "cc_by_type": { "choice": 100.0, "noul": -33.333333333333336, "score": 62.5203077080051 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 43.06232479155725 }, "ja": { "accuracy": 0.4, "cc": -12.03076737823514, "cc_by_type": { "choice": 71.40587768069896, "noul": -80.0, "score": -27.498179815404388 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 0.0 }, "ko": { "accuracy": 0.3, "cc": -9.868365203541524, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": -29.605095610624566 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.7857142857142857, "cc": 65.30590654320713, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 62.584386296288066 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 65.30590654320713 }, "nl": { "accuracy": 0.3, "cc": -19.67037363341068, "cc_by_type": { "choice": 22.077922077922082, "noul": -100.0, "score": 18.910957021845874 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.16666666666666666, "cc": -32.25806451612903, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.6153846153846154, "cc": 26.281753907614387, "cc_by_type": { "choice": 69.53025814642403, "noul": -40.0, "score": 49.31500357641914 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 26.281753907614387 }, "pt": { "accuracy": 0.5, "cc": 13.128874563158115, "cc_by_type": { "choice": 36.35827931644077, "noul": -19.999999999999996, "score": 23.028344373033573 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 13.128874563158115 }, "sv": { "accuracy": 0.2727272727272727, "cc": -1.2116300515052056, "cc_by_type": { "choice": 23.404255319148945, "noul": -60.0, "score": 32.96085452633544 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.46153846153846156, "cc": 2.8260596941021654, "cc_by_type": { "choice": 34.78260869565218, "noul": -50.0, "score": 23.695570386654317 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 2.8260596941021654 }, "uk": { "accuracy": 0.5714285714285714, "cc": 62.8586809314782, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 53.092171826692656 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 62.8586809314782 }, "zh": { "accuracy": 0.2, "cc": -36.74882052545368, "cc_by_type": { "choice": -1.694915254237285, "noul": -33.333333333333336, "score": -75.21821298879041 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 } }, "type": { "choice": { "accuracy": 0.632, "cc": 52.46751755678448, "cc_by_type": { "choice": 52.46751755678448 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 52.46751755678448 }, "noul": { "accuracy": 0.24, "cc": -52.0, "cc_by_type": { "noul": -52.0 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.45066666666666666, "cc": 17.650777067912205, "cc_by_type": { "score": 17.650777067912205 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 17.650777067912205 } }, "usecase": { "advertising": { "accuracy": 0.3181818181818182, "cc": -14.075490238875036, "cc_by_type": { "choice": 36.0, "noul": -100.0, "score": 21.77352928337489 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.25, "cc": -16.706524469431017, "cc_by_type": { "choice": 44.099378881987576, "noul": -100.0, "score": 5.7810477097193775 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 0.0 }, "customer_support": { "accuracy": 0.5714285714285714, "cc": 20.95574183367787, "cc_by_type": { "choice": 51.21951219512195, "noul": -50.0, "score": 61.64771330591166 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 20.95574183367787 }, "demand_forecasting": { "accuracy": 0.17647058823529413, "cc": -32.07708838513613, "cc_by_type": { "choice": -5.263157894736844, "noul": -100.0, "score": 9.031892739328462 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.4117647058823529, "cc": -11.862692761495312, "cc_by_type": { "choice": 42.02898550724638, "noul": -100.0, "score": 22.38293620826768 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 0.0 }, "feature_extraction": { "accuracy": 0.5263157894736842, "cc": -11.447525637999656, "cc_by_type": { "choice": 62.264150943396224, "noul": -100.0, "score": 3.3932721426048085 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.4666666666666667, "cc": 4.739723573540634, "cc_by_type": { "choice": 50.98039215686275, "noul": -71.42857142857143, "score": 34.667349992330585 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 4.739723573540634 }, "gaming": { "accuracy": 0.2727272727272727, "cc": -19.95542954822079, "cc_by_type": { "choice": 16.37630662020906, "noul": -100.0, "score": 23.757404735128574 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.3333333333333333, "cc": 5.930574972450304, "cc_by_type": { "choice": 16.666666666666668, "noul": -60.0, "score": 61.12505825068424 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 5.930574972450304 }, "knowledge_graphs": { "accuracy": 0.36363636363636365, "cc": -0.20845375228950758, "cc_by_type": { "choice": 35.76017130620985, "noul": -66.66666666666667, "score": 30.281134103588304 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.36, "cc": 1.6257635079935728, "cc_by_type": { "choice": 65.11627906976744, "noul": -77.77777777777779, "score": 17.53878923199106 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 1.6257635079935728 }, "legal_compliance": { "accuracy": 0.4, "cc": -5.984212815810622, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0, "score": 46.5634905848262 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.48148148148148145, "cc": 10.444007621596686, "cc_by_type": { "choice": 76.96737044145875, "noul": -60.0, "score": 14.364652423331304 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 10.444007621596686 }, "model_routing": { "accuracy": 0.6082949308755761, "cc": 29.775167590225077, "cc_by_type": { "choice": 69.61038961038962, "noul": -6.060606060606055, "score": 25.77571922089167 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 29.775167590225077 }, "moderation_trust_safety": { "accuracy": 0.5555555555555556, "cc": -7.229669806205081, "cc_by_type": { "choice": 65.4510556621881, "noul": -100.0, "score": 12.859934919196657 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 0.0 }, "recruiting": { "accuracy": 0.6111111111111112, "cc": 42.58687011240332, "cc_by_type": { "choice": 49.860724233983284, "noul": 33.33333333333333, "score": 44.56655276989334 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 42.58687011240332 }, "risk_assessment": { "accuracy": 0.4444444444444444, "cc": -2.6708439262919, "cc_by_type": { "choice": 70.44334975369459, "noul": -100.0, "score": 21.54411846742971 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.391304347826087, "cc": -15.274838023083172, "cc_by_type": { "choice": 27.536231884057973, "noul": -100.0, "score": 26.639254046692518 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.625, "cc": 20.18266992417227, "cc_by_type": { "choice": 78.64768683274022, "noul": -50.0, "score": 31.900322939776594 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 20.18266992417227 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 78.66945762555974, 86.16700260543708 ], "I": [ 21.21147977626547, 28.038710817207104 ], "capability": [ 51.07742366989096, 56.10220985784996 ], "composite": { "A": null, "B": null, "C": null }, "seed": 20 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 84.35778065081391, "capability": 54.78583662123155, "intelligence": 25.21389259164919 }, "per_type": { "choice": { "accuracy": 0.632, "calibration": 84.17585650065509, "cc": 52.46751755678448, "cc_by_type": { "choice": 52.46751755678448 }, "cc_tiered": 57.805310036129775, "cc_tiered_open": 68.15046810038176, "cc_tiered_sealed": 47.46015197187779, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 52.46751755678448 }, "noul": { "accuracy": 0.24, "calibration": 81.95043662539807, "cc": -52.0, "cc_by_type": { "noul": -52.0 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.45066666666666666, "calibration": 86.9470488263886, "cc": 17.650777067912205, "cc_by_type": { "score": 17.650777067912205 }, "cc_tiered": 17.8363677388178, "cc_tiered_open": 19.23654465196982, "cc_tiered_sealed": 16.436190825665776, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 17.650777067912205 } }, "raw_on_A": { "calibration": 77.52017651507438, "capability": 50.95601359015508, "intelligence": 24.391850665235772 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": null, "Emphasis on Accuracy 60:20:20": null, "Emphasis on Cost 20:20:60": null, "Emphasis on Speed 20:60:20": null, "Intelligence only": null } }, { "addendum": null, "api_flag": true, "api_subset": false, "author": "Celeris", "axes": { "calibration": 88.35471243386242, "cost": 48.39765648346799, "intelligence": 58.890192862872325, "speed": 86.97194897947064 }, "calibration": { "parts": { "choice": { "ece_hard": 0.05393650793650812, "ece_score": 89.21269841269837, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 89.21269841269837 }, "noul": { "abstention_rate": 0.21333333333333337, "brier": 0.10485600000000002, "ece": 0.06373333333333334, "n": 375, "score": 87.25333333333334 }, "score": { "interval90_coverage": 0.9093333333333333, "interval90_width": 0.25799999999999995, "n": 375, "nrps": 0.07539788888888894, "score": 88.59810555555555, "top_ece": 0.07632000000000008 } }, "score": 88.35471243386242 }, "capability": 73.62245264836737, "composite_ci95": { "A": [ 59.826250772440964, 63.11252049889689 ], "B": [ 57.66895611973631, 62.268020909903846 ], "C": [ 47.40064507700766, 63.11252049889689 ] }, "cost": { "basis": "Celeris public price, USD 0.04/M input, output free (celeris.ai/pricing, fetched 16 Oct 2026 19:27Z, saved copy sha256 c3ed0fdc...; docs.celeris.ai/pricing rates table, saved copy sha256 ff4b658d..., byte-identical to the copy saved on 14 Oct 2026): \"Celeris-1 Decision $0.04 / M input tokens, Free output tokens\"; prepaid per-token billing with no subscription or commitment. Counted as the manufacturer's standard launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops, SubQ Snap, d1-ultra, Sage 1.4, Nimble and kettei), through a scoped re-freeze of the frozen pricing module that adds this one key and nothing else, with a zero-difference proof for every other registry entry. Workspace activation at Celeris is capacity-limited (an activation waitlist, as for SubQ Snap); free or promotional credit is not used. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (1,968,382 input tokens over the 1,500 answered items, 0 output tokens), divided by the full 1,500-row basis. Celeris counts input tokens its own way: per its docs, input_tokens \"covers everything the model reads for the request, which is more than your state and questions alone\"; on the same 1,500 items it reports about 3.0x the input tokens kettei-fat reports, so the cost reflects Celeris's own count. Not a bill and not a quote.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.05249018666666667 }, "display": "Celeris-1 Decision (Celeris)", "endpoint_condition": "Vendor-hosted inference.celeris.ai/celeris-1-decision System One API (POST /v1/systemone), model `celeris-1-decision`, through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `celeris` (no proxy) before the first request (custody #20414 under #19242). Native probabilities on all three question types; Celeris returns every probability at two-decimal precision (1,500 of 1,500 rows; 332 returned probabilities are exactly 0), scored as returned. The runner recorded celeris-1-decision as the answering model on all 1,500 rows (the adapter reads the response's model field). All 1,500 items answered with HTTP 200, including the 15 long items, which fit the documented 131,072-token context window (largest reported input 81,719 tokens). Paid public fast-lane order 70c4455b, run with the customer's own API key; hosting implementation and deployed weights were not independently inspected.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 62.630814168468525, "I_sealed": 55.14957155727612, "base": 58.890192862872325, "excess": -2.358284159163304, "gap": 7.481242611192407, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 75.52874311900909, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 100.00000000000001, "hard": 71.6892104435357, "judge": 76.26895187870798, "standard": 69.8618668899121 } }, "open|noul": { "cc": 54.45410628019323, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 73.91304347826086, "judge": 19.999999999999996, "standard": 44.44444444444444 } }, "open|score": { "cc": 57.909593106203275, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 70.42424242424244, "hard": 61.16666666666667, "judge": 58.91525423728814, "standard": 43.629629629629605 } }, "sealed|choice": { "cc": 74.18524524536124, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 96.18805590851335, "hard": 68.77529779114143, "judge": 72.67384916748287, "standard": 76.2708289390424 } }, "sealed|noul": { "cc": 40.147216653650005, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 58.62068965517242, "hard": 25.454545454545464, "judge": 41.53846153846155, "standard": 58.20895522388059 } }, "sealed|score": { "cc": 51.11625277281713, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 67.62280701754383, "hard": 50.50985915492958, "judge": 46.203125, "standard": 51.445454545454524 } } } }, "jevbench_score": 61.98812284280043, "key": "celeris-1-decision", "last_measured_on": "2026-10-16", "licence": "Proprietary hosted API (no weights release)", "listing": "ranked", "measured_in": "v1.6.25", "measurement_date_status": "measured", "model_pin": "hosted-api:inference.celeris.ai/celeris-1-decision/v1/systemone#celeris-1-decision", "noul_decisive": { "acc_among_decisive": 0.9254237288135593, "decisive": 295, "decisive_rate": 0.7866666666666666, "n": 375, "supported": true, "valid": 375 }, "paid_fast_lane": true, "planned_weight_denominator": 1.0, "rank": 11, "ranked": true, "ranks": { "A": 11, "B": 11, "C": 9, "capability": 13 }, "repo": "https://docs.celeris.ai", "scores": { "A": 61.98812284280043, "B": 60.49441852880002, "C": 59.71616886783639 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.4326209495193325, "p50_s_raw": 0.4326209495193325, "p95_s_adjusted": 0.46419189785665366, "p95_s_raw": 0.46419189785665366 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.9545454545454546, "cc": 88.55555555555556, "cc_by_type": { "choice": 91.66666666666667, "noul": 100.0, "score": 74.0 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 88.55555555555556 }, "calibration_gen": { "accuracy": 0.7029702970297029, "cc": 56.85813229056203, "cc_by_type": { "choice": 77.7306468716861, "noul": 36.00000000000001, "score": 56.84374999999999 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 56.85813229056203 }, "composite_gen": { "accuracy": 0.8613861386138614, "cc": 69.98257012484656, "cc_by_type": { "choice": 86.9918699186992, "noul": 48.14814814814814, "score": 74.80769230769232 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 69.98257012484656 }, "fairness": { "accuracy": 0.9375, "cc": 85.5925925925926, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 96.77777777777777 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 85.5925925925926 }, "jaggedness": { "accuracy": 0.6222222222222222, "cc": 36.09978545606696, "cc_by_type": { "choice": 59.57651588065449, "noul": 18.918918918918926, "score": 29.803921568627445 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 36.09978545606696 }, "llm_router": { "accuracy": 0.8083067092651757, "cc": 68.70445180122648, "cc_by_type": { "choice": 74.64119045522585, "noul": 69.0721649484536, "score": 62.40000000000001 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 68.70445180122648 }, "long_state": { "accuracy": 0.4666666666666667, "cc": -28.00537634408603, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0, "score": -19.50000000000003 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.5185185185185185, "cc": 14.397316314842081, "cc_by_type": { "choice": 38.14432989690721, "noul": -25.0, "score": 30.047619047619033 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 14.397316314842081 }, "multilingual": { "accuracy": 0.7163636363636363, "cc": 56.70515248138471, "cc_by_type": { "choice": 70.6553635474405, "noul": 41.66666666666667, "score": 57.793427230046944 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 56.70515248138471 }, "negation_pair_gen": { "accuracy": 0.7674418604651163, "cc": 50.21111111111111, "cc_by_type": { "choice": 93.33333333333333, "noul": 19.999999999999996, "score": 37.3 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 50.21111111111111 }, "option_order": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 100.0 }, "perturb_pair": { "accuracy": 0.859375, "cc": 82.92889042889043, "cc_by_type": { "choice": 78.97897897897899, "noul": 100.0, "score": 69.8076923076923 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 82.92889042889043 }, "usecase": { "accuracy": 0.7045454545454546, "cc": 52.40729411764707, "cc_by_type": { "choice": 73.03921568627452, "noul": 41.66666666666667, "score": 42.51600000000001 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 52.40729411764707 } }, "lang": { "ar": { "accuracy": 0.4375, "cc": 10.50061050061049, "cc_by_type": { "choice": 23.809523809523807, "noul": 50.0, "score": -42.307692307692335 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 10.50061050061049 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 0.6923076923076923, "cc": 46.74444444444443, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 6.899999999999984 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 46.74444444444443 }, "de": { "accuracy": 0.875, "cc": 87.51365909965371, "cc_by_type": { "choice": 76.10619469026548, "noul": 100.0, "score": 86.43478260869566 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 87.51365909965371 }, "el": { "accuracy": 0.6666666666666666, "cc": 54.66014897579143, "cc_by_type": { "choice": 66.4804469273743, "noul": 0.0, "score": 97.49999999999999 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 54.66014897579143 }, "en": { "accuracy": 0.7654213727193745, "cc": 59.11116852992231, "cc_by_type": { "choice": 76.71495344211078, "noul": 46.76258992805755, "score": 53.85596221959859 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 59.11116852992231 }, "es": { "accuracy": 0.7419354838709677, "cc": 59.20020471483863, "cc_by_type": { "choice": 80.42414355628058, "noul": 75.0, "score": 22.176470588235276 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 59.20020471483863 }, "fi": { "accuracy": 0.375, "cc": 22.102003642987246, "cc_by_type": { "choice": 1.6393442622950836, "noul": -33.333333333333336, "score": 97.99999999999999 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 22.102003642987246 }, "fr": { "accuracy": 0.8571428571428571, "cc": 84.71310629514964, "cc_by_type": { "choice": 78.82352941176471, "noul": 100.0, "score": 75.31578947368422 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 84.71310629514964 }, "hi": { "accuracy": 0.7619047619047619, "cc": 51.663407821229065, "cc_by_type": { "choice": 83.24022346368716, "noul": 25.0, "score": 46.750000000000014 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 51.663407821229065 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8, "cc": 71.551683183016, "cc_by_type": { "choice": 71.94076383476228, "noul": 100.0, "score": 42.71428571428573 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 71.551683183016 }, "ja": { "accuracy": 0.64, "cc": 43.147383284793925, "cc_by_type": { "choice": 57.108816521048446, "noul": 39.99999999999999, "score": 32.33333333333333 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 43.147383284793925 }, "ko": { "accuracy": 0.7, "cc": 23.642857142857128, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": 70.9285714285714 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 23.642857142857128 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 77.16666666666667, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 98.16666666666667 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 77.16666666666667 }, "nl": { "accuracy": 0.6, "cc": 33.964213564213566, "cc_by_type": { "choice": 74.02597402597404, "noul": -33.333333333333336, "score": 61.19999999999999 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 33.964213564213566 }, "no": { "accuracy": 0.6666666666666666, "cc": 42.74193548387097, "cc_by_type": { "choice": 35.483870967741936, "noul": 50.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 42.74193548387097 }, "pl": { "accuracy": 0.717948717948718, "cc": 56.32830097071453, "cc_by_type": { "choice": 77.14769360981802, "noul": 39.99999999999999, "score": 51.83720930232556 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 56.32830097071453 }, "pt": { "accuracy": 0.6428571428571429, "cc": 50.6765829642443, "cc_by_type": { "choice": 46.96523276370065, "noul": 60.00000000000001, "score": 45.06451612903225 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 50.6765829642443 }, "sv": { "accuracy": 0.7272727272727273, "cc": 63.15602836879433, "cc_by_type": { "choice": 74.46808510638299, "noul": 60.00000000000001, "score": 55.00000000000001 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 63.15602836879433 }, "tr": { "accuracy": 0.7692307692307693, "cc": 65.375, "cc_by_type": { "choice": 100.0, "noul": 50.0, "score": 46.12500000000001 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 65.375 }, "uk": { "accuracy": 0.7142857142857143, "cc": 75.99462365591397, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 92.49999999999999 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 75.99462365591397 }, "zh": { "accuracy": 0.8, "cc": 46.111111111111114, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 5.000000000000004 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 46.111111111111114 } }, "type": { "choice": { "accuracy": 0.8093333333333333, "cc": 75.37266308195717, "cc_by_type": { "choice": 75.37266308195717 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 75.37266308195717 }, "noul": { "accuracy": 0.728, "cc": 45.599999999999994, "cc_by_type": { "noul": 45.599999999999994 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 45.599999999999994 }, "score": { "accuracy": 0.6693333333333333, "cc": 53.62184115523472, "cc_by_type": { "score": 53.62184115523472 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 53.62184115523472 } }, "usecase": { "advertising": { "accuracy": 0.45454545454545453, "cc": -3.3333333333333335, "cc_by_type": { "choice": 84.0, "noul": -100.0, "score": 5.999999999999995 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.75, "cc": 60.539178213091255, "cc_by_type": { "choice": 62.73291925465839, "noul": 100.0, "score": 18.884615384615376 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 60.539178213091255 }, "customer_support": { "accuracy": 0.7857142857142857, "cc": 53.159891598916, "cc_by_type": { "choice": 67.47967479674797, "noul": 50.0, "score": 41.99999999999999 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 53.159891598916 }, "demand_forecasting": { "accuracy": 0.23529411764705882, "cc": -5.883528265107214, "cc_by_type": { "choice": 6.432748538011691, "noul": -33.333333333333336, "score": 9.250000000000004 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.7058823529411765, "cc": 38.08121980676328, "cc_by_type": { "choice": 71.0144927536232, "noul": -33.333333333333336, "score": 76.56249999999999 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 38.08121980676328 }, "feature_extraction": { "accuracy": 0.7894736842105263, "cc": 25.78092243186579, "cc_by_type": { "choice": 74.84276729559748, "noul": 100.0, "score": -97.5000000000001 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 25.78092243186579 }, "financial_crime": { "accuracy": 0.5333333333333333, "cc": 34.35667600373482, "cc_by_type": { "choice": 60.784313725490186, "noul": 14.28571428571428, "score": 27.999999999999993 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 34.35667600373482 }, "gaming": { "accuracy": 0.5909090909090909, "cc": 25.161793667626117, "cc_by_type": { "choice": 79.09407665505228, "noul": -50.0, "score": 46.39130434782608 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 25.161793667626117 }, "insurance_claims": { "accuracy": 0.3333333333333333, "cc": -0.9074074074074095, "cc_by_type": { "choice": 37.5, "noul": -60.0, "score": 19.77777777777777 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.5909090909090909, "cc": 35.525944671554946, "cc_by_type": { "choice": 48.608137044967876, "noul": 33.33333333333333, "score": 24.636363636363644 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 35.525944671554946 }, "lead_generation": { "accuracy": 0.76, "cc": 53.53648892020985, "cc_by_type": { "choice": 88.3720930232558, "noul": 55.55555555555556, "score": 16.681818181818187 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 53.53648892020985 }, "legal_compliance": { "accuracy": 0.7, "cc": 61.327956989247305, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 48.499999999999986 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 61.327956989247305 }, "llm_guardrails": { "accuracy": 0.6666666666666666, "cc": 64.07245681381958, "cc_by_type": { "choice": 76.96737044145875, "noul": 60.00000000000001, "score": 55.24999999999998 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 64.07245681381958 }, "model_routing": { "accuracy": 0.8755760368663594, "cc": 75.8801857864358, "cc_by_type": { "choice": 89.48051948051949, "noul": 87.87878787878789, "score": 50.28125000000002 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 75.8801857864358 }, "moderation_trust_safety": { "accuracy": 0.7777777777777778, "cc": 19.180836250047037, "cc_by_type": { "choice": 88.48368522072937, "noul": -100.0, "score": 69.05882352941175 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 19.180836250047037 }, "recruiting": { "accuracy": 0.7222222222222222, "cc": 58.28367021403233, "cc_by_type": { "choice": 83.28690807799443, "noul": 33.33333333333333, "score": 58.23076923076924 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 58.28367021403233 }, "risk_assessment": { "accuracy": 0.5555555555555556, "cc": 27.49618468076886, "cc_by_type": { "choice": 55.665024630541865, "noul": 0.0, "score": 26.823529411764714 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 27.49618468076886 }, "scientific_discovery": { "accuracy": 0.7391304347826086, "cc": 54.415942028985505, "cc_by_type": { "choice": 71.01449275362319, "noul": 33.33333333333333, "score": 58.9 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 54.415942028985505 }, "search_retrieval": { "accuracy": 0.9583333333333334, "cc": 94.34210526315788, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 83.02631578947367 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 94.34210526315788 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 83.48768643386241, 89.93833656481475 ], "I": [ 53.294216021322725, 63.07132903126318 ], "capability": [ 69.15361998514439, 75.62243550116447 ], "composite": { "A": [ 59.826250772440964, 63.11252049889689 ], "B": [ 57.66895611973631, 62.268020909903846 ], "C": [ 47.40064507700766, 63.11252049889689 ] }, "seed": 21 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 88.35471243386242, "capability": 73.62245264836737, "intelligence": 58.890192862872325 }, "per_type": { "choice": { "accuracy": 0.8093333333333333, "calibration": 89.21269841269837, "cc": 75.37266308195717, "cc_by_type": { "choice": 75.37266308195717 }, "cc_tiered": 74.85699418218516, "cc_tiered_open": 75.52874311900909, "cc_tiered_sealed": 74.18524524536124, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 75.37266308195717 }, "noul": { "accuracy": 0.728, "calibration": 87.25333333333334, "cc": 45.599999999999994, "cc_by_type": { "noul": 45.599999999999994 }, "cc_tiered": 47.30066146692162, "cc_tiered_open": 54.45410628019323, "cc_tiered_sealed": 40.147216653650005, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 45.599999999999994 }, "score": { "accuracy": 0.6693333333333333, "calibration": 88.59810555555555, "cc": 53.62184115523472, "cc_by_type": { "score": 53.62184115523472 }, "cc_tiered": 54.512922939510204, "cc_tiered_open": 57.909593106203275, "cc_tiered_sealed": 51.11625277281713, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 53.62184115523472 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 57.198815857984826, "Emphasis on Accuracy 60:20:20": 56.37222398373547, "Emphasis on Cost 20:20:60": 51.78418767887396, "Emphasis on Speed 20:60:20": 64.94142661662302, "Intelligence only": 55.17618045386969 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "vLLM Semantic Router Team (vllm-sr)", "axes": { "calibration": 87.47631981746561, "cost": 48.581027833630024, "intelligence": 35.82270435669186, "speed": 89.93254869015362 }, "calibration": { "parts": { "choice": { "ece_hard": 0.05090298449957041, "ece_score": 89.81940310008592, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 89.81940310008592 }, "noul": { "abstention_rate": 0.46648793565683644, "brier": 0.14462735182051856, "ece": 0.08496977140192581, "n": 373, "score": 83.00604571961485 }, "score": { "interval90_coverage": 0.9731182795698925, "interval90_width": 0.5766129032258064, "n": 372, "nrps": 0.09926110585728637, "score": 89.60351063269607, "top_ece": 0.054334340744396026 } }, "score": 87.47631981746561 }, "capability": 61.64951208707873, "composite_ci95": { "A": [ 19.99633761602885, 35.36027322164116 ], "B": [ 17.58558128220527, 32.356474460814916 ], "C": [ 13.886345566686703, 24.555745292806353 ] }, "cost": { "basis": "ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: lineage base Qwen/Qwen3.5-9B (the card's base vllm-sr/Decision-1.0-Lux-9B is itself a fine-tune and RAISES in the module; Qwen3.5-9B lineage verified by config comparison in the earlier v1.5 measurement) -> qwen/qwen3.5-9b USD 0.09999999999999999/M input, USD 0.15/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens as reported by the author's runtime (output_tokens 0, nothing generated).", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.0517566 }, "display": "Decision 2.0 Lux 9B (vLLM-SR, Qwen3.5-9B lineage)", "endpoint_condition": "Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Authors' own in-process runtime behind the evaluator's loopback shim (verbatim from the earlier v1.5 measurement), serial requests.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 41.41506937866855, "I_sealed": 30.230339334715172, "base": 35.82270435669186, "excess": 1.3452032735976651, "gap": 11.184730043953376, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 65.97342471411105, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 76.64864864864865, "hard": 64.1396665618119, "judge": 72.31377719182598, "standard": 54.79280033486815 } }, "open|noul": { "cc": 10.641821946169772, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 57.14285714285714, "hard": -4.347826086956519, "judge": 0.0, "standard": 33.33333333333333 } }, "open|score": { "cc": 47.62996147572483, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 57.01722460619536, "hard": 42.15889431879225, "judge": 48.06315937576442, "standard": 53.22866737429535 } }, "sealed|choice": { "cc": 59.49041617562122, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 75.22236340533672, "hard": 53.16294668671215, "judge": 55.92556317335946, "standard": 69.62666104197426 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 24.13793103448276, "hard": -9.090909090909093, "judge": -41.53846153846153, "standard": 16.417910447761198 } }, "sealed|score": { "cc": 31.200601828524306, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 52.52957611300084, "hard": 27.31494795658824, "judge": 23.847538197285967, "standard": 39.33701787701567 } } } }, "jevbench_score": 27.281155601129864, "key": "decision2-lux-9b", "last_measured_on": "2026-10-16", "licence": "Apache-2.0", "listing": "ranked", "measured_in": "v1.6.24", "measurement_date_status": "measured", "model_pin": "vllm-sr/Decision-2.0-Lux-9B@78bf3c03d9147aeb30b641edfe0e30ed04887ca5", "noul_decisive": { "acc_among_decisive": 0.9447236180904522, "decisive": 199, "decisive_rate": 0.5335120643431636, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": 17, "ranked": true, "ranks": { "A": 17, "B": 17, "C": 18, "capability": 23 }, "repo": "https://huggingface.co/vllm-sr/Decision-2.0-Lux-9B", "scores": { "A": 27.281155601129864, "B": 24.482458207210655, "C": 18.945246945229066 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.2866953719989397, "p50_s_raw": 0.06834768599946983, "p95_s_adjusted": 0.3542619008774636, "p95_s_raw": 0.10213095043873181 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.8181818181818182, "cc": 65.38134362243356, "cc_by_type": { "choice": 66.66666666666667, "noul": 100.0, "score": 29.477364200633982 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 65.38134362243356 }, "calibration_gen": { "accuracy": 0.4158415841584158, "cc": 4.099569847308422, "cc_by_type": { "choice": 30.010604453870634, "noul": -28.000000000000004, "score": 10.288105088054634 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 4.099569847308422 }, "composite_gen": { "accuracy": 0.6237623762376238, "cc": 26.33004954263009, "cc_by_type": { "choice": 77.23577235772358, "noul": -48.14814814814815, "score": 49.90252441831484 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 26.33004954263009 }, "fairness": { "accuracy": 0.8125, "cc": 62.654116824285275, "cc_by_type": { "choice": 100.0, "noul": 19.999999999999996, "score": 67.96235047285583 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 62.654116824285275 }, "jaggedness": { "accuracy": 0.4, "cc": -3.3916129517076663, "cc_by_type": { "choice": 30.702598652550538, "noul": -56.75675675675676, "score": 15.879319249083224 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.8146964856230032, "cc": 67.14156045601494, "cc_by_type": { "choice": 77.01857885004843, "noul": 67.01030927835052, "score": 57.395793239645876 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 67.14156045601494 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.4074074074074074, "cc": -11.681952585069832, "cc_by_type": { "choice": 50.51546391752578, "noul": -100.0, "score": 14.43867832726472 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.6363636363636364, "cc": 33.066005227920634, "cc_by_type": { "choice": 71.63351809585916, "noul": -11.111111111111116, "score": 38.67560869901385 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 33.066005227920634 }, "negation_pair_gen": { "accuracy": 0.4883720930232558, "cc": 2.67222418279357, "cc_by_type": { "choice": 56.66666666666668, "noul": -60.0, "score": 11.350005881714031 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 2.67222418279357 }, "option_order": { "accuracy": 0.8958333333333334, "cc": 86.68442077230361, "cc_by_type": { "choice": 86.68442077230361 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 86.68442077230361 }, "perturb_pair": { "accuracy": 0.8125, "cc": 74.77426699693255, "cc_by_type": { "choice": 72.97297297297297, "noul": 80.0, "score": 71.3498280178247 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 74.77426699693255 }, "usecase": { "accuracy": 0.509090909090909, "cc": 26.66485457151172, "cc_by_type": { "choice": 47.30392156862747, "noul": -4.1666666666666625, "score": 36.85730881257436 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 26.66485457151172 } }, "lang": { "ar": { "accuracy": 0.5, "cc": 9.056945780385973, "cc_by_type": { "choice": 42.857142857142854, "noul": 0.0, "score": -15.686305515984934 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 9.056945780385973 }, "cs": { "accuracy": 0.7142857142857143, "cc": 78.94736842105263, "cc_by_type": { "choice": 57.89473684210526, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 78.94736842105263 }, "da": { "accuracy": 0.6153846153846154, "cc": 17.326598840370945, "cc_by_type": { "choice": 58.52534562211981, "noul": -33.333333333333336, "score": 26.787784232326363 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 17.326598840370945 }, "de": { "accuracy": 0.8125, "cc": 67.11320598246611, "cc_by_type": { "choice": 76.10619469026548, "noul": 75.0, "score": 50.233423257132834 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 67.11320598246611 }, "el": { "accuracy": 0.5333333333333333, "cc": 33.09357426464621, "cc_by_type": { "choice": 66.4804469273743, "noul": -66.66666666666667, "score": 99.46694253323099 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 33.09357426464621 }, "en": { "accuracy": 0.6046915725456126, "cc": 31.4531064905039, "cc_by_type": { "choice": 58.701992897328545, "noul": 0.0, "score": 35.657326574183145 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 31.4531064905039 }, "es": { "accuracy": 0.7741935483870968, "cc": 63.51130608648939, "cc_by_type": { "choice": 80.42414355628058, "noul": 100.0, "score": 10.10977470318759 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 63.51130608648939 }, "fi": { "accuracy": 0.5, "cc": 19.74093935418468, "cc_by_type": { "choice": 67.21311475409836, "noul": -33.333333333333336, "score": 25.34303664178902 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 19.74093935418468 }, "fr": { "accuracy": 0.7142857142857143, "cc": 40.99009270088603, "cc_by_type": { "choice": 57.64705882352941, "noul": 0.0, "score": 65.32321927912866 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 40.99009270088603 }, "hi": { "accuracy": 0.5714285714285714, "cc": 18.916283367574735, "cc_by_type": { "choice": 83.24022346368716, "noul": -50.0, "score": 23.508626639037043 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 18.916283367574735 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8666666666666667, "cc": 74.61768112255443, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 23.853043367663275 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 74.61768112255443 }, "ja": { "accuracy": 0.48, "cc": 14.22770772278479, "cc_by_type": { "choice": 28.514694201747414, "noul": -19.999999999999996, "score": 34.16842896660695 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 14.22770772278479 }, "ko": { "accuracy": 0.5, "cc": -9.761328844855484, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0, "score": 15.160457909878 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.8571428571428571, "cc": 72.09322796696891, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 82.9463505675734 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 72.09322796696891 }, "nl": { "accuracy": 0.6, "cc": 9.435501837148886, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": 28.306505511446655 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 9.435501837148886 }, "no": { "accuracy": 0.3333333333333333, "cc": -7.258064516129032, "cc_by_type": { "choice": 35.483870967741936, "noul": -50.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.6410256410256411, "cc": 39.71775347984494, "cc_by_type": { "choice": 92.382564536606, "noul": -19.999999999999996, "score": 46.77069590292883 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 39.71775347984494 }, "pt": { "accuracy": 0.75, "cc": 66.8390486833019, "cc_by_type": { "choice": 57.57218621096051, "noul": 100.0, "score": 42.94495983894516 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 66.8390486833019 }, "sv": { "accuracy": 0.2727272727272727, "cc": 3.6562373029894957, "cc_by_type": { "choice": 23.404255319148945, "noul": -60.0, "score": 47.564456589819535 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 3.6562373029894957 }, "tr": { "accuracy": 0.6923076923076923, "cc": 24.52332791887444, "cc_by_type": { "choice": 78.26086956521739, "noul": 50.0, "score": -54.69088580859407 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 24.52332791887444 }, "uk": { "accuracy": 0.7142857142857143, "cc": 69.66005480942651, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 41.23822894440856 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 69.66005480942651 }, "zh": { "accuracy": 0.7, "cc": 6.234913945334711, "cc_by_type": { "choice": 100.0, "noul": -33.333333333333336, "score": -47.961924830662525 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 6.234913945334711 } }, "type": { "choice": { "accuracy": 0.7026666666666667, "cc": 61.595131938996154, "cc_by_type": { "choice": 61.595131938996154 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 61.595131938996154 }, "noul": { "accuracy": 0.5013333333333333, "cc": 0.2666666666666595, "cc_by_type": { "noul": 0.2666666666666595 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.2666666666666595 }, "score": { "accuracy": 0.5573333333333333, "cc": 35.76324809476839, "cc_by_type": { "score": 35.76324809476839 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 35.76324809476839 } }, "usecase": { "advertising": { "accuracy": 0.3181818181818182, "cc": -21.716999577358354, "cc_by_type": { "choice": 52.0, "noul": -100.0, "score": -17.150998732075063 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.5416666666666666, "cc": 30.171019428261456, "cc_by_type": { "choice": 62.73291925465839, "noul": 11.111111111111116, "score": 16.669027919014855 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 30.171019428261456 }, "customer_support": { "accuracy": 0.9285714285714286, "cc": 66.97464414728488, "cc_by_type": { "choice": 100.0, "noul": 50.0, "score": 50.923932441854646 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 66.97464414728488 }, "demand_forecasting": { "accuracy": 0.29411764705882354, "cc": -15.449793509057187, "cc_by_type": { "choice": 6.432748538011691, "noul": -33.333333333333336, "score": -19.44879573184992 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5294117647058824, "cc": 23.250864501749692, "cc_by_type": { "choice": 71.0144927536232, "noul": -33.333333333333336, "score": 32.071434084959215 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 23.250864501749692 }, "feature_extraction": { "accuracy": 0.2631578947368421, "cc": -65.29273959291613, "cc_by_type": { "choice": 11.949685534591195, "noul": -66.66666666666667, "score": -141.16123764667293 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5, "cc": 21.627541374440955, "cc_by_type": { "choice": 70.58823529411765, "noul": -42.85714285714286, "score": 37.15153168634807 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 21.627541374440955 }, "gaming": { "accuracy": 0.4090909090909091, "cc": 10.26483205645294, "cc_by_type": { "choice": 26.82926829268293, "noul": -50.0, "score": 53.96522787667589 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 10.26483205645294 }, "insurance_claims": { "accuracy": 0.3333333333333333, "cc": 9.522250670781048, "cc_by_type": { "choice": 37.5, "noul": -60.0, "score": 51.066752012343144 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 9.522250670781048 }, "knowledge_graphs": { "accuracy": 0.5, "cc": 21.435076662665292, "cc_by_type": { "choice": 35.76017130620985, "noul": 0.0, "score": 28.545058681786028 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 21.435076662665292 }, "lead_generation": { "accuracy": 0.56, "cc": 33.301792034961935, "cc_by_type": { "choice": 88.3720930232558, "noul": -11.111111111111116, "score": 22.644394192741114 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 33.301792034961935 }, "legal_compliance": { "accuracy": 0.3, "cc": -31.99789804370859, "cc_by_type": { "choice": -29.032258064516128, "noul": -33.333333333333336, "score": -33.62810273327632 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.5185185185185185, "cc": 25.452132963671094, "cc_by_type": { "choice": 65.4510556621881, "noul": -19.999999999999996, "score": 30.90534322882519 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 25.452132963671094 }, "model_routing": { "accuracy": 0.8341013824884793, "cc": 67.04627110748747, "cc_by_type": { "choice": 84.80519480519482, "noul": 72.72727272727273, "score": 43.60634578999484 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 67.04627110748747 }, "moderation_trust_safety": { "accuracy": 0.6111111111111112, "cc": 6.899543267261227, "cc_by_type": { "choice": 53.93474088291747, "noul": -100.0, "score": 66.76388891886621 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 6.899543267261227 }, "recruiting": { "accuracy": 0.6111111111111112, "cc": 50.511962779197546, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 51.62873884827046 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 50.511962779197546 }, "risk_assessment": { "accuracy": 0.4444444444444444, "cc": 1.2750601937910584, "cc_by_type": { "choice": 70.44334975369459, "noul": -50.0, "score": -16.618169172321416 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 1.2750601937910584 }, "scientific_discovery": { "accuracy": 0.4782608695652174, "cc": 12.33326926457949, "cc_by_type": { "choice": 27.536231884057973, "noul": -33.333333333333336, "score": 42.79690924301383 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 12.33326926457949 }, "search_retrieval": { "accuracy": 0.9166666666666666, "cc": 91.439262104532, "cc_by_type": { "choice": 89.3238434163701, "noul": 100.0, "score": 84.99394289722593 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 91.439262104532 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 81.27359148468967, 87.45181415099808 ], "I": [ 31.570921248305364, 40.02833941226646 ], "capability": [ 57.271187540384254, 62.826327947917356 ], "composite": { "A": [ 19.99633761602885, 35.36027322164116 ], "B": [ 17.58558128220527, 32.356474460814916 ], "C": [ 13.886345566686703, 24.555745292806353 ] }, "seed": 22 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 87.47631981746561, "capability": 61.64951208707873, "intelligence": 35.82270435669186 }, "per_type": { "choice": { "accuracy": 0.7026666666666667, "calibration": 89.81940310008592, "cc": 61.595131938996154, "cc_by_type": { "choice": 61.595131938996154 }, "cc_tiered": 62.731920444866134, "cc_tiered_open": 65.97342471411105, "cc_tiered_sealed": 59.49041617562122, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 61.595131938996154 }, "noul": { "accuracy": 0.5013333333333333, "calibration": 83.00604571961485, "cc": 0.2666666666666595, "cc_by_type": { "noul": 0.2666666666666595 }, "cc_tiered": 5.320910973084886, "cc_tiered_open": 10.641821946169772, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.2666666666666595 }, "score": { "accuracy": 0.5573333333333333, "calibration": 89.60351063269607, "cc": 35.76324809476839, "cc_by_type": { "score": 35.76324809476839 }, "cc_tiered": 39.41528165212457, "cc_tiered_open": 47.62996147572483, "cc_tiered_sealed": 31.200601828524306, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 35.76324809476839 } }, "raw_on_A": { "calibration": 81.01141299701578, "capability": 59.06449579734621, "intelligence": 37.11757859767665 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 24.38415108691931, "Emphasis on Accuracy 60:20:20": 20.986910304488326, "Emphasis on Cost 20:20:60": 24.040014660535597, "Emphasis on Speed 20:60:20": 29.599220607376115, "Intelligence only": 17.359151250992237 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "mohit67890", "axes": { "calibration": 86.77653179414553, "cost": 51.89260538310575, "intelligence": 21.160522003324633, "speed": 81.55302418750827 }, "calibration": { "parts": { "choice": { "ece_hard": 0.03246566765774192, "ece_score": 93.5068664684516, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 93.5068664684516 }, "noul": { "abstention_rate": 0.6246648793565683, "brier": 0.2028168457960074, "ece": 0.11038035259514468, "n": 373, "score": 77.92392948097107 }, "score": { "interval90_coverage": 0.9408602150537635, "interval90_width": 0.6883960573476698, "n": 372, "nrps": 0.1322491683106928, "score": 88.89879943301392, "top_ece": 0.044887421514514296 } }, "score": 86.77653179414553 }, "capability": 53.96852689873508, "composite_ci95": { "A": [ 4.765764179440689, 10.876577779842576 ], "B": [ 3.788068661906245, 9.124326060904862 ], "C": [ 3.3095584579449224, 7.553179013779569 ] }, "cost": { "basis": "ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-2B -> deepinfra:Qwen/Qwen3.5-2B USD 0.02/M input, USD 0.1/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens: input = 4-pass sum counted offline (see disclosures), output 0.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.040140226666666674 }, "display": "Imajev-2B (mohit67890, Qwen3.5-2B LoRA)", "endpoint_condition": "Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's own scripts/playground/server.py --backend torch, 4 option orders, 4,096-token request limit.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 24.18657872532247, "I_sealed": 18.134465281326797, "base": 21.160522003324633, "excess": -3.7874133263600385, "gap": 6.0521134439956725, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 49.471301249374704, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 76.64864864864865, "hard": 37.71626297577854, "judge": 52.537903757415954, "standard": 54.79280033486815 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": -42.85714285714286, "hard": -13.043478260869568, "judge": -60.0, "standard": -11.111111111111116 } }, "open|score": { "cc": 23.08843492659271, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 56.907855488528355, "hard": 14.056431441245543, "judge": 33.77791221517532, "standard": 8.208515683445306 } }, "sealed|choice": { "cc": 41.84862177374901, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 54.2566709021601, "hard": 42.23430091361164, "judge": 38.29578844270325, "standard": 40.20248892638684 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": -27.586206896551722, "hard": -49.09090909090909, "judge": -63.07692307692307, "standard": -7.462686567164178 } }, "sealed|score": { "cc": 12.554774070231389, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 26.48562308030278, "hard": 9.945290312389155, "judge": 9.940140408699815, "standard": 14.730267573177514 } } } }, "jevbench_score": 7.932621224877588, "key": "imajev-2b", "last_measured_on": "2026-10-16", "licence": "Apache-2.0 (adapter; base Apache-2.0)", "listing": "ranked", "measured_in": "v1.6.24", "measurement_date_status": "measured", "model_pin": "mohit67890/imajev-2b@66e4b8c808bae45ca98bed5edd2de6323bdb0576", "noul_decisive": { "acc_among_decisive": 0.8428571428571429, "decisive": 140, "decisive_rate": 0.3753351206434316, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": 28, "ranked": true, "ranks": { "A": 28, "B": 28, "C": 30, "capability": 29 }, "repo": "https://huggingface.co/mohit67890/imajev-2b", "scores": { "A": 7.932621224877588, "B": 6.509575518686488, "C": 5.508764739498324 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.7288981560035609, "p50_s_raw": 0.28944907800178044, "p95_s_adjusted": 0.9594685189804295, "p95_s_raw": 0.40473425949021474 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.7272727272727273, "cc": 27.940174431020967, "cc_by_type": { "choice": 66.66666666666667, "noul": 50.0, "score": -32.846143373603766 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 27.940174431020967 }, "calibration_gen": { "accuracy": 0.297029702970297, "cc": -17.5634975525963, "cc_by_type": { "choice": 14.103923647932142, "noul": -60.0, "score": -6.7944163057210405 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.43564356435643564, "cc": -4.661934044854566, "cc_by_type": { "choice": 38.211382113821145, "noul": -62.96296296296296, "score": 10.765778714578122 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 0.0 }, "fairness": { "accuracy": 0.6458333333333334, "cc": 52.590783388621844, "cc_by_type": { "choice": 100.0, "noul": 19.999999999999996, "score": 37.772350165865554 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 52.590783388621844 }, "jaggedness": { "accuracy": 0.3277777777777778, "cc": -7.850740882891453, "cc_by_type": { "choice": 17.70933589990377, "noul": -40.54054054054054, "score": -0.7210180080375883 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.5175718849840255, "cc": 18.658985210953194, "cc_by_type": { "choice": 51.65976930527428, "noul": -23.71134020618557, "score": 28.02852653377088 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 18.658985210953194 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.37037037037037035, "cc": -18.44398052388307, "cc_by_type": { "choice": 25.773195876288657, "noul": -75.0, "score": -6.105137447937858 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.5309090909090909, "cc": 16.31443135528656, "cc_by_type": { "choice": 56.9611998695794, "noul": -36.111111111111114, "score": 28.09320530739138 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 16.31443135528656 }, "negation_pair_gen": { "accuracy": 0.36046511627906974, "cc": -14.889109686015951, "cc_by_type": { "choice": 29.999999999999996, "noul": -73.33333333333334, "score": -1.3339957247145096 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.0 }, "option_order": { "accuracy": 0.5, "cc": 36.08521970705726, "cc_by_type": { "choice": 36.08521970705726 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 36.08521970705726 }, "perturb_pair": { "accuracy": 0.6875, "cc": 52.302256042037875, "cc_by_type": { "choice": 57.95795795795795, "noul": 39.99999999999999, "score": 58.948810168155674 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 52.302256042037875 }, "usecase": { "accuracy": 0.4727272727272727, "cc": 8.637486944515624, "cc_by_type": { "choice": 53.43137254901962, "noul": -45.833333333333336, "score": 18.31442161786059 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 8.637486944515624 } }, "lang": { "ar": { "accuracy": 0.25, "cc": -28.225026395455945, "cc_by_type": { "choice": 23.809523809523807, "noul": -50.0, "score": -58.48460299589164 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.7142857142857143, "cc": -10.526315789473685, "cc_by_type": { "choice": 78.94736842105263, "noul": -100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 0.0 }, "da": { "accuracy": 0.6153846153846154, "cc": 24.065748870216918, "cc_by_type": { "choice": 79.26267281105991, "noul": -33.333333333333336, "score": 26.267907132924172 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 24.065748870216918 }, "de": { "accuracy": 0.71875, "cc": 42.362310149525264, "cc_by_type": { "choice": 84.07079646017698, "noul": 0.0, "score": 43.016133988398806 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 42.362310149525264 }, "el": { "accuracy": 0.4, "cc": 19.38235434668214, "cc_by_type": { "choice": 32.960893854748605, "noul": -66.66666666666667, "score": 91.85283585196449 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 19.38235434668214 }, "en": { "accuracy": 0.44917463075586445, "cc": 5.074350761629012, "cc_by_type": { "choice": 40.24969185145407, "noul": -38.12949640287769, "score": 13.10285683631066 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 5.074350761629012 }, "es": { "accuracy": 0.7741935483870968, "cc": 46.5562811978586, "cc_by_type": { "choice": 70.63621533442088, "noul": 50.0, "score": 19.032628259154937 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 46.5562811978586 }, "fi": { "accuracy": 0.25, "cc": -10.63743944027403, "cc_by_type": { "choice": 34.42622950819673, "noul": -100.0, "score": 33.661452170981185 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 0.0 }, "fr": { "accuracy": 0.5714285714285714, "cc": 3.2263733482564874, "cc_by_type": { "choice": 57.64705882352941, "noul": -100.0, "score": 52.03206122124004 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 3.2263733482564874 }, "hi": { "accuracy": 0.42857142857142855, "cc": -10.475686348831847, "cc_by_type": { "choice": 49.720670391061454, "noul": -100.0, "score": 18.852270562443007 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 0.0 }, "id": { "accuracy": 0.5, "cc": -22.222222222222225, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.4666666666666667, "cc": -19.711858509492984, "cc_by_type": { "choice": 71.94076383476228, "noul": -100.0, "score": -31.076339363241235 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 0.0 }, "ja": { "accuracy": 0.64, "cc": 34.29447363096327, "cc_by_type": { "choice": 57.108816521048446, "noul": 19.999999999999996, "score": 25.77460437184137 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 34.29447363096327 }, "ko": { "accuracy": 0.5, "cc": -9.900639833853115, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0, "score": 14.742524942885105 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.6428571428571429, "cc": 58.50507211449983, "cc_by_type": { "choice": 83.33333333333333, "noul": 33.33333333333333, "score": 58.84854967683284 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 58.50507211449983 }, "nl": { "accuracy": 0.4, "cc": 3.7552735572627327, "cc_by_type": { "choice": 22.077922077922082, "noul": -33.333333333333336, "score": 22.521231927199448 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 3.7552735572627327 }, "no": { "accuracy": 0.16666666666666666, "cc": -32.25806451612903, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.5128205128205128, "cc": 16.728251852982496, "cc_by_type": { "choice": 77.14769360981802, "noul": -40.0, "score": 13.03706194912947 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 16.728251852982496 }, "pt": { "accuracy": 0.39285714285714285, "cc": 20.212949326324164, "cc_by_type": { "choice": 25.75132586918091, "noul": 19.999999999999996, "score": 14.887522109791584 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 20.212949326324164 }, "sv": { "accuracy": 0.2727272727272727, "cc": -0.25603227399320855, "cc_by_type": { "choice": 48.93617021276595, "noul": -100.0, "score": 50.29573296525442 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.6153846153846154, "cc": 28.348191067910236, "cc_by_type": { "choice": 34.78260869565218, "noul": 50.0, "score": 0.2619645080785271 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 28.348191067910236 }, "uk": { "accuracy": 0.5714285714285714, "cc": 3.1976636096176585, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0, "score": 74.10911986111104 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 3.1976636096176585 }, "zh": { "accuracy": 0.5, "cc": 8.401209830979823, "cc_by_type": { "choice": 49.152542372881356, "noul": -33.333333333333336, "score": 9.384420453391451 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 8.401209830979823 } }, "type": { "choice": { "accuracy": 0.5666666666666667, "cc": 44.02877973172085, "cc_by_type": { "choice": 44.02877973172085 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 44.02877973172085 }, "noul": { "accuracy": 0.31466666666666665, "cc": -37.06666666666667, "cc_by_type": { "noul": -37.06666666666667 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.42133333333333334, "cc": 15.622459826305535, "cc_by_type": { "score": 15.622459826305535 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 15.622459826305535 } }, "usecase": { "advertising": { "accuracy": 0.09090909090909091, "cc": -33.193862534650656, "cc_by_type": { "choice": 4.0, "noul": -100.0, "score": -3.5815876039519523 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.375, "cc": -6.089719732637479, "cc_by_type": { "choice": 44.099378881987576, "noul": -55.55555555555556, "score": -6.8129825243444575 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 0.0 }, "customer_support": { "accuracy": 0.9285714285714286, "cc": 62.03940267844437, "cc_by_type": { "choice": 100.0, "noul": 50.0, "score": 36.11820803533312 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 62.03940267844437 }, "demand_forecasting": { "accuracy": 0.23529411764705882, "cc": -37.42857709364494, "cc_by_type": { "choice": 18.128654970760234, "noul": -100.0, "score": -30.414386251695056 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5882352941176471, "cc": 10.87022836857408, "cc_by_type": { "choice": 56.52173913043478, "noul": -33.333333333333336, "score": 9.422279308620796 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 10.87022836857408 }, "feature_extraction": { "accuracy": 0.42105263157894735, "cc": -8.016826814669251, "cc_by_type": { "choice": 11.949685534591195, "noul": -33.333333333333336, "score": -2.6668326452656155 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5, "cc": 9.059322026345937, "cc_by_type": { "choice": 70.58823529411765, "noul": -71.42857142857143, "score": 28.018302213491587 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 9.059322026345937 }, "gaming": { "accuracy": 0.2727272727272727, "cc": -15.154690545329686, "cc_by_type": { "choice": 26.82926829268293, "noul": -100.0, "score": 27.706660071328017 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.26666666666666666, "cc": -11.501806643589282, "cc_by_type": { "choice": 37.5, "noul": -100.0, "score": 27.994580069232157 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.3181818181818182, "cc": -10.82046302383224, "cc_by_type": { "choice": 10.064239828693788, "noul": -33.333333333333336, "score": -9.192295566857167 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.44, "cc": -1.191894960640904, "cc_by_type": { "choice": 88.3720930232558, "noul": -77.77777777777779, "score": -14.170000127400728 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 0.0 }, "legal_compliance": { "accuracy": 0.4, "cc": -25.461994188704736, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0, "score": -11.869853533856144 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.37037037037037035, "cc": -0.9809923099182641, "cc_by_type": { "choice": 53.93474088291747, "noul": -60.0, "score": 3.1222821873277407 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 0.0 }, "model_routing": { "accuracy": 0.5576036866359447, "cc": 21.130055825784556, "cc_by_type": { "choice": 63.76623376623377, "noul": -24.242424242424242, "score": 23.86635795354414 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 21.130055825784556 }, "moderation_trust_safety": { "accuracy": 0.4444444444444444, "cc": -13.619363650781176, "cc_by_type": { "choice": 42.41842610364683, "noul": -100.0, "score": 16.72348294400964 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 0.0 }, "recruiting": { "accuracy": 0.7777777777777778, "cc": 53.4817305604329, "cc_by_type": { "choice": 83.28690807799443, "noul": 33.33333333333333, "score": 43.824950269970955 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 53.4817305604329 }, "risk_assessment": { "accuracy": 0.4444444444444444, "cc": -1.6382323766226665, "cc_by_type": { "choice": 70.44334975369459, "noul": -100.0, "score": 24.64195311643741 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.43478260869565216, "cc": 9.317909661738637, "cc_by_type": { "choice": 27.536231884057973, "noul": -33.333333333333336, "score": 33.75083043449128 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 9.317909661738637 }, "search_retrieval": { "accuracy": 0.5833333333333334, "cc": 21.221925502780437, "cc_by_type": { "choice": 67.97153024911032, "noul": -50.0, "score": 45.694246259230994 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 21.221925502780437 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 79.43421288501723, 86.71206265736095 ], "I": [ 17.326649934333265, 24.037678700610133 ], "capability": [ 49.531033607629034, 54.375113736460825 ], "composite": { "A": [ 4.765764179440689, 10.876577779842576 ], "B": [ 3.788068661906245, 9.124326060904862 ], "C": [ 3.3095584579449224, 7.553179013779569 ] }, "seed": 23 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 86.77653179414553, "capability": 53.96852689873508, "intelligence": 21.160522003324633 }, "per_type": { "choice": { "accuracy": 0.5666666666666667, "calibration": 93.5068664684516, "cc": 44.02877973172085, "cc_by_type": { "choice": 44.02877973172085 }, "cc_tiered": 45.65996151156186, "cc_tiered_open": 49.471301249374704, "cc_tiered_sealed": 41.84862177374901, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 44.02877973172085 }, "noul": { "accuracy": 0.31466666666666665, "calibration": 77.92392948097107, "cc": -37.06666666666667, "cc_by_type": { "noul": -37.06666666666667 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.42133333333333334, "calibration": 88.89879943301392, "cc": 15.622459826305535, "cc_by_type": { "score": 15.622459826305535 }, "cc_tiered": 17.82160449841205, "cc_tiered_open": 23.08843492659271, "cc_tiered_sealed": 12.554774070231389, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 15.622459826305535 } }, "raw_on_A": { "calibration": 83.85821528875951, "capability": 52.090117601921726, "intelligence": 20.322019915083942 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 6.8196331060835575, "Emphasis on Accuracy 60:20:20": 5.167366048927712, "Emphasis on Cost 20:20:60": 7.632524424872417, "Emphasis on Speed 20:60:20": 8.66807101272237, "Intelligence only": 3.789999234949207 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "mohit67890", "axes": { "calibration": 88.10935553643324, "cost": 30.923505253025198, "intelligence": 36.7327337780622, "speed": 79.48842001125843 }, "calibration": { "parts": { "choice": { "ece_hard": 0.05597354899568825, "ece_score": 88.80529020086236, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 88.80529020086236 }, "noul": { "abstention_rate": 0.4584450402144772, "brier": 0.13920542517421816, "ece": 0.08183623766154813, "n": 373, "score": 83.63275246769038 }, "score": { "interval90_coverage": 0.967741935483871, "interval90_width": 0.5192652329749106, "n": 372, "nrps": 0.09346155028027302, "score": 91.89002394074701, "top_ece": 0.034368985452393384 } }, "score": 88.10935553643324 }, "capability": 62.42104465724772, "composite_ci95": { "A": [ 7.365294687696036, 12.818089044625026 ], "B": [ 6.817840903770886, 12.323734305732241 ], "C": [ 5.114787977566692, 8.901450725434048 ] }, "cost": { "basis": "ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-9B -> qwen/qwen3.5-9b USD 0.09999999999999999/M input, USD 0.15/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens: input = 4-pass sum counted offline (see disclosures), output 0. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.200701 per 1,000 decisions is 3.11x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 precedent) and falls outside the official-caps board views by construction.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.20070113333333328 }, "display": "Imajev-9B (mohit67890, Qwen3.5-9B LoRA)", "endpoint_condition": "Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's own scripts/playground/server.py --backend torch, 4 option orders, 4,096-token request limit.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 41.23991434770264, "I_sealed": 32.22555320842176, "base": 36.7327337780622, "excess": -0.8251656310748317, "gap": 9.01436113928088, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 67.89294454901062, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 88.32432432432431, "hard": 62.25228059138094, "judge": 72.31377719182598, "standard": 62.32733361239012 } }, "open|noul": { "cc": 13.632160110420985, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 57.14285714285714, "hard": 21.739130434782616, "judge": -9.999999999999998, "standard": 11.111111111111116 } }, "open|score": { "cc": 42.19463838367629, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 48.06511169132162, "hard": 33.49205778853101, "judge": 43.74474924939493, "standard": 54.33939662156624 } }, "sealed|choice": { "cc": 61.96527273043063, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 73.3163913595934, "hard": 54.72418179715507, "judge": 62.09598432908915, "standard": 70.57582788441259 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 20.68965517241379, "hard": -9.090909090909093, "judge": -38.46153846153846, "standard": 13.432835820895516 } }, "sealed|score": { "cc": 34.71138689483466, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 55.85407978179955, "hard": 30.051996837602136, "judge": 30.53981630846708, "standard": 39.716176445368625 } } } }, "jevbench_score": 9.890575407965947, "key": "imajev-9b", "last_measured_on": "2026-10-16", "licence": "Apache-2.0 (adapter; base Apache-2.0)", "listing": "ranked", "measured_in": "v1.6.24", "measurement_date_status": "measured", "model_pin": "mohit67890/imajev-9b@7301600591665b941d324b1a297e097b56724f9b", "noul_decisive": { "acc_among_decisive": 0.9257425742574258, "decisive": 202, "decisive_rate": 0.5415549597855228, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": 26, "ranked": true, "ranks": { "A": 26, "B": 26, "C": 26, "capability": 22 }, "repo": "https://huggingface.co/mohit67890/imajev-9b", "scores": { "A": 9.890575407965947, "B": 9.32323661454407, "C": 6.868455144420794 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.9162941940070596, "p50_s_raw": 0.3831470970035298, "p95_s_adjusted": 1.227787094612722, "p95_s_raw": 0.5388935473063611 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.7727272727272727, "cc": 52.430958850575394, "cc_by_type": { "choice": 66.66666666666667, "noul": 100.0, "score": -9.373790114940505 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 52.430958850575394 }, "calibration_gen": { "accuracy": 0.46534653465346537, "cc": 5.028160919179807, "cc_by_type": { "choice": 42.73594909862143, "noul": -36.0, "score": 8.34853365891799 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 5.028160919179807 }, "composite_gen": { "accuracy": 0.5841584158415841, "cc": 19.016879555076702, "cc_by_type": { "choice": 67.47967479674797, "noul": -48.14814814814815, "score": 37.7191120166303 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 19.016879555076702 }, "fairness": { "accuracy": 0.8125, "cc": 77.21353987562888, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 71.64061962688662 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 77.21353987562888 }, "jaggedness": { "accuracy": 0.4777777777777778, "cc": 8.891936687850714, "cc_by_type": { "choice": 46.58325312800771, "noul": -51.35135135135135, "score": 31.443908286895773 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 8.891936687850714 }, "llm_router": { "accuracy": 0.7987220447284346, "cc": 62.159239624897964, "cc_by_type": { "choice": 76.22611605177423, "noul": 54.639175257731964, "score": 55.612427565187716 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 62.159239624897964 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.2962962962962963, "cc": -26.109429454822063, "cc_by_type": { "choice": 13.402061855670107, "noul": -100.0, "score": 8.269649779863697 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.6327272727272727, "cc": 34.94749358869319, "cc_by_type": { "choice": 66.7427453537659, "noul": -8.333333333333337, "score": 46.43306874564702 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 34.94749358869319 }, "negation_pair_gen": { "accuracy": 0.46511627906976744, "cc": 0.2664008509962761, "cc_by_type": { "choice": 63.333333333333336, "noul": -66.66666666666667, "score": 4.132535886322164 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.2664008509962761 }, "option_order": { "accuracy": 0.8125, "cc": 76.03195739014647, "cc_by_type": { "choice": 76.03195739014647 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 76.03195739014647 }, "perturb_pair": { "accuracy": 0.828125, "cc": 77.05660729696058, "cc_by_type": { "choice": 75.97597597597598, "noul": 100.0, "score": 55.19384591490575 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 77.05660729696058 }, "usecase": { "accuracy": 0.5590909090909091, "cc": 31.572158461704316, "cc_by_type": { "choice": 60.78431372549021, "noul": 0.0, "score": 33.932161659622736 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 31.572158461704316 } }, "lang": { "ar": { "accuracy": 0.375, "cc": -5.14602112844375, "cc_by_type": { "choice": 23.809523809523807, "noul": 0.0, "score": -39.24758719485506 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.7142857142857143, "cc": 78.94736842105263, "cc_by_type": { "choice": 57.89473684210526, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 78.94736842105263 }, "da": { "accuracy": 0.6923076923076923, "cc": 40.62744831788267, "cc_by_type": { "choice": 58.52534562211981, "noul": 33.33333333333333, "score": 30.023665998194872 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 40.62744831788267 }, "de": { "accuracy": 0.84375, "cc": 65.36973340580671, "cc_by_type": { "choice": 84.07079646017698, "noul": 50.0, "score": 62.03840375724315 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 65.36973340580671 }, "el": { "accuracy": 0.4, "cc": 15.425167488428194, "cc_by_type": { "choice": 49.720670391061454, "noul": -100.0, "score": 96.55483207422313 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 15.425167488428194 }, "en": { "accuracy": 0.6194613379669852, "cc": 32.65410958205473, "cc_by_type": { "choice": 63.53473840934329, "noul": -0.7194244604316502, "score": 35.14701479725255 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 32.65410958205473 }, "es": { "accuracy": 0.7741935483870968, "cc": 62.56797666185319, "cc_by_type": { "choice": 80.42414355628058, "noul": 100.0, "score": 7.279786429279 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 62.56797666185319 }, "fi": { "accuracy": 0.5, "cc": 18.91749265853117, "cc_by_type": { "choice": 67.21311475409836, "noul": -100.0, "score": 89.53936322149515 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 18.91749265853117 }, "fr": { "accuracy": 0.7142857142857143, "cc": 47.06376202766997, "cc_by_type": { "choice": 57.64705882352941, "noul": 0.0, "score": 83.54422725948051 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 47.06376202766997 }, "hi": { "accuracy": 0.5238095238095238, "cc": 9.1729321161986, "cc_by_type": { "choice": 49.720670391061454, "noul": -75.0, "score": 52.798125957534346 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 9.1729321161986 }, "id": { "accuracy": 0.5, "cc": -22.222222222222225, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8, "cc": 63.89055758345746, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": 5.701290832991235 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 63.89055758345746 }, "ja": { "accuracy": 0.56, "cc": 29.244543319970006, "cc_by_type": { "choice": 57.108816521048446, "noul": 19.999999999999996, "score": 10.62481343886158 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 29.244543319970006 }, "ko": { "accuracy": 0.5, "cc": 2.3136682734769445, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0, "score": 51.385449264875284 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 2.3136682734769445 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 76.97684526925495, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 97.59720247443153 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 76.97684526925495 }, "nl": { "accuracy": 0.5, "cc": 7.280856601680536, "cc_by_type": { "choice": 74.02597402597404, "noul": -100.0, "score": 47.81659577906757 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 7.280856601680536 }, "no": { "accuracy": 0.3333333333333333, "cc": -14.516129032258064, "cc_by_type": { "choice": -29.032258064516128, "noul": 0.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.6666666666666666, "cc": 40.43577295910621, "cc_by_type": { "choice": 84.76512907321202, "noul": 0.0, "score": 36.542189804106606 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 40.43577295910621 }, "pt": { "accuracy": 0.6071428571428571, "cc": 48.520372352488316, "cc_by_type": { "choice": 36.35827931644077, "noul": 60.00000000000001, "score": 49.202837741024176 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 48.520372352488316 }, "sv": { "accuracy": 0.2727272727272727, "cc": 8.986079710155908, "cc_by_type": { "choice": -2.127659574468081, "noul": -19.999999999999996, "score": 49.0858987049358 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 8.986079710155908 }, "tr": { "accuracy": 0.6153846153846154, "cc": 14.927301478010037, "cc_by_type": { "choice": 56.52173913043478, "noul": 0.0, "score": -11.739834696404671 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 14.927301478010037 }, "uk": { "accuracy": 0.8571428571428571, "cc": 86.27310580160078, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 58.81931740480236 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 86.27310580160078 }, "zh": { "accuracy": 0.7, "cc": 19.69631508464502, "cc_by_type": { "choice": 100.0, "noul": -33.333333333333336, "score": -7.577721412731608 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 19.69631508464502 } }, "type": { "choice": { "accuracy": 0.72, "cc": 63.83398074972732, "cc_by_type": { "choice": 63.83398074972732 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 63.83398074972732 }, "noul": { "accuracy": 0.49866666666666665, "cc": -0.2666666666666706, "cc_by_type": { "noul": -0.2666666666666706 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.5546666666666666, "cc": 36.90290826999303, "cc_by_type": { "score": 36.90290826999303 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 36.90290826999303 } }, "usecase": { "advertising": { "accuracy": 0.36363636363636365, "cc": -11.75076322631999, "cc_by_type": { "choice": 52.0, "noul": -75.0, "score": -12.252289678959972 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.4583333333333333, "cc": 7.659686301813146, "cc_by_type": { "choice": 25.465838509316768, "noul": 11.111111111111116, "score": -13.597890714988448 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 7.659686301813146 }, "customer_support": { "accuracy": 0.7142857142857143, "cc": 68.11810035858215, "cc_by_type": { "choice": 34.959349593495936, "noul": 100.0, "score": 69.39495148225052 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 68.11810035858215 }, "demand_forecasting": { "accuracy": 0.29411764705882354, "cc": -25.53781816870939, "cc_by_type": { "choice": 29.82456140350877, "noul": -100.0, "score": -6.438015909636929 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.6470588235294118, "cc": 28.50650905063489, "cc_by_type": { "choice": 71.0144927536232, "noul": -33.333333333333336, "score": 47.838367731614795 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 28.50650905063489 }, "feature_extraction": { "accuracy": 0.5263157894736842, "cc": -0.967848961333846, "cc_by_type": { "choice": 37.106918238993714, "noul": -33.333333333333336, "score": -6.6771317896619165 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5333333333333333, "cc": 29.47687169859932, "cc_by_type": { "choice": 70.58823529411765, "noul": -14.28571428571429, "score": 32.1280940873946 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 29.47687169859932 }, "gaming": { "accuracy": 0.3181818181818182, "cc": -8.77246629290122, "cc_by_type": { "choice": 47.73519163763067, "noul": -100.0, "score": 25.947409483665673 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.3333333333333333, "cc": 11.06914599494948, "cc_by_type": { "choice": 58.33333333333333, "noul": -60.0, "score": 34.87410465151511 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 11.06914599494948 }, "knowledge_graphs": { "accuracy": 0.4090909090909091, "cc": 26.706378828218817, "cc_by_type": { "choice": 10.064239828693788, "noul": 33.33333333333333, "score": 36.721563322629336 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 26.706378828218817 }, "lead_generation": { "accuracy": 0.52, "cc": 12.856796666407638, "cc_by_type": { "choice": 88.3720930232558, "noul": -55.55555555555556, "score": 5.753852531522674 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 12.856796666407638 }, "legal_compliance": { "accuracy": 0.5, "cc": -11.06811721251573, "cc_by_type": { "choice": 67.74193548387096, "noul": -100.0, "score": -0.9462871214181545 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.5185185185185185, "cc": 25.34251136401156, "cc_by_type": { "choice": 65.4510556621881, "noul": -19.999999999999996, "score": 30.57647842984659 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 25.34251136401156 }, "model_routing": { "accuracy": 0.8202764976958525, "cc": 63.2648273073167, "cc_by_type": { "choice": 82.46753246753246, "noul": 63.63636363636365, "score": 43.690585818054 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 63.2648273073167 }, "moderation_trust_safety": { "accuracy": 0.6111111111111112, "cc": 7.461255389755934, "cc_by_type": { "choice": 65.4510556621881, "noul": -100.0, "score": 56.93271050707971 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 7.461255389755934 }, "recruiting": { "accuracy": 0.6111111111111112, "cc": 50.77332873916746, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 52.4128367281802 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 50.77332873916746 }, "risk_assessment": { "accuracy": 0.5555555555555556, "cc": 17.749833188153378, "cc_by_type": { "choice": 70.44334975369459, "noul": -50.0, "score": 32.80614981076555 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 17.749833188153378 }, "scientific_discovery": { "accuracy": 0.5652173913043478, "cc": 30.59909361582899, "cc_by_type": { "choice": 42.02898550724638, "noul": 0.0, "score": 49.768295340240584 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 30.59909361582899 }, "search_retrieval": { "accuracy": 0.9166666666666666, "cc": 77.04547999886454, "cc_by_type": { "choice": 100.0, "noul": 50.0, "score": 81.13643999659361 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 77.04547999886454 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 82.33973332691625, 88.98871160764558 ], "I": [ 32.506331847623684, 41.172796760574926 ], "capability": [ 58.38420161355362, 64.17699789788875 ], "composite": { "A": [ 7.365294687696036, 12.818089044625026 ], "B": [ 6.817840903770886, 12.323734305732241 ], "C": [ 5.114787977566692, 8.901450725434048 ] }, "seed": 24 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 88.10935553643324, "capability": 62.42104465724772, "intelligence": 36.7327337780622 }, "per_type": { "choice": { "accuracy": 0.72, "calibration": 88.80529020086236, "cc": 63.83398074972732, "cc_by_type": { "choice": 63.83398074972732 }, "cc_tiered": 64.92910863972062, "cc_tiered_open": 67.89294454901062, "cc_tiered_sealed": 61.96527273043063, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 63.83398074972732 }, "noul": { "accuracy": 0.49866666666666665, "calibration": 83.63275246769038, "cc": -0.2666666666666706, "cc_by_type": { "noul": -0.2666666666666706 }, "cc_tiered": 6.8160800552104925, "cc_tiered_open": 13.632160110420985, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.5546666666666666, "calibration": 91.89002394074701, "cc": 36.90290826999303, "cc_by_type": { "score": 36.90290826999303 }, "cc_tiered": 38.453012639255476, "cc_tiered_open": 42.19463838367629, "cc_tiered_sealed": 34.71138689483466, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 36.90290826999303 } }, "raw_on_A": { "calibration": 83.37739283046191, "capability": 60.48742810868637, "intelligence": 37.59746338691083 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 8.584936549197561, "Emphasis on Accuracy 60:20:20": 8.15411454169331, "Emphasis on Cost 20:20:60": 7.544520104835787, "Emphasis on Speed 20:60:20": 10.608359264420562, "Intelligence only": 7.583281348791939 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Frontier Infra", "axes": { "calibration": 89.67673072100749, "cost": 48.987014575391086, "intelligence": 37.15024789587638, "speed": 91.39972504887733 }, "calibration": { "parts": { "choice": { "ece_hard": 0.0545485704918033, "ece_score": 89.09028590163935, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 89.09028590163935 }, "noul": { "abstention_rate": 0.4048257372654156, "brier": 0.12966919167437543, "ece": 0.02817181233243971, "n": 373, "score": 94.36563753351206 }, "score": { "interval90_coverage": 0.9408602150537635, "interval90_width": 0.4444444444444445, "n": 372, "nrps": 0.10121750178666497, "score": 85.57426872787106, "top_ece": 0.09364856182795694 } }, "score": 89.67673072100749 }, "capability": 63.41348930844194, "composite_ci95": { "A": [ 21.15380580389746, 37.98738977479574 ], "B": [ 18.608381870587298, 34.83635376233961 ], "C": [ 14.690142919373237, 26.380131788052598 ] }, "cost": { "basis": "ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-9B -> qwen/qwen3.5-9b USD 0.09999999999999999/M input, USD 0.15/M output. Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 533d80f0...d960e5, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens as reported by the author's own server (750,303 input over the 1,485 answered items, output_tokens 1 per answered request).", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.05016869999999999 }, "display": "Jebadiah 9B v2.1 (Frontier Infra, Qwen3.5-9B LoRA)", "endpoint_condition": "Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA GeForce RTX 5090 32 GB (sm_120, driver 595.71.05), Romania (EU) - different hardware from the RTX PRO 6000 the author measures on, so Speed is a lower bound on this card and not comparable to their 49.9 ms median. The author's OWN standalone server `jebadiah-serve --model --device cuda` from getainode/jebadiah @ 74a0cd92965529e90a411532e9887b544c931b2d (the commit behind the published jebadiah-27b row, independently code-reviewed PASS), installed from the author's own frozen uv.lock (`uv sync --frozen --no-dev --extra cuda`, torch 2.14.0+cu130, transformers 5.17.0, flash-linear-attention 0.5.2), all author defaults (dtype auto = bf16, batch-size 8, --max-prompt-tokens 8192, strict prompt-contract check ON, the repo's own temperatures.json, fp32 candidate logits), JEBADIAH_API_KEY unset. Isolation: server started inside a network namespace with loopback only (unshare -n), the cut proven before the sealed input was staged and re-proven immediately before the first sealed request, synthetic smoke gate passed in between; runner unchanged (run_v15.py, 0 resumes, rc 0); one use of input-selfhosted-1500 (custody #20323, deputy #20315). Pod removed by exact id and verified gone four ways. Published Speed uses the standing x2 + 0.15 s self-hosted adjustment; raw p50 0.0502 s / p95 0.0837 s.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 45.02812038711811, "I_sealed": 29.272375404634655, "base": 37.15024789587638, "excess": 5.916218212127742, "gap": 15.755744982483453, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 65.08523959814883, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 88.32432432432431, "hard": 60.36489462094998, "judge": 60.44825313117996, "standard": 69.8618668899121 } }, "open|noul": { "cc": 22.648033126294003, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 42.85714285714286, "hard": 21.739130434782616, "judge": 10.000000000000009, "standard": 33.33333333333333 } }, "open|score": { "cc": 47.35108843691149, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 50.52056666666667, "hard": 53.42431944444446, "judge": 36.94557627118644, "standard": 49.22815555555554 } }, "sealed|choice": { "cc": 56.24785832875334, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 69.50444726810673, "hard": 49.52006476234531, "judge": 55.04407443682666, "standard": 64.88082682978275 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 37.93103448275863, "hard": -3.6363636363636376, "judge": -26.15384615384615, "standard": 25.373134328358216 } }, "sealed|score": { "cc": 31.56926788515063, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 54.8699552631579, "hard": 24.433701408450716, "judge": 25.61958750000002, "standard": 43.11457772727274 } } } }, "jevbench_score": 30.532291970812633, "key": "jebadiah-9b-v2-1", "last_measured_on": "2026-10-16", "licence": "Apache-2.0 (weights and base)", "listing": "ranked", "measured_in": "v1.6.25", "measurement_date_status": "measured", "model_pin": "frontier-infra/jebadiah-9b-v2-1@c5e49fab7ec8e9923e1f907d85911a53fb8e9c32", "noul_decisive": { "acc_among_decisive": 0.9279279279279279, "decisive": 222, "decisive_rate": 0.5951742627345844, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": 16, "ranked": true, "ranks": { "A": 16, "B": 16, "C": 17, "capability": 19 }, "repo": "https://huggingface.co/frontier-infra/jebadiah-9b-v2-1", "scores": { "A": 30.532291970812633, "B": 27.501958548930418, "C": 21.202980535286557 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.2362852499994915, "p50_s_raw": 0.04314262499974575, "p95_s_adjusted": 0.30661322509549793, "p95_s_raw": 0.07830661254774895 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.7272727272727273, "cc": 45.461111111111116, "cc_by_type": { "choice": 58.333333333333336, "noul": 100.0, "score": -21.94999999999998 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 45.461111111111116 }, "calibration_gen": { "accuracy": 0.39603960396039606, "cc": 8.965266960498408, "cc_by_type": { "choice": 23.647932131495228, "noul": -4.0000000000000036, "score": 7.247868749999997 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 8.965266960498408 }, "composite_gen": { "accuracy": 0.5742574257425742, "cc": 25.502561765455255, "cc_by_type": { "choice": 64.22764227642277, "noul": -18.518518518518523, "score": 30.79856153846151 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 25.502561765455255 }, "fairness": { "accuracy": 0.875, "cc": 79.02633518518519, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 77.07900555555555 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 79.02633518518519 }, "jaggedness": { "accuracy": 0.42777777777777776, "cc": 7.105247243920597, "cc_by_type": { "choice": 32.14629451395574, "noul": -35.13513513513513, "score": 24.304582352941186 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 7.105247243920597 }, "llm_router": { "accuracy": 0.7955271565495208, "cc": 68.52774777747052, "cc_by_type": { "choice": 70.67887646385489, "noul": 71.13402061855669, "score": 63.770346249999996 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 68.52774777747052 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.37037037037037035, "cc": -21.368183456062837, "cc_by_type": { "choice": 50.51546391752578, "noul": -100.0, "score": -14.620014285714289 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.6145454545454545, "cc": 30.838639188636876, "cc_by_type": { "choice": 65.76459080534725, "noul": -16.666666666666664, "score": 43.41799342723004 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 30.838639188636876 }, "negation_pair_gen": { "accuracy": 0.5116279069767442, "cc": 10.267682222222229, "cc_by_type": { "choice": 59.99999999999999, "noul": -33.333333333333336, "score": 4.136380000000028 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 10.267682222222229 }, "option_order": { "accuracy": 0.8958333333333334, "cc": 86.68442077230361, "cc_by_type": { "choice": 86.68442077230361 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 86.68442077230361 }, "perturb_pair": { "accuracy": 0.8125, "cc": 70.84381691306692, "cc_by_type": { "choice": 69.96996996996997, "noul": 80.0, "score": 62.56148076923078 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 70.84381691306692 }, "usecase": { "accuracy": 0.55, "cc": 27.858113095424844, "cc_by_type": { "choice": 52.205882352941195, "noul": -4.1666666666666625, "score": 35.535123600000006 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 27.858113095424844 } }, "lang": { "ar": { "accuracy": 0.5, "cc": 4.7372142857142805, "cc_by_type": { "choice": 42.857142857142854, "noul": 0.0, "score": -28.645500000000013 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 4.7372142857142805 }, "cs": { "accuracy": 0.8571428571428571, "cc": 89.47368421052632, "cc_by_type": { "choice": 78.94736842105263, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 89.47368421052632 }, "da": { "accuracy": 0.6923076923076923, "cc": 39.28832631848439, "cc_by_type": { "choice": 58.52534562211981, "noul": 33.33333333333333, "score": 26.006300000000017 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 39.28832631848439 }, "de": { "accuracy": 0.8125, "cc": 70.35387155316147, "cc_by_type": { "choice": 68.14159292035399, "noul": 75.0, "score": 67.92002173913043 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 70.35387155316147 }, "el": { "accuracy": 0.4666666666666667, "cc": 21.789848975791433, "cc_by_type": { "choice": 66.4804469273743, "noul": -100.0, "score": 98.8891 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 21.789848975791433 }, "en": { "accuracy": 0.6072980017376195, "cc": 35.34340890177539, "cc_by_type": { "choice": 56.28562014132117, "noul": 14.388489208633093, "score": 35.35611735537191 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 35.34340890177539 }, "es": { "accuracy": 0.7741935483870968, "cc": 66.55888314621119, "cc_by_type": { "choice": 80.42414355628058, "noul": 100.0, "score": 19.252505882352942 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 66.55888314621119 }, "fi": { "accuracy": 0.5, "cc": 29.99688269581056, "cc_by_type": { "choice": 67.21311475409836, "noul": -33.333333333333336, "score": 56.110866666666666 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 29.99688269581056 }, "fr": { "accuracy": 0.8571428571428571, "cc": 55.07443787409701, "cc_by_type": { "choice": 78.82352941176471, "noul": 0.0, "score": 86.39978421052632 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 55.07443787409701 }, "hi": { "accuracy": 0.5238095238095238, "cc": 3.0043369878957136, "cc_by_type": { "choice": 83.24022346368716, "noul": -50.0, "score": -24.227212500000018 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 3.0043369878957136 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.7333333333333333, "cc": 61.87361175444456, "cc_by_type": { "choice": 71.94076383476228, "noul": 100.0, "score": 13.680071428571395 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 61.87361175444456 }, "ja": { "accuracy": 0.56, "cc": 25.42277845379931, "cc_by_type": { "choice": 42.81175536139793, "noul": 0.0, "score": 33.45658 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 25.42277845379931 }, "ko": { "accuracy": 0.3, "cc": -7.526621957671959, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0, "score": 21.864578571428574 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 76.65298333333334, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 96.62561666666667 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 76.65298333333334 }, "nl": { "accuracy": 0.4, "cc": -0.26805731601731964, "cc_by_type": { "choice": 48.05194805194805, "noul": -100.0, "score": 51.14387999999999 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.16666666666666666, "cc": -32.25806451612903, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.5897435897435898, "cc": 32.30207713293114, "cc_by_type": { "choice": 84.76512907321202, "noul": -19.999999999999996, "score": 32.141102325581386 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 32.30207713293114 }, "pt": { "accuracy": 0.7142857142857143, "cc": 45.18160400580404, "cc_by_type": { "choice": 57.57218621096051, "noul": 19.999999999999996, "score": 57.97262580645162 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 45.18160400580404 }, "sv": { "accuracy": 0.2727272727272727, "cc": -0.5227982269503476, "cc_by_type": { "choice": 23.404255319148945, "noul": -60.0, "score": 35.02735000000001 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.6153846153846154, "cc": 10.888600543478256, "cc_by_type": { "choice": 56.52173913043478, "noul": 50.0, "score": -73.85593750000001 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 10.888600543478256 }, "uk": { "accuracy": 0.7142857142857143, "cc": 69.77072365591397, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 73.82829999999998 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 69.77072365591397 }, "zh": { "accuracy": 0.7, "cc": 13.044855555555557, "cc_by_type": { "choice": 100.0, "noul": -33.333333333333336, "score": -27.532099999999993 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 13.044855555555557 } }, "type": { "choice": { "accuracy": 0.6813333333333333, "cc": 58.83962571040395, "cc_by_type": { "choice": 58.83962571040395 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 58.83962571040395 }, "noul": { "accuracy": 0.5493333333333333, "cc": 9.866666666666667, "cc_by_type": { "noul": 9.866666666666667 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 9.866666666666667 }, "score": { "accuracy": 0.544, "cc": 36.35312382671484, "cc_by_type": { "score": 36.35312382671484 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 36.35312382671484 } }, "usecase": { "advertising": { "accuracy": 0.4090909090909091, "cc": -11.250268627450984, "cc_by_type": { "choice": 68.0, "noul": -50.0, "score": -51.75080588235295 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.5, "cc": 16.864947946417512, "cc_by_type": { "choice": 44.099378881987576, "noul": 11.111111111111116, "score": -4.6156461538461535 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 16.864947946417512 }, "customer_support": { "accuracy": 0.7142857142857143, "cc": 36.58365826558265, "cc_by_type": { "choice": 67.47967479674797, "noul": 0.0, "score": 42.27129999999999 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 36.58365826558265 }, "demand_forecasting": { "accuracy": 0.23529411764705882, "cc": -25.61673798732944, "cc_by_type": { "choice": 6.432748538011691, "noul": -100.0, "score": 16.717037499999996 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5294117647058824, "cc": 16.98094864130434, "cc_by_type": { "choice": 42.02898550724638, "noul": -33.333333333333336, "score": 42.24719374999998 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 16.98094864130434 }, "feature_extraction": { "accuracy": 0.3157894736842105, "cc": -50.77358270440254, "cc_by_type": { "choice": 24.528301886792455, "noul": -100.0, "score": -76.84905000000009 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5666666666666667, "cc": 14.320775486604896, "cc_by_type": { "choice": 80.3921568627451, "noul": -71.42857142857143, "score": 33.998741025641024 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 14.320775486604896 }, "gaming": { "accuracy": 0.36363636363636365, "cc": -9.09330015654193, "cc_by_type": { "choice": 37.2822299651568, "noul": -100.0, "score": 35.437869565217404 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.2, "cc": -0.8783444444444442, "cc_by_type": { "choice": 16.666666666666668, "noul": -60.0, "score": 40.698299999999996 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.5, "cc": 33.657132663898075, "cc_by_type": { "choice": 22.912205567451824, "noul": 33.33333333333333, "score": 44.72585909090909 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 33.657132663898075 }, "lead_generation": { "accuracy": 0.6, "cc": 32.8623636676846, "cc_by_type": { "choice": 88.3720930232558, "noul": -11.111111111111116, "score": 21.326109090909096 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 32.8623636676846 }, "legal_compliance": { "accuracy": 0.2, "cc": -54.82240618279571, "cc_by_type": { "choice": 3.2258064516129026, "noul": -100.0, "score": -67.69302500000003 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.5555555555555556, "cc": 22.22430251652804, "cc_by_type": { "choice": 53.93474088291747, "noul": -19.999999999999996, "score": 32.73816666666666 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 22.22430251652804 }, "model_routing": { "accuracy": 0.8248847926267281, "cc": 69.68780904581529, "cc_by_type": { "choice": 80.12987012987014, "noul": 75.75757575757575, "score": 53.17598124999998 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 69.68780904581529 }, "moderation_trust_safety": { "accuracy": 0.7222222222222222, "cc": 15.739794068721551, "cc_by_type": { "choice": 76.96737044145875, "noul": -100.0, "score": 70.25201176470588 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 15.739794068721551 }, "recruiting": { "accuracy": 0.7222222222222222, "cc": 56.08058175249388, "cc_by_type": { "choice": 83.28690807799443, "noul": 33.33333333333333, "score": 51.62150384615387 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 56.08058175249388 }, "risk_assessment": { "accuracy": 0.5, "cc": 22.892261682604083, "cc_by_type": { "choice": 70.44334975369459, "noul": 0.0, "score": -1.7665647058823453 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 22.892261682604083 }, "scientific_discovery": { "accuracy": 0.43478260869565216, "cc": 3.1867240579710128, "cc_by_type": { "choice": 42.02898550724638, "noul": -66.66666666666667, "score": 34.19785333333333 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 3.1867240579710128 }, "search_retrieval": { "accuracy": 0.875, "cc": 76.21831885808827, "cc_by_type": { "choice": 89.3238434163701, "noul": 50.0, "score": 89.33111315789473 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 76.21831885808827 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 82.80134503834545, 89.14356377374882 ], "I": [ 31.98351341783755, 40.81472001755972 ], "capability": [ 58.46443039305731, 64.1153266139154 ], "composite": { "A": [ 21.15380580389746, 37.98738977479574 ], "B": [ 18.608381870587298, 34.83635376233961 ], "C": [ 14.690142919373237, 26.380131788052598 ] }, "seed": 25 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 89.67673072100749, "capability": 63.41348930844194, "intelligence": 37.15024789587638 }, "per_type": { "choice": { "accuracy": 0.6813333333333333, "calibration": 89.09028590163935, "cc": 58.83962571040395, "cc_by_type": { "choice": 58.83962571040395 }, "cc_tiered": 60.666548963451085, "cc_tiered_open": 65.08523959814883, "cc_tiered_sealed": 56.24785832875334, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 58.83962571040395 }, "noul": { "accuracy": 0.5493333333333333, "calibration": 94.36563753351206, "cc": 9.866666666666667, "cc_by_type": { "noul": 9.866666666666667 }, "cc_tiered": 11.324016563147001, "cc_tiered_open": 22.648033126294003, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 9.866666666666667 }, "score": { "accuracy": 0.544, "calibration": 85.57426872787106, "cc": 36.35312382671484, "cc_by_type": { "score": 36.35312382671484 }, "cc_tiered": 39.46017816103106, "cc_tiered_open": 47.35108843691149, "cc_tiered_sealed": 31.56926788515063, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 36.35312382671484 } }, "raw_on_A": { "calibration": 83.97311382409126, "capability": 61.0776332772807, "intelligence": 38.182152730470136 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 27.28128732735876, "Emphasis on Accuracy 60:20:20": 23.634149120606594, "Emphasis on Cost 20:20:60": 26.736489001488906, "Emphasis on Speed 20:60:20": 33.055927997224416, "Intelligence only": 19.686434980021833 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Supersonic Labs / @supersonicai (Hugging Face SupersonicLabs)", "axes": { "calibration": 17.26803648238681, "cost": 89.8795371983372, "intelligence": 2.4412732607743255, "speed": 79.42119248319958 }, "calibration": { "parts": { "choice": { "ece_hard": 0.49162634593303195, "ece_score": 1.67473081339361, "mean_tvd": null, "n": 739, "n_ece_hard": 304, "n_tvd": 0, "score": 1.67473081339361 }, "noul": { "abstention_rate": 0.27419354838709675, "brier": 0.45131804975332956, "ece": 0.4322726807691505, "n": 372, "score": 13.545463846169902 }, "score": { "interval90_coverage": 0.5887096774193549, "interval90_width": 0.3958333333333335, "n": 372, "nrps": 0.26832170424806157, "score": 36.583914787596925, "top_ece": 0.5187092957892186 } }, "score": 17.26803648238681 }, "capability": 9.854654871580568, "composite_ci95": { "A": [ 0.0008434089914363179, 0.12224092525207648 ], "B": [ 0.0005466521487834601, 0.08792103692592891 ], "C": [ 0.0005857006884974428, 0.08488953142505314 ] }, "cost": { "basis": "ESTIMATE (*est.), size-class reference; self-hosted open weights, no public bookable tariff for these weights. The frozen pricing_v15 path is asserted with ONE authorised, scoped correction: CoS deputy decision #18761 added the declared base jhu-clsp/mmBERT-small to pricing_v15.UNLISTED_BASE_MODELS, exactly as openbmb/MiniCPM5-2B and answerdotai/ModernBERT-base already are, so reference() returns no market reference for it rather than raising and the ancestry can be carried in base_model instead of being dropped to null. The addition changes no other row: it was proved behaviour-identical over every registry entry on the box before it was installed (r108/receipts/PRICING-EQUIV-R108.json) and independently reviewed (r108/receipts/REVIEW-PRICING-R108.json). Florian's standing 24 Sep rule then gives a clearly labelled size-class estimate: the base-size-encoder line the PUBLISHED v1.6 field already carries - USD 0.005 per 1M input and 0.0 per 1M output, the line modernbert-jev-v2, tacet-sonata and watt-flash-0.1 carry there (no row of THIS draw carries it, and lev-350m carries 0.01/0.0, not this line) (four base-size encoders in the dated 25 Sep 2026 DeepInfra snapshot sha256 84eb7926... all price at exactly that) - so no new release-lane pricing line is created. x this row's own measured tokens: usage.input_tokens comes from the author's OWN audit function engine.encoding_info() and usage.output_tokens is 0 by construction (one encoder forward pass read at marker positions, nothing generated). Estimated, not charged.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.0021744466666666668 }, "display": "Julia-1 (Supersonic Labs)", "endpoint_condition": "Sandy, the evaluator's own shared CPU host (AMD Ryzen 5 3600, 6 cores / 12 threads, Zen 2, AVX2), NOT a dedicated machine: other jobs ran beside it and the 1-minute load average was logged every 60 s into logs/CONDITIONS-julia-1.jsonl (11 samples, 10.87-13.78, mean 12.5). The author ships NO server, so the HTTP surface is OURS: r107/src/julia_shim_r107.py, a pure transport that makes no mapping decision, because the author's own julia.typed.predict_typed already speaks JevBench's typed-question shape exactly (choice keyed by the item's own criteria keys, score keyed '0'..'n-1' with score = sum(i*p_i), noul = p(true)). Engine built with the author's own inference-policy.json (max_length 8192, head_length 512, strict_encoding true), torch backend, device cpu, float32 (their julia_config weight_dtype), thread count left at their own default of 4 - all four read back from the run's own /health rather than asserted. Run inside a loopback-only network namespace whose outbound probe failed as required, offline (HF_HUB_OFFLINE=1), credential-free, nice 5, in a systemd scope with MemoryMax=8G and no swap. Speed is therefore a loaded-shared-CPU figure and a lower bound for this model on a dedicated core or a GPU; the standard self-hosted x2 + 0.15 s adjustment is applied on top and the raw p50/p95 are kept. Measured for this release on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720...) under CoS deputy decision #18209, which admitted this row only after an independent source review and a synthetic-only smoke, both of which were met in that order before the first sealed item. DISCLOSED: the author's own strict_encoding policy REFUSES rather than truncates, so 17 of the 1,500 items (15 over their lossless context budget, 2 with an option above their 48-token option contract) are honest item-level misses recorded as HTTP 422 - predicted exactly, class split included, before the run.", "endpoint_kind": "cpu", "full_coverage": true, "intelligence": { "I_open": 2.994076934245856, "I_sealed": 1.888469587302795, "base": 2.4412732607743255, "excess": -8.733919423412651, "gap": 1.105607346943061, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 8.982230802737568, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 2.7027027027027115, "hard": 13.180245360176164, "judge": 5.075807514831907, "standard": 9.585600669736293 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": -57.14285714285714, "hard": -4.347826086956519, "judge": 0.0, "standard": -22.22222222222222 } }, "open|score": { "cc": 0.0, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": -4.595553956916043, "hard": -21.831788639496995, "judge": -76.77226748673378, "standard": -19.64084658486791 } }, "sealed|choice": { "cc": 5.6654087619083855, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 12.325285895806868, "hard": 0.6013646351335588, "judge": 11.85112634671892, "standard": 3.1849820712929935 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": -41.379310344827594, "hard": -41.81818181818182, "judge": -50.76923076923077, "standard": -19.402985074626866 } }, "sealed|score": { "cc": 0.0, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": -36.27109595021023, "hard": -40.51450244498809, "judge": -18.32435058181421, "standard": -13.222659125532354 } } } }, "jevbench_score": 0.019411107562972558, "key": "julia-1", "last_measured_on": "2026-10-13", "licence": "Apache-2.0 (Hugging Face cardData.license and README; the pinned tree contains no LICENSE file)", "listing": "ranked", "measured_in": "v1.6.15", "measurement_date_status": "measured", "model_pin": "SupersonicLabs/Julia-1@a85b127321d580d65176c89ced8273f305745d85", "noul_decisive": { "acc_among_decisive": 0.45555555555555555, "decisive": 270, "decisive_rate": 0.7258064516129032, "n": 375, "supported": true, "valid": 372 }, "planned_weight_denominator": 1.0, "rank": 34, "ranked": true, "ranks": { "A": 34, "B": 34, "C": 34, "capability": 34 }, "repo": "https://huggingface.co/SupersonicLabs/Julia-1", "scores": { "A": 0.019411107562972558, "B": 0.01323121125368159, "C": 0.013479935807619834 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.7337018699967303, "p50_s_raw": 0.2918509349983651, "p95_s_adjusted": 1.5572599903855007, "p95_s_raw": 0.7036299951927504 }, "status": { "answered_ok": 1483, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.2727272727272727, "cc": -27.879433144127862, "cc_by_type": { "choice": 0.0, "noul": 0.0, "score": -83.63829943238359 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 0.0 }, "calibration_gen": { "accuracy": 0.3564356435643564, "cc": -3.761010359494321, "cc_by_type": { "choice": 4.559915164369044, "noul": 4.0000000000000036, "score": -19.842946242852012 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.25742574257425743, "cc": -31.644341476391443, "cc_by_type": { "choice": -7.317073170731707, "noul": -18.518518518518523, "score": -69.0974327399241 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 0.0 }, "fairness": { "accuracy": 0.3541666666666667, "cc": -5.944000384605097, "cc_by_type": { "choice": 4.761904761904761, "noul": -60.0, "score": 37.40609408427995 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 0.0 }, "jaggedness": { "accuracy": 0.37222222222222223, "cc": -8.884037970016676, "cc_by_type": { "choice": 24.927815206929747, "noul": -45.945945945945944, "score": -5.6339831710338295 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.23003194888178913, "cc": -20.437719287542667, "cc_by_type": { "choice": -0.6427753808223877, "noul": -38.144329896907216, "score": -22.526052584898395 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 0.0 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.18518518518518517, "cc": -32.03827054080279, "cc_by_type": { "choice": 1.0309278350515454, "noul": -75.0, "score": -22.145739457459925 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.29454545454545455, "cc": -25.78754754021827, "cc_by_type": { "choice": 17.835017932833402, "noul": -55.55555555555556, "score": -39.64210499793266 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 0.0 }, "negation_pair_gen": { "accuracy": 0.3372093023255814, "cc": -37.503184879754805, "cc_by_type": { "choice": 6.666666666666665, "noul": -13.33333333333333, "score": -105.84288797259775 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.0 }, "option_order": { "accuracy": 0.20833333333333334, "cc": -1.1984021304926722, "cc_by_type": { "choice": -1.1984021304926722 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 0.0 }, "perturb_pair": { "accuracy": 0.265625, "cc": -25.52938809242961, "cc_by_type": { "choice": -5.105105105105101, "noul": 19.999999999999996, "score": -91.48305917218373 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 0.0 }, "usecase": { "accuracy": 0.2409090909090909, "cc": -16.515408619322642, "cc_by_type": { "choice": 3.1862745098039587, "noul": -25.0, "score": -27.732500367771884 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 0.0 } }, "lang": { "ar": { "accuracy": 0.375, "cc": -2.776462098911864, "cc_by_type": { "choice": 42.857142857142854, "noul": 0.0, "score": -51.186529153878446 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.14285714285714285, "cc": 36.84210526315789, "cc_by_type": { "choice": -26.315789473684216, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 36.84210526315789 }, "da": { "accuracy": 0.3076923076923077, "cc": -37.718345339022456, "cc_by_type": { "choice": 37.78801843317972, "noul": -100.0, "score": -50.9430544502471 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 0.0 }, "de": { "accuracy": 0.4375, "cc": 22.354227161079496, "cc_by_type": { "choice": 20.353982300884955, "noul": 25.0, "score": 21.70869918235354 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 22.354227161079496 }, "el": { "accuracy": 0.2, "cc": 0.6000833683740154, "cc_by_type": { "choice": -17.31843575418994, "noul": -33.333333333333336, "score": 52.452019192645324 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 0.6000833683740154 }, "en": { "accuracy": 0.2797567332754127, "cc": -16.97684900279254, "cc_by_type": { "choice": 3.7844302607973472, "noul": -26.618705035971225, "score": -28.096272233203745 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 0.0 }, "es": { "accuracy": 0.41935483870967744, "cc": -11.423096652088782, "cc_by_type": { "choice": 31.484502446982063, "noul": -25.0, "score": -40.75379240324841 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 0.0 }, "fi": { "accuracy": 0.125, "cc": -39.640298601530766, "cc_by_type": { "choice": 1.6393442622950836, "noul": -100.0, "score": -20.56024006688737 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 0.0 }, "fr": { "accuracy": 0.21428571428571427, "cc": -41.1336147354147, "cc_by_type": { "choice": 15.294117647058824, "noul": -100.0, "score": -38.69496185330294 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 0.0 }, "hi": { "accuracy": 0.14285714285714285, "cc": -67.07248359384337, "cc_by_type": { "choice": -0.5586592178770929, "noul": -100.0, "score": -100.65879156365303 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 0.0 }, "id": { "accuracy": 0.0, "cc": -66.66666666666667, "cc_by_type": { "choice": -33.333333333333336, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.3333333333333333, "cc": -25.47727987845241, "cc_by_type": { "choice": 15.822291504286824, "noul": -33.333333333333336, "score": -58.92079780631072 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 0.0 }, "ja": { "accuracy": 0.24, "cc": -18.87257764229911, "cc_by_type": { "choice": 28.514694201747414, "noul": -80.0, "score": -5.132427128644745 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 0.0 }, "ko": { "accuracy": 0.0, "cc": -77.83191986347525, "cc_by_type": { "choice": -33.333333333333336, "noul": -100.0, "score": -100.16242625709242 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.35714285714285715, "cc": -10.156143207655505, "cc_by_type": { "choice": 16.666666666666668, "noul": -33.333333333333336, "score": -13.801762956299847 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 0.0 }, "nl": { "accuracy": 0.2, "cc": -51.11558643410791, "cc_by_type": { "choice": -29.87012987012987, "noul": -33.333333333333336, "score": -90.14329609886053 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.16666666666666666, "cc": -32.25806451612903, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.23076923076923078, "cc": -31.24251604325812, "cc_by_type": { "choice": 16.208209902666106, "noul": -80.0, "score": -29.93575803244046 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 0.0 }, "pt": { "accuracy": 0.39285714285714285, "cc": 2.5150561249822148, "cc_by_type": { "choice": 25.75132586918091, "noul": 19.999999999999996, "score": -38.20615749423426 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 2.5150561249822148 }, "sv": { "accuracy": 0.09090909090909091, "cc": -37.55166369327866, "cc_by_type": { "choice": -2.127659574468081, "noul": -100.0, "score": -10.52733150536791 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.38461538461538464, "cc": -27.04550827576324, "cc_by_type": { "choice": 34.78260869565218, "noul": -100.0, "score": -15.919133522941898 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 0.0 }, "uk": { "accuracy": 0.14285714285714285, "cc": -41.81990641185333, "cc_by_type": { "choice": 3.225806451612906, "noul": -100.0, "score": -28.68552568717291 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 0.0 }, "zh": { "accuracy": 0.3, "cc": -68.47760196419559, "cc_by_type": { "choice": 23.72881355932204, "noul": -100.0, "score": -129.1616194519088 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 } }, "type": { "choice": { "accuracy": 0.272, "cc": 5.968349949291043, "cc_by_type": { "choice": 5.968349949291043 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 5.968349949291043 }, "noul": { "accuracy": 0.328, "cc": -34.4, "cc_by_type": { "noul": -34.4 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.24533333333333332, "cc": -29.344915811166672, "cc_by_type": { "score": -29.344915811166672 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "usecase": { "advertising": { "accuracy": 0.22727272727272727, "cc": -38.4024160564452, "cc_by_type": { "choice": -12.0, "noul": -75.0, "score": -28.207248169335596 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.3333333333333333, "cc": -2.238256630096153, "cc_by_type": { "choice": 44.099378881987576, "noul": -11.111111111111116, "score": -39.70303766116492 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 0.0 }, "customer_support": { "accuracy": 0.42857142857142855, "cc": -3.7905034661254127, "cc_by_type": { "choice": 18.699186991869915, "noul": 50.0, "score": -80.07069739024615 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 0.0 }, "demand_forecasting": { "accuracy": 0.29411764705882354, "cc": -50.406556307064626, "cc_by_type": { "choice": 18.128654970760234, "noul": -100.0, "score": -69.34832389195411 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.35294117647058826, "cc": -8.733832465640946, "cc_by_type": { "choice": 27.53623188405797, "noul": -33.333333333333336, "score": -20.40439594764747 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 0.0 }, "feature_extraction": { "accuracy": 0.42105263157894735, "cc": -52.37905037945314, "cc_by_type": { "choice": 11.949685534591195, "noul": 0.0, "score": -169.08683667295062 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.23333333333333334, "cc": -29.56253397544977, "cc_by_type": { "choice": 21.568627450980394, "noul": -71.42857142857143, "score": -38.827657948758265 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 0.0 }, "gaming": { "accuracy": 0.09090909090909091, "cc": -54.303624656078085, "cc_by_type": { "choice": -4.529616724738673, "noul": -100.0, "score": -58.38125724349559 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.06666666666666667, "cc": -27.932241245491838, "cc_by_type": { "choice": -4.166666666666665, "noul": -100.0, "score": 20.369942930191154 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.22727272727272727, "cc": -4.197964663778406, "cc_by_type": { "choice": -2.7837259100642404, "noul": 0.0, "score": -9.810168081270977 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.24, "cc": -7.467987900190109, "cc_by_type": { "choice": 6.976744186046504, "noul": -33.333333333333336, "score": 3.9526254467165067 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 0.0 }, "legal_compliance": { "accuracy": 0.3, "cc": -26.741444679738834, "cc_by_type": { "choice": -29.032258064516128, "noul": -33.333333333333336, "score": -17.858742641367044 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.2962962962962963, "cc": -17.699209031887154, "cc_by_type": { "choice": 19.385796545105567, "noul": -60.0, "score": -12.483423640767022 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 0.0 }, "model_routing": { "accuracy": 0.22580645161290322, "cc": -17.59559356925139, "cc_by_type": { "choice": -5.194805194805169, "noul": -33.333333333333336, "score": -14.258642179615656 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 0.0 }, "moderation_trust_safety": { "accuracy": 0.16666666666666666, "cc": -51.88151472980022, "cc_by_type": { "choice": -3.646833013435703, "noul": -100.0, "score": -51.99771117596499 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 0.0 }, "recruiting": { "accuracy": 0.4444444444444444, "cc": 0.4935502236767717, "cc_by_type": { "choice": -0.2785515320334252, "noul": 33.33333333333333, "score": -31.57413113026959 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 0.4935502236767717 }, "risk_assessment": { "accuracy": 0.1111111111111111, "cc": -49.23270679758713, "cc_by_type": { "choice": -18.226600985221676, "noul": -100.0, "score": -29.471519407539716 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.08695652173913043, "cc": -38.04052057494357, "cc_by_type": { "choice": -15.942028985507239, "noul": -66.66666666666667, "score": -31.5128660726568 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.20833333333333334, "cc": -9.065556816294313, "cc_by_type": { "choice": 3.914590747330961, "noul": 0.0, "score": -31.111261196213903 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 0.0 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 13.55098588837316, 21.743888220941447 ], "I": [ 0.8280039115282368, 4.706094428283649 ], "capability": [ 7.749315132293431, 12.685986145322472 ], "composite": { "A": [ 0.0008434089914363179, 0.12224092525207648 ], "B": [ 0.0005466521487834601, 0.08792103692592891 ], "C": [ 0.0005857006884974428, 0.08488953142505314 ] }, "seed": 26 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 17.26803648238681, "capability": 9.854654871580568, "intelligence": 2.4412732607743255 }, "per_type": { "choice": { "accuracy": 0.272, "calibration": 1.67473081339361, "cc": 5.968349949291043, "cc_by_type": { "choice": 5.968349949291043 }, "cc_tiered": 7.3238197823229765, "cc_tiered_open": 8.982230802737568, "cc_tiered_sealed": 5.6654087619083855, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 5.968349949291043 }, "noul": { "accuracy": 0.328, "calibration": 13.545463846169902, "cc": -34.4, "cc_by_type": { "noul": -34.4 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.24533333333333332, "calibration": 36.583914787596925, "cc": -29.344915811166672, "cc_by_type": { "score": -29.344915811166672 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "raw_on_A": { "calibration": 19.70003977436594, "capability": 11.004198815490696, "intelligence": 2.3083578566154506 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.014666666666666717, "n": 750 }, "noul": { "invalid_rate": 0.008000000000000007, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 0.01650387186472358, "Emphasis on Accuracy 60:20:20": 0.009516032366765197, "Emphasis on Cost 20:20:60": 0.026162982329893978, "Emphasis on Speed 20:60:20": 0.025995787379506426, "Intelligence only": 0.005819814930094012 } }, { "addendum": null, "api_flag": true, "api_subset": false, "author": "Sqwish", "axes": { "calibration": 84.10309591585386, "cost": 62.821416134464876, "intelligence": 62.182714942969874, "speed": 95.09167642607092 }, "calibration": { "parts": { "choice": { "ece_hard": 0.09316854395599922, "ece_score": 81.36629120880016, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 81.36629120880016 }, "noul": { "abstention_rate": 0.13404825737265413, "brier": 0.09010097510472304, "ece": 0.0736929319861733, "n": 373, "score": 85.26141360276533 }, "score": { "interval90_coverage": 0.8736559139784946, "interval90_width": 0.24484767025089602, "n": 372, "nrps": 0.08489712327077574, "score": 85.68158293599608, "top_ece": 0.10073560900465137 } }, "score": 84.10309591585386 }, "capability": 73.14290542941187, "composite_ci95": { "A": [ 70.78143738671345, 75.04123375455337 ], "B": [ 67.48712532292784, 73.09808670441342 ], "C": [ 62.98636206016132, 75.04123375455337 ] }, "cost": { "basis": "Sqwish public price list, USD 0.04/M input, output not billed (docs.sqwish.ai/guides/pricing, fetched 16 Oct 2026 16:17Z, saved copy sha256 698a29e5...): kettei chub at USD 0.04 per million input tokens for decisions; \"The models generate no output tokens, and none are billed\"; \"standard list prices, the same since launch on 14 October 2026\". Counted as the manufacturer's standard launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops, SubQ Snap, d1-ultra, Sage 1.4 and Nimble), through a scoped re-freeze of the frozen pricing module that adds the two kettei keys and nothing else, with a zero-difference proof for every other registry entry. The once-per-inbox USD 5.00 starter credit is a trial credit and is not used. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (650,605 input tokens over the 1,485 answered items, 0 output tokens), divided by the full 1,500-row basis the scorer uses; the 15 items refused with HTTP 422 count as zero tokens and as wrong answers, as every refused request on this draw does (d1-ultra's identical 15 refusals included). The unchanged runner also wrote a conservative cost ceiling into those 15 refused rows (65,536 input tokens each, the model's input limit); the scoring copy drops that one field on those 15 rows only. With the ceiling charged, the figure would be USD 0.043564 per 1,000 decisions (0.674x the official cost cap); that is a sensitivity, not this row's cost. Not a bill and not a quote.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.017349466666666667 }, "display": "kettei-chub (Sqwish)", "endpoint_condition": "Vendor-hosted api.sqwish.ai/v1/systemone API, model `kettei-chub` (Sqwish docs: kettei-chub v1.0, released 14 Oct 2026), through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `sqwish` before the first request (custody #20263, deputy #20305/#20306). Native probabilities on all three question types. The runner recorded kettei-chub as the answering model on all 1,500 rows (the adapter reads the response's model field), so no row was answered by a fallback size. 15 of 1,500 items exceed the model's documented 65,536-token input limit (HTTP 422 context_too_long) and are scored as failures. Paid public fast-lane order 6e10efe6, run with the customer's own API key; hosting implementation and deployed weights were not independently inspected.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 65.83625821719068, "I_sealed": 58.52917166874907, "base": 62.182714942969874, "excess": -2.532440221914108, "gap": 7.307086548441603, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 74.058241050527, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 96.10810810810811, "hard": 62.25228059138094, "judge": 80.22412656558998, "standard": 77.39640016743407 } }, "open|noul": { "cc": 71.38647342995169, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 56.52173913043479, "judge": 70.0, "standard": 88.88888888888889 } }, "open|score": { "cc": 52.064060171093324, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 59.46376557960574, "hard": 63.16197369506109, "judge": 50.34751163838864, "standard": 28.74320321795858 } }, "sealed|choice": { "cc": 74.30006625391061, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 88.56416772554002, "hard": 69.81612119810339, "judge": 69.14789422135163, "standard": 83.86416367854883 } }, "sealed|noul": { "cc": 54.66369142951747, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 86.20689655172413, "hard": 41.81818181818182, "judge": 56.92307692307692, "standard": 61.19402985074627 } }, "sealed|score": { "cc": 46.62375732281914, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 75.69949020400291, "hard": 37.65779302682666, "judge": 46.53750160134448, "standard": 50.147203056424175 } } } }, "jevbench_score": 73.5211485770867, "key": "kettei-chub", "last_measured_on": "2026-10-16", "licence": "Proprietary hosted API (no weights release)", "listing": "ranked", "measured_in": "v1.6.24", "measurement_date_status": "measured", "model_pin": "hosted-api:api.sqwish.ai/v1/systemone#kettei-chub", "noul_decisive": { "acc_among_decisive": 0.9380804953560371, "decisive": 323, "decisive_rate": 0.8659517426273459, "n": 375, "supported": true, "valid": 373 }, "paid_fast_lane": true, "planned_weight_denominator": 1.0, "rank": 3, "ranked": true, "ranks": { "A": 3, "B": 3, "C": 3, "capability": 14 }, "repo": "https://docs.sqwish.ai", "scores": { "A": 73.5211485770867, "B": 70.93430771264555, "C": 73.5211485770867 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.16667446299106814, "p50_s_raw": 0.16667446299106814, "p95_s_adjusted": 0.18576474376022814, "p95_s_raw": 0.18576474376022814 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.9090909090909091, "cc": 63.43823636827809, "cc_by_type": { "choice": 83.33333333333333, "noul": 100.0, "score": 6.981375771500953 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 63.43823636827809 }, "calibration_gen": { "accuracy": 0.801980198019802, "cc": 63.075183513321825, "cc_by_type": { "choice": 87.27465535524921, "noul": 60.00000000000001, "score": 41.950895184716266 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 63.075183513321825 }, "composite_gen": { "accuracy": 0.9702970297029703, "cc": 94.94824383168736, "cc_by_type": { "choice": 96.7479674796748, "noul": 100.0, "score": 88.09676401538727 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 94.94824383168736 }, "fairness": { "accuracy": 0.9166666666666666, "cc": 79.68499498178473, "cc_by_type": { "choice": 100.0, "noul": 46.66666666666666, "score": 92.38831827868752 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 79.68499498178473 }, "jaggedness": { "accuracy": 0.5888888888888889, "cc": 24.7828907621076, "cc_by_type": { "choice": 74.01347449470644, "noul": 8.108108108108114, "score": -7.772910316491766 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 24.7828907621076 }, "llm_router": { "accuracy": 0.8146964856230032, "cc": 71.82742478150287, "cc_by_type": { "choice": 68.3014880690323, "noul": 83.50515463917525, "score": 63.675631636301034 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 71.82742478150287 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.5555555555555556, "cc": 23.877597944583083, "cc_by_type": { "choice": 25.773195876288657, "noul": 25.0, "score": 20.859597957460597 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 23.877597944583083 }, "multilingual": { "accuracy": 0.7709090909090909, "cc": 63.90048221318355, "cc_by_type": { "choice": 74.5679817411151, "noul": 61.111111111111114, "score": 56.02235378732444 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 63.90048221318355 }, "negation_pair_gen": { "accuracy": 0.872093023255814, "cc": 72.10157892455614, "cc_by_type": { "choice": 93.33333333333333, "noul": 53.33333333333334, "score": 69.63807010700175 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 72.10157892455614 }, "option_order": { "accuracy": 0.9375, "cc": 92.01065246338216, "cc_by_type": { "choice": 92.01065246338216 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 92.01065246338216 }, "perturb_pair": { "accuracy": 0.921875, "cc": 86.78313259496274, "cc_by_type": { "choice": 87.98798798798799, "noul": 100.0, "score": 72.36140979690022 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 86.78313259496274 }, "usecase": { "accuracy": 0.6818181818181818, "cc": 55.999296711189196, "cc_by_type": { "choice": 64.4607843137255, "noul": 50.0, "score": 53.53710581984207 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 55.999296711189196 } }, "lang": { "ar": { "accuracy": 0.625, "cc": 28.24640647098315, "cc_by_type": { "choice": 42.857142857142854, "noul": 50.0, "score": -8.117923444193398 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 28.24640647098315 }, "cs": { "accuracy": 0.7142857142857143, "cc": 78.94736842105263, "cc_by_type": { "choice": 57.89473684210526, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 78.94736842105263 }, "da": { "accuracy": 0.7692307692307693, "cc": 71.36869026296806, "cc_by_type": { "choice": 58.52534562211981, "noul": 100.0, "score": 55.58072516678434 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 71.36869026296806 }, "de": { "accuracy": 0.875, "cc": 83.2987215870243, "cc_by_type": { "choice": 68.14159292035399, "noul": 100.0, "score": 81.7545718407189 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 83.2987215870243 }, "el": { "accuracy": 0.7333333333333333, "cc": 71.77688256637983, "cc_by_type": { "choice": 49.720670391061454, "noul": 66.66666666666667, "score": 98.94331064141133 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 71.77688256637983 }, "en": { "accuracy": 0.7732406602953953, "cc": 61.55931282182781, "cc_by_type": { "choice": 75.39693193883403, "noul": 61.15107913669065, "score": 48.12992738995876 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 61.55931282182781 }, "es": { "accuracy": 0.8709677419354839, "cc": 76.64478118166096, "cc_by_type": { "choice": 90.2120717781403, "noul": 100.0, "score": 39.722271766842574 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 76.64478118166096 }, "fi": { "accuracy": 0.75, "cc": 65.73563989131308, "cc_by_type": { "choice": 67.21311475409836, "noul": 33.33333333333333, "score": 96.66047158650755 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 65.73563989131308 }, "fr": { "accuracy": 0.7857142857142857, "cc": 79.01802864795974, "cc_by_type": { "choice": 57.64705882352941, "noul": 100.0, "score": 79.40702712034981 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 79.01802864795974 }, "hi": { "accuracy": 0.6666666666666666, "cc": 24.206837327473764, "cc_by_type": { "choice": 49.720670391061454, "noul": 50.0, "score": -27.10015840864015 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 24.206837327473764 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8666666666666667, "cc": 76.75527283143386, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": 44.295436576920444 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 76.75527283143386 }, "ja": { "accuracy": 0.72, "cc": 48.64168346881549, "cc_by_type": { "choice": 85.70293884034949, "noul": 60.00000000000001, "score": 0.22211156609699412 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 48.64168346881549 }, "ko": { "accuracy": 0.6, "cc": 49.09081184412445, "cc_by_type": { "choice": 100.0, "noul": 0.0, "score": 47.27243553237336 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 49.09081184412445 }, "mixed": { "accuracy": 1.0, "cc": 98.50661742898735, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 95.51985228696206 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 98.50661742898735 }, "nl": { "accuracy": 0.7, "cc": 60.007042228304385, "cc_by_type": { "choice": 48.05194805194805, "noul": 33.33333333333333, "score": 98.63584529963177 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 60.007042228304385 }, "no": { "accuracy": 0.6666666666666666, "cc": 50.0, "cc_by_type": { "choice": 100.0, "noul": 0.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 50.0 }, "pl": { "accuracy": 0.7692307692307693, "cc": 61.31631791133893, "cc_by_type": { "choice": 92.382564536606, "noul": 39.99999999999999, "score": 51.566389197410786 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 61.31631791133893 }, "pt": { "accuracy": 0.7857142857142857, "cc": 63.091184704198746, "cc_by_type": { "choice": 68.1791396582204, "noul": 60.00000000000001, "score": 61.09441445437582 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 63.091184704198746 }, "sv": { "accuracy": 0.6363636363636364, "cc": 54.828846215251076, "cc_by_type": { "choice": 48.93617021276595, "noul": 60.00000000000001, "score": 55.55036843298726 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 54.828846215251076 }, "tr": { "accuracy": 0.7692307692307693, "cc": 56.63034829618355, "cc_by_type": { "choice": 78.26086956521739, "noul": 100.0, "score": -8.369824676666738 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 56.63034829618355 }, "uk": { "accuracy": 0.8571428571428571, "cc": 82.98189368154357, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 81.20374556075973 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 82.98189368154357 }, "zh": { "accuracy": 0.8, "cc": 48.085566824177704, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 10.92336713919978 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 48.085566824177704 } }, "type": { "choice": { "accuracy": 0.8066666666666666, "cc": 75.02822480338315, "cc_by_type": { "choice": 75.02822480338315 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 75.02822480338315 }, "noul": { "accuracy": 0.808, "cc": 61.60000000000001, "cc_by_type": { "noul": 61.60000000000001 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 61.60000000000001 }, "score": { "accuracy": 0.672, "cc": 49.184305491330974, "cc_by_type": { "score": 49.184305491330974 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 49.184305491330974 } }, "usecase": { "advertising": { "accuracy": 0.5454545454545454, "cc": 32.94973949397677, "cc_by_type": { "choice": 36.0, "noul": 25.0, "score": 37.8492184819303 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 32.94973949397677 }, "code_linting": { "accuracy": 0.5416666666666666, "cc": 21.86713652267862, "cc_by_type": { "choice": 44.099378881987576, "noul": 33.33333333333333, "score": -11.831302647285046 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 21.86713652267862 }, "customer_support": { "accuracy": 0.7857142857142857, "cc": 58.41468804345843, "cc_by_type": { "choice": 67.47967479674797, "noul": 100.0, "score": 7.764389333627342 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 58.41468804345843 }, "demand_forecasting": { "accuracy": 0.4117647058823529, "cc": 21.307121699711207, "cc_by_type": { "choice": 6.432748538011691, "noul": 33.33333333333333, "score": 24.155283227788605 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 21.307121699711207 }, "ecommerce_marketplaces": { "accuracy": 0.7058823529411765, "cc": 52.732202808738386, "cc_by_type": { "choice": 71.0144927536232, "noul": 33.33333333333333, "score": 53.84878233925865 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 52.732202808738386 }, "feature_extraction": { "accuracy": 0.631578947368421, "cc": -17.90003745614136, "cc_by_type": { "choice": 37.106918238993714, "noul": 66.66666666666667, "score": -157.47369727408446 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.6333333333333333, "cc": 50.15463162953395, "cc_by_type": { "choice": 70.58823529411765, "noul": 42.85714285714286, "score": 37.018516737341336 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 50.15463162953395 }, "gaming": { "accuracy": 0.6818181818181818, "cc": 46.19866186338035, "cc_by_type": { "choice": 58.18815331010453, "noul": 0.0, "score": 80.40783228003652 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 46.19866186338035 }, "insurance_claims": { "accuracy": 0.6, "cc": 55.891544536440826, "cc_by_type": { "choice": 37.5, "noul": 60.00000000000001, "score": 70.1746336093225 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 55.891544536440826 }, "knowledge_graphs": { "accuracy": 0.7272727272727273, "cc": 61.85269108394834, "cc_by_type": { "choice": 74.30406852248395, "noul": 66.66666666666667, "score": 44.58733806269443 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 61.85269108394834 }, "lead_generation": { "accuracy": 0.76, "cc": 72.34106793586746, "cc_by_type": { "choice": 88.3720930232558, "noul": 77.77777777777777, "score": 50.87333300656878 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 72.34106793586746 }, "legal_compliance": { "accuracy": 0.6, "cc": 40.88339357227535, "cc_by_type": { "choice": 67.74193548387096, "noul": 33.33333333333333, "score": 21.574911899621764 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 40.88339357227535 }, "llm_guardrails": { "accuracy": 0.7407407407407407, "cc": 51.77378065698675, "cc_by_type": { "choice": 76.96737044145875, "noul": 19.999999999999996, "score": 58.3539715295015 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 51.77378065698675 }, "model_routing": { "accuracy": 0.8847926267281107, "cc": 78.0080271816572, "cc_by_type": { "choice": 84.80519480519482, "noul": 96.96969696969697, "score": 52.24918977007981 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 78.0080271816572 }, "moderation_trust_safety": { "accuracy": 0.6666666666666666, "cc": 41.95050174639673, "cc_by_type": { "choice": 65.4510556621881, "noul": 0.0, "score": 60.40044957700209 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 41.95050174639673 }, "recruiting": { "accuracy": 0.7222222222222222, "cc": 56.20139697883294, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 68.69704144717662 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 56.20139697883294 }, "risk_assessment": { "accuracy": 0.5, "cc": 36.63384674487239, "cc_by_type": { "choice": 40.88669950738917, "noul": 50.0, "score": 19.014840727228023 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 36.63384674487239 }, "scientific_discovery": { "accuracy": 0.6521739130434783, "cc": 40.942225675125954, "cc_by_type": { "choice": 56.52173913043478, "noul": 0.0, "score": 66.30493789494308 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 40.942225675125954 }, "search_retrieval": { "accuracy": 0.9583333333333334, "cc": 94.41000709626154, "cc_by_type": { "choice": 89.3238434163701, "noul": 100.0, "score": 93.90617787241452 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 94.41000709626154 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 79.21230885259558, 86.02583441129816 ], "I": [ 56.503594594655, 66.61118532910479 ], "capability": [ 68.56105847631343, 75.56327474473197 ], "composite": { "A": [ 70.78143738671345, 75.04123375455337 ], "B": [ 67.48712532292784, 73.09808670441342 ], "C": [ 62.98636206016132, 75.04123375455337 ] }, "seed": 27 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 84.10309591585386, "capability": 73.14290542941187, "intelligence": 62.182714942969874 }, "per_type": { "choice": { "accuracy": 0.8066666666666666, "calibration": 81.36629120880016, "cc": 75.02822480338315, "cc_by_type": { "choice": 75.02822480338315 }, "cc_tiered": 74.17915365221882, "cc_tiered_open": 74.058241050527, "cc_tiered_sealed": 74.30006625391061, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 75.02822480338315 }, "noul": { "accuracy": 0.808, "calibration": 85.26141360276533, "cc": 61.60000000000001, "cc_by_type": { "noul": 61.60000000000001 }, "cc_tiered": 63.02508242973458, "cc_tiered_open": 71.38647342995169, "cc_tiered_sealed": 54.66369142951747, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 61.60000000000001 }, "score": { "accuracy": 0.672, "calibration": 85.68158293599608, "cc": 49.184305491330974, "cc_by_type": { "score": 49.184305491330974 }, "cc_tiered": 49.34390874695623, "cc_tiered_open": 52.064060171093324, "cc_tiered_sealed": 46.62375732281914, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 49.184305491330974 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 70.56176150481356, "Emphasis on Accuracy 60:20:20": 66.95302542436683, "Emphasis on Cost 20:20:60": 67.24748560406725, "Emphasis on Speed 20:60:20": 78.68033506525428, "Intelligence only": 62.182714942969874 } }, { "addendum": null, "api_flag": true, "api_subset": false, "author": "Sqwish", "axes": { "calibration": 92.78078361709031, "cost": 57.53867836279444, "intelligence": 79.83918785475791, "speed": 93.19124027169539 }, "calibration": { "parts": { "choice": { "ece_hard": 0.024857318118089264, "ece_score": 95.02853637638215, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 95.02853637638215 }, "noul": { "abstention_rate": 0.04557640750670244, "brier": 0.04418057925447822, "ece": 0.03819155325961758, "n": 373, "score": 92.36168934807648 }, "score": { "interval90_coverage": 0.9301075268817204, "interval90_width": 0.1270161290322581, "n": 372, "nrps": 0.044540233632597244, "score": 90.95212512681232, "top_ece": 0.0682086319155783 } }, "score": 92.78078361709031 }, "capability": 86.3099857359241, "composite_ci95": { "A": [ 76.35150719879172, 78.53169061641754 ], "B": [ 76.27234694852882, 79.41308115212806 ], "C": [ 76.35150719879172, 78.53169061641754 ] }, "cost": { "basis": "Sqwish public price list, USD 0.06/M input, output not billed (docs.sqwish.ai/guides/pricing, fetched 16 Oct 2026 16:17Z, saved copy sha256 698a29e5...): kettei fat at USD 0.06 per million input tokens for decisions; \"The models generate no output tokens, and none are billed\"; \"standard list prices, the same since launch on 14 October 2026\". Counted as the manufacturer's standard launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops, SubQ Snap, d1-ultra, Sage 1.4 and Nimble), through a scoped re-freeze of the frozen pricing module that adds the two kettei keys and nothing else, with a zero-difference proof for every other registry entry. The once-per-inbox USD 5.00 starter credit is a trial credit and is not used. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (650,605 input tokens over the 1,485 answered items, 0 output tokens), divided by the full 1,500-row basis the scorer uses; the 15 items refused with HTTP 422 count as zero tokens and as wrong answers, as every refused request on this draw does (d1-ultra's identical 15 refusals included). The unchanged runner also wrote a conservative cost ceiling into those 15 refused rows (65,536 input tokens each, the model's input limit); the scoring copy drops that one field on those 15 rows only. With the ceiling charged, the figure would be USD 0.065346 per 1,000 decisions (1.012x the official cost cap); that is a sensitivity, not this row's cost. Not a bill and not a quote.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.026024199999999994 }, "display": "kettei-fat (Sqwish)", "endpoint_condition": "Vendor-hosted api.sqwish.ai/v1/systemone API, model `kettei-fat` (Sqwish docs: kettei-fat v1.0, released 14 Oct 2026), through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `sqwish` before the first request (custody #20263, deputy #20305/#20306). Native probabilities on all three question types. The runner recorded kettei-fat as the answering model on all 1,500 rows (the adapter reads the response's model field), so no row was answered by a fallback size. 15 of 1,500 items exceed the model's documented 65,536-token input limit (HTTP 422 context_too_long) and are scored as failures. Paid public fast-lane order a87f712c, run with the customer's own API key; hosting implementation and deployed weights were not independently inspected.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 79.51365708398285, "I_sealed": 80.16471862553297, "base": 79.83918785475791, "excess": -10.490588311905832, "gap": -0.6510615415501206, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 87.36347037521236, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 92.21621621621622, "hard": 86.78829820698331, "judge": 88.13447593935399, "standard": 84.93093344495604 } }, "open|noul": { "cc": 80.07729468599035, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 91.30434782608697, "judge": 60.00000000000001, "standard": 77.77777777777777 } }, "open|score": { "cc": 71.10020619074588, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 82.05684261096903, "hard": 73.02709979951186, "judge": 72.7279902420528, "standard": 59.32642468614199 } }, "sealed|choice": { "cc": 87.64254301487159, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 100.0, "hard": 81.78559037816582, "judge": 89.42213516160628, "standard": 90.50833157561695 } }, "sealed|noul": { "cc": 83.10326832200224, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 93.10344827586208, "hard": 74.54545454545453, "judge": 93.84615384615384, "standard": 79.10447761194031 } }, "sealed|score": { "cc": 69.74834453972508, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 89.41045649485207, "hard": 62.86673539910686, "judge": 69.73247418036178, "standard": 73.70431238244299 } } } }, "jevbench_score": 77.80056651426555, "key": "kettei-fat", "last_measured_on": "2026-10-16", "licence": "Proprietary hosted API (no weights release)", "listing": "ranked", "measured_in": "v1.6.24", "measurement_date_status": "measured", "model_pin": "hosted-api:api.sqwish.ai/v1/systemone#kettei-fat", "noul_decisive": { "acc_among_decisive": 0.9634831460674157, "decisive": 356, "decisive_rate": 0.9544235924932976, "n": 375, "supported": true, "valid": 373 }, "paid_fast_lane": true, "planned_weight_denominator": 1.0, "rank": 1, "ranked": true, "ranks": { "A": 1, "B": 1, "C": 1, "capability": 1 }, "repo": "https://docs.sqwish.ai", "scores": { "A": 77.80056651426555, "B": 78.19991933501608, "C": 77.80056651426555 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.20005768549162894, "p50_s_raw": 0.20005768549162894, "p95_s_adjusted": 0.2397290877619525, "p95_s_raw": 0.2397290877619525 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.9545454545454546, "cc": 96.7507472305415, "cc_by_type": { "choice": 91.66666666666667, "noul": 100.0, "score": 98.58557502495783 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 96.7507472305415 }, "calibration_gen": { "accuracy": 0.9603960396039604, "cc": 93.47881080044199, "cc_by_type": { "choice": 96.8186638388123, "noul": 92.0, "score": 91.6177685625137 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 93.47881080044199 }, "composite_gen": { "accuracy": 0.9900990099009901, "cc": 98.5804818840001, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 95.74144565200028 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 98.5804818840001 }, "fairness": { "accuracy": 0.9375, "cc": 84.86062527550072, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 94.58187582650217 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 84.86062527550072 }, "jaggedness": { "accuracy": 0.6777777777777778, "cc": 48.399546734792864, "cc_by_type": { "choice": 74.01347449470644, "noul": 35.13513513513513, "score": 36.05003057453702 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 48.399546734792864 }, "llm_router": { "accuracy": 0.952076677316294, "cc": 87.82993917915225, "cc_by_type": { "choice": 96.83014880690322, "noul": 97.93814432989691, "score": 68.72152440065662 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 87.82993917915225 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.8518518518518519, "cc": 83.41157557420702, "cc_by_type": { "choice": 62.886597938144334, "noul": 100.0, "score": 87.34812878447671 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 83.41157557420702 }, "multilingual": { "accuracy": 0.8618181818181818, "cc": 78.9798737677216, "cc_by_type": { "choice": 80.43690903162701, "noul": 86.11111111111111, "score": 70.39160116042665 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 78.9798737677216 }, "negation_pair_gen": { "accuracy": 0.9767441860465116, "cc": 94.85365625545666, "cc_by_type": { "choice": 100.0, "noul": 86.66666666666667, "score": 97.8943020997033 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 94.85365625545666 }, "option_order": { "accuracy": 0.9375, "cc": 92.01065246338216, "cc_by_type": { "choice": 92.01065246338216 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 92.01065246338216 }, "perturb_pair": { "accuracy": 0.9375, "cc": 89.64749433680471, "cc_by_type": { "choice": 93.993993993994, "noul": 100.0, "score": 74.94848901642015 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 89.64749433680471 }, "usecase": { "accuracy": 0.8590909090909091, "cc": 79.73490795932031, "cc_by_type": { "choice": 88.97058823529412, "noul": 75.0, "score": 75.2341356426668 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 79.73490795932031 } }, "lang": { "ar": { "accuracy": 0.875, "cc": 86.21024275740876, "cc_by_type": { "choice": 80.95238095238095, "noul": 100.0, "score": 77.67834731984534 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 86.21024275740876 }, "cs": { "accuracy": 0.8571428571428571, "cc": 89.47368421052632, "cc_by_type": { "choice": 78.94736842105263, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 89.47368421052632 }, "da": { "accuracy": 0.8461538461538461, "cc": 75.04719104747973, "cc_by_type": { "choice": 79.26267281105991, "noul": 100.0, "score": 45.87890033137929 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 75.04719104747973 }, "de": { "accuracy": 0.875, "cc": 86.628058606648, "cc_by_type": { "choice": 68.14159292035399, "noul": 100.0, "score": 91.74258289959003 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 86.628058606648 }, "el": { "accuracy": 0.8, "cc": 77.61393066873585, "cc_by_type": { "choice": 66.4804469273743, "noul": 66.66666666666667, "score": 99.69467841216657 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 77.61393066873585 }, "en": { "accuracy": 0.8835794960903562, "cc": 80.4883103640632, "cc_by_type": { "choice": 89.67549822433213, "noul": 80.57553956834533, "score": 71.21389329951214 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 80.4883103640632 }, "es": { "accuracy": 0.967741935483871, "cc": 90.39349165083537, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 71.18047495250612 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 90.39349165083537 }, "fi": { "accuracy": 0.875, "cc": 77.5139405386418, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 99.2084882825921 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 77.5139405386418 }, "fr": { "accuracy": 0.9285714285714286, "cc": 91.9301217185329, "cc_by_type": { "choice": 78.82352941176471, "noul": 100.0, "score": 96.966835743834 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 91.9301217185329 }, "hi": { "accuracy": 0.9047619047619048, "cc": 79.08766846078485, "cc_by_type": { "choice": 83.24022346368716, "noul": 100.0, "score": 54.022781918667405 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 79.08766846078485 }, "id": { "accuracy": 0.75, "cc": 77.77777777777777, "cc_by_type": { "choice": 55.55555555555555, "noul": 100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 77.77777777777777 }, "it": { "accuracy": 0.9333333333333333, "cc": 80.52064370654271, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": 55.59154920224702 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 80.52064370654271 }, "ja": { "accuracy": 0.92, "cc": 80.92306844140934, "cc_by_type": { "choice": 100.0, "noul": 80.0, "score": 62.76920532422802 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 80.92306844140934 }, "ko": { "accuracy": 0.8, "cc": 85.64585125280645, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 56.93755375841938 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 85.64585125280645 }, "mixed": { "accuracy": 1.0, "cc": 97.20181470806706, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 91.60544412420121 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 97.20181470806706 }, "nl": { "accuracy": 0.7, "cc": 15.179397099416787, "cc_by_type": { "choice": 74.02597402597404, "noul": 33.33333333333333, "score": -61.821116061057005 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 15.179397099416787 }, "no": { "accuracy": 0.8333333333333334, "cc": 67.74193548387098, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 67.74193548387098 }, "pl": { "accuracy": 0.8717948717948718, "cc": 86.0234580413718, "cc_by_type": { "choice": 84.76512907321202, "noul": 100.0, "score": 73.30524505090337 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 86.0234580413718 }, "pt": { "accuracy": 0.8214285714285714, "cc": 75.9847623403091, "cc_by_type": { "choice": 68.1791396582204, "noul": 100.0, "score": 59.7751473627069 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 75.9847623403091 }, "sv": { "accuracy": 0.7272727272727273, "cc": 67.67066900776906, "cc_by_type": { "choice": 48.93617021276595, "noul": 100.0, "score": 54.07583681054125 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 67.67066900776906 }, "tr": { "accuracy": 0.8461538461538461, "cc": 83.57430186853024, "cc_by_type": { "choice": 78.26086956521739, "noul": 100.0, "score": 72.46203604037333 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 83.57430186853024 }, "uk": { "accuracy": 0.8571428571428571, "cc": 81.78317833096776, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 77.60759950903231 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 81.78317833096776 }, "zh": { "accuracy": 0.8, "cc": 46.85913519178663, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 7.244072242026545 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 46.85913519178663 } }, "type": { "choice": { "accuracy": 0.9053333333333333, "cc": 87.77244111062208, "cc_by_type": { "choice": 87.77244111062208 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 87.77244111062208 }, "noul": { "accuracy": 0.9146666666666666, "cc": 82.93333333333332, "cc_by_type": { "noul": 82.93333333333332 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 82.93333333333332 }, "score": { "accuracy": 0.7973333333333333, "cc": 70.94010320323034, "cc_by_type": { "score": 70.94010320323034 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 70.94010320323034 } }, "usecase": { "advertising": { "accuracy": 0.6818181818181818, "cc": 49.66614749506439, "cc_by_type": { "choice": 68.0, "noul": 50.0, "score": 30.998442485193188 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 49.66614749506439 }, "code_linting": { "accuracy": 0.9166666666666666, "cc": 91.94328546901535, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 75.82985640704605 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 91.94328546901535 }, "customer_support": { "accuracy": 0.7857142857142857, "cc": 51.2790489623809, "cc_by_type": { "choice": 67.47967479674797, "noul": 100.0, "score": -13.642527909605295 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 51.2790489623809 }, "demand_forecasting": { "accuracy": 0.47058823529411764, "cc": 14.651774389394646, "cc_by_type": { "choice": 18.128654970760234, "noul": 100.0, "score": -74.1733318025763 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 14.651774389394646 }, "ecommerce_marketplaces": { "accuracy": 0.9411764705882353, "cc": 90.47904341608493, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 71.43713024825477 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 90.47904341608493 }, "feature_extraction": { "accuracy": 0.8421052631578947, "cc": 47.918819771310574, "cc_by_type": { "choice": 87.42138364779873, "noul": 66.66666666666667, "score": -10.331591000533692 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 47.918819771310574 }, "financial_crime": { "accuracy": 0.8333333333333334, "cc": 77.48796585461322, "cc_by_type": { "choice": 100.00000000000001, "noul": 71.42857142857142, "score": 61.03532613526828 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 77.48796585461322 }, "gaming": { "accuracy": 0.7727272727272727, "cc": 66.73764536574002, "cc_by_type": { "choice": 79.09407665505228, "noul": 50.0, "score": 71.11885944216777 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 66.73764536574002 }, "insurance_claims": { "accuracy": 0.8666666666666667, "cc": 77.28653895219419, "cc_by_type": { "choice": 79.16666666666667, "noul": 60.00000000000001, "score": 92.69295018991592 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 77.28653895219419 }, "knowledge_graphs": { "accuracy": 0.8181818181818182, "cc": 75.55654372603391, "cc_by_type": { "choice": 87.15203426124197, "noul": 66.66666666666667, "score": 72.85093025019309 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 75.55654372603391 }, "lead_generation": { "accuracy": 0.88, "cc": 87.19694978627531, "cc_by_type": { "choice": 88.3720930232558, "noul": 100.0, "score": 73.21875633557016 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 87.19694978627531 }, "legal_compliance": { "accuracy": 0.7, "cc": 52.35634073814115, "cc_by_type": { "choice": 35.483870967741936, "noul": 33.33333333333333, "score": 88.25181791334818 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 52.35634073814115 }, "llm_guardrails": { "accuracy": 0.9259259259259259, "cc": 82.92727279464067, "cc_by_type": { "choice": 100.00000000000001, "noul": 60.00000000000001, "score": 88.781818383922 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 82.92727279464067 }, "model_routing": { "accuracy": 0.9262672811059908, "cc": 84.3082332710354, "cc_by_type": { "choice": 94.15584415584415, "noul": 100.0, "score": 58.76885565726209 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 84.3082332710354 }, "moderation_trust_safety": { "accuracy": 0.8333333333333334, "cc": 56.11137155389184, "cc_by_type": { "choice": 88.48368522072937, "noul": 0.0, "score": 79.85042944094617 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 56.11137155389184 }, "recruiting": { "accuracy": 0.8888888888888888, "cc": 89.83757082391922, "cc_by_type": { "choice": 83.28690807799443, "noul": 100.0, "score": 86.22580439376324 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 89.83757082391922 }, "risk_assessment": { "accuracy": 0.7777777777777778, "cc": 69.35928856610843, "cc_by_type": { "choice": 70.44334975369459, "noul": 100.0, "score": 37.63451594463072 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 69.35928856610843 }, "scientific_discovery": { "accuracy": 0.8695652173913043, "cc": 84.81707822400728, "cc_by_type": { "choice": 71.01449275362319, "noul": 100.0, "score": 83.43674191839865 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 84.81707822400728 }, "search_retrieval": { "accuracy": 1.0, "cc": 98.38110705803847, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 95.14332117411539 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 98.38110705803847 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 88.95834437759348, 93.41199196963947 ], "I": [ 75.82329828139399, 83.13267707914414 ], "capability": [ 82.88107787682691, 87.86094343703664 ], "composite": { "A": [ 76.35150719879172, 78.53169061641754 ], "B": [ 76.27234694852882, 79.41308115212806 ], "C": [ 76.35150719879172, 78.53169061641754 ] }, "seed": 28 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 92.78078361709031, "capability": 86.3099857359241, "intelligence": 79.83918785475791 }, "per_type": { "choice": { "accuracy": 0.9053333333333333, "calibration": 95.02853637638215, "cc": 87.77244111062208, "cc_by_type": { "choice": 87.77244111062208 }, "cc_tiered": 87.50300669504198, "cc_tiered_open": 87.36347037521236, "cc_tiered_sealed": 87.64254301487159, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 87.77244111062208 }, "noul": { "accuracy": 0.9146666666666666, "calibration": 92.36168934807648, "cc": 82.93333333333332, "cc_by_type": { "noul": 82.93333333333332 }, "cc_tiered": 81.5902815039963, "cc_tiered_open": 80.07729468599035, "cc_tiered_sealed": 83.10326832200224, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 82.93333333333332 }, "score": { "accuracy": 0.7973333333333333, "calibration": 90.95212512681232, "cc": 70.94010320323034, "cc_by_type": { "score": 70.94010320323034 }, "cc_tiered": 70.42427536523547, "cc_tiered_open": 71.10020619074588, "cc_tiered_sealed": 69.74834453972508, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 70.94010320323034 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 73.8272296338628, "Emphasis on Accuracy 60:20:20": 76.11998940961655, "Emphasis on Cost 20:20:60": 66.31771104702096, "Emphasis on Speed 20:60:20": 80.51963114967444, "Intelligence only": 79.83918785475791 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "guru0381", "axes": { "calibration": 62.82741496733333, "cost": 54.45313889836773, "intelligence": 21.99889248592056, "speed": 91.85485240061512 }, "calibration": { "parts": { "choice": { "ece_hard": 0.06249555555555558, "ece_score": 87.50088888888888, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 87.50088888888888 }, "noul": { "abstention_rate": 0.16533333333333333, "brier": 0.2664339086133331, "ece": 0.2359746666666666, "n": 375, "score": 52.805066666666676 }, "score": { "interval90_coverage": 0.6186666666666667, "interval90_width": 0.2953333333333333, "n": 375, "nrps": 0.21633341307111098, "score": 48.17628934644445, "top_ece": 0.4100704 } }, "score": 62.82741496733333 }, "capability": 42.413153726626945, "composite_ci95": { "A": [ 3.6455898800704327, 13.977339719064457 ], "B": [ 2.885102056071331, 12.023251156152215 ], "C": [ 2.5316596389378008, 9.706485916016982 ] }, "cost": { "basis": "v1.5-M2 base-model reference: frozen DeepInfra snapshot jevbench-v15-measure-20260925 receipts/DEEPINFRA-MODELS-20260925.json (SHA-256 84eb7926acd4e93b71d695be24898e2ca999b20011d1b66cc0348ce7c6c3546f); listing Qwen/Qwen3.5-2B, USD 0.02/M input and USD 0.10/M output. The released checkpoint names Qwen/Qwen3.5-2B-Base; the frozen DeepInfra listing omits the -Base suffix. The requester cited the Qwen3.5-4B USD 0.03/M input reference, but the matching 2B family listing is used here (precedent classone-qwen3.5-2b). Self-hosted GPU deployment has no provider tariff, so price_kind is estimate. Kev generates no free-form text. However, kev/api.py output_tokens() tokenizes JSON-serialized answers, so reported usage.output_tokens is nonzero for normal answers; it is not zero by construction.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.03297833333333333 }, "display": "Krino-2B", "endpoint_condition": "evaluator-owned Lium GPU pod (H100), offline read-only container (--network none); author's `python3 -m krino.serve` (Kev @fe64b127 serve path, documented per-type temperature map choice 1.95 / noul 0.30 / score 0.50 from krino.json), KEV_TRUNCATE_STATES=1, reference kernels (KEV_FUSED=0: the documented pip install has no flash-linear-attention) Paid fast-lane order 8c7dbe11 (public). Kev architecture (LoRA r16 + pointer head) on Qwen/Qwen3.5-2B-Base @b1485b2f. Runtime: reviewed pytorch 2.11 base + Kev uv.lock closure (torch 2.8.0) installed offline from SHA-256-pinned wheels (PR #335). Kev reports usage.output_tokens as the token count of the serialized answer JSON (nothing is generated); priced as reported at the reference output rate.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 26.37590231932532, "I_sealed": 17.6218826525158, "base": 21.99889248592056, "excess": -1.0855071035461918, "gap": 8.75401966680952, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 36.29349492821419, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 49.40540540540541, "hard": 37.71626297577854, "judge": 32.76203032300594, "standard": 32.18920050230222 } }, "open|noul": { "cc": 21.265010351966872, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 57.14285714285714, "hard": 13.043478260869556, "judge": -9.999999999999998, "standard": 66.66666666666667 } }, "open|score": { "cc": 21.5692016777949, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 51.23363636363638, "hard": 9.25462962962964, "judge": 16.76983050847457, "standard": 38.56518518518517 } }, "sealed|choice": { "cc": 36.28277020740255, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 46.632782719186785, "hard": 30.785243437030175, "judge": 35.651322233104814, "standard": 43.049989453701755 } }, "sealed|noul": { "cc": 16.58287775014486, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 24.13793103448276, "hard": 9.090909090909083, "judge": 26.15384615384615, "standard": 13.432835820895516 } }, "sealed|score": { "cc": 0.0, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": -5.103771929824585, "hard": -28.450816901408405, "judge": -18.68588541666667, "standard": 12.30559090909088 } } } }, "jevbench_score": 8.54428010154306, "key": "krino-2b", "last_measured_on": "2026-10-12", "licence": "Apache-2.0 (adapter, head and code; Kev Apache-2.0; base Qwen3.5-2B-Base Apache-2.0)", "listing": "ranked", "measured_in": "v1.6.10", "measurement_date_status": "measured", "model_pin": "Guru0381/krino-2b@ff185abf129206174c67ebbe6058cab4cc9ccaed", "noul_decisive": { "acc_among_decisive": 0.7124600638977636, "decisive": 313, "decisive_rate": 0.8346666666666667, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 27, "ranked": true, "ranks": { "A": 27, "B": 27, "C": 27, "capability": 32 }, "repo": "https://huggingface.co/Guru0381/krino-2b", "scores": { "A": 8.54428010154306, "B": 7.11267113145192, "C": 5.933527848293792 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.21046640197746455, "p50_s_raw": 0.03023320098873228, "p95_s_adjusted": 0.30997879329370326, "p95_s_raw": 0.07998939664685163 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.45454545454545453, "cc": -3.9711111111111044, "cc_by_type": { "choice": 16.666666666666668, "noul": 50.0, "score": -78.57999999999998 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 0.0 }, "calibration_gen": { "accuracy": 0.504950495049505, "cc": 31.44464464033227, "cc_by_type": { "choice": 49.09862142099682, "noul": 52.0, "score": -6.764687500000011 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 31.44464464033227 }, "composite_gen": { "accuracy": 0.36633663366336633, "cc": -31.327821902578023, "cc_by_type": { "choice": 41.463414634146346, "noul": -11.111111111111116, "score": -124.3357692307693 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 0.0 }, "fairness": { "accuracy": 0.6041666666666666, "cc": 48.12166666666667, "cc_by_type": { "choice": 100.0, "noul": 6.666666666666665, "score": 37.69833333333336 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 48.12166666666667 }, "jaggedness": { "accuracy": 0.35, "cc": 1.3896814773217478, "cc_by_type": { "choice": 17.70933589990377, "noul": -18.918918918918916, "score": 5.378627450980389 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 1.3896814773217478 }, "llm_router": { "accuracy": 0.670926517571885, "cc": 54.443170038415836, "cc_by_type": { "choice": 50.86730650700009, "noul": 64.94845360824742, "score": 47.51374999999999 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 54.443170038415836 }, "long_state": { "accuracy": 0.26666666666666666, "cc": -24.669193548387096, "cc_by_type": { "choice": 9.677419354838706, "noul": -100.0, "score": 16.315 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.2222222222222222, "cc": -23.08969072164948, "cc_by_type": { "choice": 1.0309278350515454, "noul": -25.0, "score": -45.29999999999998 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.5018181818181818, "cc": 15.536204017083938, "cc_by_type": { "choice": 39.3544179980437, "noul": 13.888888888888884, "score": -6.634694835680777 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 15.536204017083938 }, "negation_pair_gen": { "accuracy": 0.38372093023255816, "cc": -51.32922222222222, "cc_by_type": { "choice": 26.666666666666668, "noul": -13.33333333333333, "score": -167.321 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.0 }, "option_order": { "accuracy": 0.375, "cc": 20.106524633821575, "cc_by_type": { "choice": 20.106524633821575 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 20.106524633821575 }, "perturb_pair": { "accuracy": 0.625, "cc": 45.63698198198199, "cc_by_type": { "choice": 45.94594594594596, "noul": 60.00000000000001, "score": 30.964999999999996 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 45.63698198198199 }, "usecase": { "accuracy": 0.41363636363636364, "cc": 5.069023006535965, "cc_by_type": { "choice": 31.372549019607874, "noul": -12.5, "score": -3.6654799999999765 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 5.069023006535965 } }, "lang": { "ar": { "accuracy": 0.1875, "cc": -33.39369963369963, "cc_by_type": { "choice": -14.285714285714286, "noul": -50.0, "score": -35.895384615384614 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 0.46153846153846156, "cc": -9.327886840757811, "cc_by_type": { "choice": 79.26267281105991, "noul": -33.333333333333336, "score": -73.91300000000001 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 0.0 }, "de": { "accuracy": 0.625, "cc": 45.03688341669872, "cc_by_type": { "choice": 52.21238938053097, "noul": 50.0, "score": 32.89826086956522 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 45.03688341669872 }, "el": { "accuracy": 0.26666666666666666, "cc": 12.071002482929854, "cc_by_type": { "choice": -0.5586592178770929, "noul": -33.333333333333336, "score": 70.10499999999999 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 12.071002482929854 }, "en": { "accuracy": 0.4778453518679409, "cc": 15.730797082087726, "cc_by_type": { "choice": 35.85628684053157, "noul": 17.266187050359715, "score": -5.93008264462811 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 15.730797082087726 }, "es": { "accuracy": 0.5483870967741935, "cc": 11.50404119886126, "cc_by_type": { "choice": 51.06035889070147, "noul": 75.0, "score": -91.54823529411769 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 11.50404119886126 }, "fi": { "accuracy": 0.375, "cc": 4.108670309653917, "cc_by_type": { "choice": 1.6393442622950836, "noul": 33.33333333333333, "score": -22.64666666666666 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 4.108670309653917 }, "fr": { "accuracy": 0.42857142857142855, "cc": 11.742827657378745, "cc_by_type": { "choice": 36.47058823529412, "noul": 0.0, "score": -1.2421052631578888 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 11.742827657378745 }, "hi": { "accuracy": 0.38095238095238093, "cc": -23.99129422718808, "cc_by_type": { "choice": 16.201117318435756, "noul": 0.0, "score": -88.175 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 0.0 }, "id": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0, "noul": 100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 100.0 }, "it": { "accuracy": 0.8, "cc": 18.527619047619037, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": -144.4171428571429 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 18.527619047619037 }, "ja": { "accuracy": 0.52, "cc": 21.515655458476743, "cc_by_type": { "choice": 14.2176330420969, "noul": 39.99999999999999, "score": 10.329333333333345 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 21.515655458476743 }, "ko": { "accuracy": 0.3, "cc": -12.233253968253969, "cc_by_type": { "choice": -33.333333333333336, "noul": 0.0, "score": -3.36642857142857 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.7857142857142857, "cc": 79.685, "cc_by_type": { "choice": 66.66666666666667, "noul": 100.0, "score": 72.38833333333332 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 79.685 }, "nl": { "accuracy": 0.5, "cc": 26.801093795093788, "cc_by_type": { "choice": 48.05194805194805, "noul": 33.33333333333333, "score": -0.9820000000000162 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 26.801093795093788 }, "no": { "accuracy": 0.0, "cc": -64.51612903225806, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.6923076923076923, "cc": 42.704345349985395, "cc_by_type": { "choice": 84.76512907321202, "noul": 39.99999999999999, "score": 3.3479069767441416 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 42.704345349985395 }, "pt": { "accuracy": 0.5, "cc": 21.2760213786505, "cc_by_type": { "choice": 4.537418974661164, "noul": 60.00000000000001, "score": -0.7093548387096638 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 21.2760213786505 }, "sv": { "accuracy": 0.2727272727272727, "cc": 4.250780141843975, "cc_by_type": { "choice": -2.127659574468081, "noul": -19.999999999999996, "score": 34.88 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 4.250780141843975 }, "tr": { "accuracy": 0.5384615384615384, "cc": 3.729202898550722, "cc_by_type": { "choice": 34.78260869565218, "noul": 50.0, "score": -73.59500000000001 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 3.729202898550722 }, "uk": { "accuracy": 0.7142857142857143, "cc": 77.39897849462365, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 64.455 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 77.39897849462365 }, "zh": { "accuracy": 0.5, "cc": 3.433069679849351, "cc_by_type": { "choice": 49.152542372881356, "noul": -33.333333333333336, "score": -5.519999999999969 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 3.433069679849351 } }, "type": { "choice": { "accuracy": 0.512, "cc": 36.96779502095333, "cc_by_type": { "choice": 36.96779502095333 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 36.96779502095333 }, "noul": { "accuracy": 0.5946666666666667, "cc": 18.933333333333337, "cc_by_type": { "noul": 18.933333333333337 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 18.933333333333337 }, "score": { "accuracy": 0.328, "cc": -6.174584837544961, "cc_by_type": { "score": -6.174584837544961 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "usecase": { "advertising": { "accuracy": 0.3181818181818182, "cc": -42.54725490196079, "cc_by_type": { "choice": 36.0, "noul": -50.0, "score": -113.64176470588237 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.3333333333333333, "cc": -12.739704039921422, "cc_by_type": { "choice": 6.832298136645961, "noul": -33.333333333333336, "score": -11.718076923076893 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 0.0 }, "customer_support": { "accuracy": 0.7857142857142857, "cc": 55.12989159891598, "cc_by_type": { "choice": 67.47967479674797, "noul": 100.0, "score": -2.090000000000014 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 55.12989159891598 }, "demand_forecasting": { "accuracy": 0.47058823529411764, "cc": 34.89788499025341, "cc_by_type": { "choice": 18.128654970760234, "noul": 100.0, "score": -13.43500000000002 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 34.89788499025341 }, "ecommerce_marketplaces": { "accuracy": 0.4117647058823529, "cc": 0.4748007246376762, "cc_by_type": { "choice": 42.02898550724638, "noul": -33.333333333333336, "score": -7.271250000000018 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 0.4748007246376762 }, "feature_extraction": { "accuracy": 0.15789473684210525, "cc": -116.71251572327047, "cc_by_type": { "choice": -13.207547169811319, "noul": -100.0, "score": -236.9300000000001 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5, "cc": 11.735613014436538, "cc_by_type": { "choice": 50.98039215686275, "noul": -42.85714285714286, "score": 27.08358974358973 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 11.735613014436538 }, "gaming": { "accuracy": 0.18181818181818182, "cc": -12.115642074433163, "cc_by_type": { "choice": -14.98257839721254, "noul": -50.0, "score": 28.63565217391304 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.4666666666666667, "cc": 26.11814814814815, "cc_by_type": { "choice": 16.666666666666668, "noul": 60.00000000000001, "score": 1.687777777777777 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 26.11814814814815 }, "knowledge_graphs": { "accuracy": 0.36363636363636365, "cc": -3.2785867237687394, "cc_by_type": { "choice": 10.064239828693788, "noul": 0.0, "score": -19.900000000000006 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.6, "cc": 31.765082217524082, "cc_by_type": { "choice": 76.74418604651163, "noul": 33.33333333333333, "score": -14.782272727272705 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 31.765082217524082 }, "legal_compliance": { "accuracy": 0.5, "cc": 9.02948476702508, "cc_by_type": { "choice": 35.483870967741936, "noul": 33.33333333333333, "score": -41.728750000000026 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 9.02948476702508 }, "llm_guardrails": { "accuracy": 0.2962962962962963, "cc": -19.13207400298573, "cc_by_type": { "choice": 30.9021113243762, "noul": -60.0, "score": -28.29833333333338 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 0.0 }, "model_routing": { "accuracy": 0.6728110599078341, "cc": 52.803651357323226, "cc_by_type": { "choice": 59.0909090909091, "noul": 60.606060606060595, "score": 38.713984374999995 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 52.803651357323226 }, "moderation_trust_safety": { "accuracy": 0.5, "cc": 16.1079067404313, "cc_by_type": { "choice": 42.41842610364683, "noul": 0.0, "score": 5.905294117647076 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 16.1079067404313 }, "recruiting": { "accuracy": 0.3333333333333333, "cc": -14.35318239173391, "cc_by_type": { "choice": 33.147632311977716, "noul": -33.333333333333336, "score": -42.87384615384611 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 0.0 }, "risk_assessment": { "accuracy": 0.2222222222222222, "cc": -30.597434560030916, "cc_by_type": { "choice": 11.330049261083742, "noul": -50.0, "score": -53.12235294117649 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.391304347826087, "cc": 11.78285507246376, "cc_by_type": { "choice": 27.536231884057973, "noul": 0.0, "score": 7.81233333333331 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 11.78285507246376 }, "search_retrieval": { "accuracy": 0.5416666666666666, "cc": 29.112966223387648, "cc_by_type": { "choice": 67.97153024911032, "noul": 0.0, "score": 19.367368421052632 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 29.112966223387648 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 56.40809109915794, 64.67415216355847 ], "I": [ 15.782504352448152, 26.824484705343906 ], "capability": [ 37.193702561392236, 45.08001649702074 ], "composite": { "A": [ 3.6455898800704327, 13.977339719064457 ], "B": [ 2.885102056071331, 12.023251156152215 ], "C": [ 2.5316596389378008, 9.706485916016982 ] }, "seed": 29 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 62.82741496733333, "capability": 42.413153726626945, "intelligence": 21.99889248592056 }, "per_type": { "choice": { "accuracy": 0.512, "calibration": 87.50088888888888, "cc": 36.96779502095333, "cc_by_type": { "choice": 36.96779502095333 }, "cc_tiered": 36.28813256780837, "cc_tiered_open": 36.29349492821419, "cc_tiered_sealed": 36.28277020740255, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 36.96779502095333 }, "noul": { "accuracy": 0.5946666666666667, "calibration": 52.805066666666676, "cc": 18.933333333333337, "cc_by_type": { "noul": 18.933333333333337 }, "cc_tiered": 18.923944051055866, "cc_tiered_open": 21.265010351966872, "cc_tiered_sealed": 16.58287775014486, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 18.933333333333337 }, "score": { "accuracy": 0.328, "calibration": 48.17628934644445, "cc": -6.174584837544961, "cc_by_type": { "score": -6.174584837544961 }, "cc_tiered": 10.78460083889745, "cc_tiered_open": 21.5692016777949, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "raw_on_A": { "calibration": 63.43795542912849, "capability": 42.556073742724294, "intelligence": 21.674192056320095 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 7.773486221374046, "Emphasis on Accuracy 60:20:20": 5.844057834080417, "Emphasis on Cost 20:20:60": 8.6856620739047, "Emphasis on Speed 20:60:20": 10.03200434666341, "Intelligence only": 4.258556788204026 } }, { "addendum": null, "api_flag": true, "api_subset": false, "author": "Liquid AI", "axes": { "calibration": 91.25367387086692, "cost": 62.43173802145793, "intelligence": 69.70041168706942, "speed": 88.99977436791615 }, "calibration": { "parts": { "choice": { "ece_hard": 0.038374286188809156, "ece_score": 92.32514276223817, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 92.32514276223817 }, "noul": { "abstention_rate": 0.14477211796246647, "brier": 0.0787106673820862, "ece": 0.060856253290503357, "n": 373, "score": 87.82874934189933 }, "score": { "interval90_coverage": 0.967741935483871, "interval90_width": 0.26276881720430106, "n": 372, "nrps": 0.05673524666969873, "score": 93.60712950846329, "top_ece": 0.03556108158051766 } }, "score": 91.25367387086692 }, "capability": 80.47704277896817, "composite_ci95": { "A": [ 74.01470401465426, 77.00710380044042 ], "B": [ 72.06197415888809, 76.27993771737268 ], "C": [ 74.01470401465426, 77.00710380044042 ] }, "cost": { "basis": "Liquid AI public price list, USD 0.042/M input, output free (fetched 16 Oct 2026): the models table of Liquid's public pricing page lists d1-ultra at USD 0.042 per 1M input tokens and USD 0.00 per 1M output tokens (saved copy sha256 89fb8ef2...); the same rate Liquid's API billed on the measurement's smoke call (48 input tokens -> USD 0.000002016). Counted as the manufacturer's standard launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops and SubQ Snap), through a scoped re-freeze of the frozen pricing module that adds this one key and nothing else, with a zero-difference proof for every other registry entry. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (638,436 input tokens over the 1,485 answered items, 0 output tokens), divided by the full 1,500-row basis the scorer uses; the 15 items refused with HTTP 422 count as zero tokens and as wrong answers. Not a bill and not a quote.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.017876208 }, "display": "d1-ultra (Liquid AI)", "endpoint_condition": "Vendor-hosted api.liquid.ai/decisions/v1/systemone API, model `d1-ultra`, through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `liquid` before the first request (Florian GO 15 Oct 2026). Native probabilities on all three question types. 15 of 1,500 items exceed the model's 32,768-token input limit (HTTP 422) and are scored as failures.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 75.72153851823367, "I_sealed": 63.679284855905166, "base": 69.70041168706942, "excess": 2.2027268919727945, "gap": 12.042253662328505, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 88.89079890008239, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 100.00000000000001, "hard": 81.12614029569046, "judge": 88.13447593935399, "standard": 100.0 } }, "open|noul": { "cc": 75.17046238785369, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 85.71428571428572, "hard": 82.6086956521739, "judge": 60.00000000000001, "standard": 77.77777777777777 } }, "open|score": { "cc": 63.10335426676494, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 58.60054737282905, "hard": 68.18905752206545, "judge": 75.94709495533684, "standard": 35.917740170274 } }, "sealed|choice": { "cc": 79.10284436684508, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 90.47013977128337, "hard": 72.93859141898925, "judge": 79.72575905974536, "standard": 84.81333052098715 } }, "sealed|noul": { "cc": 53.0885768888857, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 72.41379310344827, "hard": 43.636363636363626, "judge": 53.846153846153854, "standard": 61.19402985074627 } }, "sealed|score": { "cc": 58.84643331198474, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 81.7986845210918, "hard": 56.171536782426415, "judge": 58.372812971884905, "standard": 53.43053127669758 } } } }, "jevbench_score": 76.10486449058568, "key": "liquid-d1-ultra", "last_measured_on": "2026-10-15", "licence": "Proprietary hosted API (no weights release)", "listing": "ranked", "measured_in": "v1.6.21", "measurement_date_status": "measured", "model_pin": "hosted-api:api.liquid.ai/decisions/v1/systemone#d1-ultra", "noul_decisive": { "acc_among_decisive": 0.9373040752351097, "decisive": 319, "decisive_rate": 0.8552278820375335, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": 2, "ranked": true, "ranks": { "A": 2, "B": 2, "C": 2, "capability": 5 }, "repo": "https://www.liquid.ai", "scores": { "A": 76.10486449058568, "B": 74.73151686997831, "C": 76.10486449058568 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.2313630650169216, "p50_s_raw": 0.2313630650169216, "p95_s_adjusted": 0.5441624051978579, "p95_s_raw": 0.5441624051978579 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.9545454545454546, "cc": 80.02551491048317, "cc_by_type": { "choice": 91.66666666666667, "noul": 100.0, "score": 48.40987806478281 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 80.02551491048317 }, "calibration_gen": { "accuracy": 0.693069306930693, "cc": 53.5127160507667, "cc_by_type": { "choice": 55.461293743372224, "noul": 60.00000000000001, "score": 45.07685440892786 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 53.5127160507667 }, "composite_gen": { "accuracy": 0.9306930693069307, "cc": 89.03033227085523, "cc_by_type": { "choice": 93.4959349593496, "noul": 92.59259259259258, "score": 81.00246926062347 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 89.03033227085523 }, "fairness": { "accuracy": 0.9375, "cc": 85.90669247576356, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 97.72007742729068 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 85.90669247576356 }, "jaggedness": { "accuracy": 0.7111111111111111, "cc": 52.548678213136746, "cc_by_type": { "choice": 62.463907603464875, "noul": 35.13513513513513, "score": 60.04699190081021 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 52.548678213136746 }, "llm_router": { "accuracy": 0.9041533546325878, "cc": 80.3197048932887, "cc_by_type": { "choice": 91.28290921898389, "noul": 89.69072164948453, "score": 59.98548381139766 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 80.3197048932887 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.7407407407407407, "cc": 40.01561610820875, "cc_by_type": { "choice": 75.2577319587629, "noul": 0.0, "score": 44.789116365863336 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 40.01561610820875 }, "multilingual": { "accuracy": 0.7418181818181818, "cc": 55.53031034406329, "cc_by_type": { "choice": 80.43690903162701, "noul": 25.0, "score": 61.15402200056285 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 55.53031034406329 }, "negation_pair_gen": { "accuracy": 0.9302325581395349, "cc": 76.76565063912945, "cc_by_type": { "choice": 100.0, "noul": 80.0, "score": 50.296951917388334 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 76.76565063912945 }, "option_order": { "accuracy": 0.9375, "cc": 92.01065246338216, "cc_by_type": { "choice": 92.01065246338216 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 92.01065246338216 }, "perturb_pair": { "accuracy": 0.875, "cc": 80.13869805419485, "cc_by_type": { "choice": 87.98798798798799, "noul": 100.0, "score": 52.42810617459654 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 80.13869805419485 }, "usecase": { "accuracy": 0.7681818181818182, "cc": 61.12254914211481, "cc_by_type": { "choice": 80.39215686274511, "noul": 41.66666666666667, "score": 61.30882389693265 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 61.12254914211481 } }, "lang": { "ar": { "accuracy": 0.6875, "cc": 27.790054081448684, "cc_by_type": { "choice": 80.95238095238095, "noul": 0.0, "score": 2.417781291965093 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 27.790054081448684 }, "cs": { "accuracy": 0.42857142857142855, "cc": 57.89473684210526, "cc_by_type": { "choice": 15.789473684210522, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 57.89473684210526 }, "da": { "accuracy": 0.7692307692307693, "cc": 70.84904875775025, "cc_by_type": { "choice": 58.52534562211981, "noul": 100.0, "score": 54.021800651130945 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 70.84904875775025 }, "de": { "accuracy": 0.90625, "cc": 89.08391810246775, "cc_by_type": { "choice": 84.07079646017698, "noul": 100.0, "score": 83.18095784722625 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 89.08391810246775 }, "el": { "accuracy": 0.6, "cc": 44.35440994419011, "cc_by_type": { "choice": 66.4804469273743, "noul": -33.333333333333336, "score": 99.91611623852936 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 44.35440994419011 }, "en": { "accuracy": 0.8236316246741964, "cc": 69.98283524145941, "cc_by_type": { "choice": 80.88868820248715, "noul": 67.62589928057554, "score": 61.433918241315546 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 69.98283524145941 }, "es": { "accuracy": 0.7741935483870968, "cc": 66.08416705224533, "cc_by_type": { "choice": 70.63621533442088, "noul": 100.0, "score": 27.616285822315124 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 66.08416705224533 }, "fi": { "accuracy": 0.875, "cc": 76.59925311455045, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 96.46442601031806 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 76.59925311455045 }, "fr": { "accuracy": 0.9285714285714286, "cc": 96.12745113668352, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 88.38235341005056 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 96.12745113668352 }, "hi": { "accuracy": 0.8571428571428571, "cc": 69.2291292495233, "cc_by_type": { "choice": 99.99999999999999, "noul": 50.0, "score": 57.68738774856992 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 69.2291292495233 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8666666666666667, "cc": 62.7322379583571, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": -11.803286124928691 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 62.7322379583571 }, "ja": { "accuracy": 0.56, "cc": 24.203419893141085, "cc_by_type": { "choice": 85.70293884034949, "noul": 0.0, "score": -13.092679160926245 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 24.203419893141085 }, "ko": { "accuracy": 0.5, "cc": 18.422112493954483, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": 55.26633748186346 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 18.422112493954483 }, "mixed": { "accuracy": 1.0, "cc": 99.4730866883171, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 98.41926006495127 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 99.4730866883171 }, "nl": { "accuracy": 0.5, "cc": -5.121222995410733, "cc_by_type": { "choice": 74.02597402597404, "noul": -100.0, "score": 10.610356987793757 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.3333333333333333, "cc": -7.258064516129032, "cc_by_type": { "choice": 35.483870967741936, "noul": -50.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.8461538461538461, "cc": 66.50566804059815, "cc_by_type": { "choice": 92.382564536606, "noul": 39.99999999999999, "score": 67.13443958518846 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 66.50566804059815 }, "pt": { "accuracy": 0.8571428571428571, "cc": 85.83933080677154, "cc_by_type": { "choice": 89.39304655274012, "noul": 100.0, "score": 68.1249458675745 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 85.83933080677154 }, "sv": { "accuracy": 0.6363636363636364, "cc": 45.882356110583025, "cc_by_type": { "choice": 74.46808510638299, "noul": 19.999999999999996, "score": 43.17898322536608 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 45.882356110583025 }, "tr": { "accuracy": 0.7692307692307693, "cc": 47.56476228162157, "cc_by_type": { "choice": 78.26086956521739, "noul": 0.0, "score": 64.43341727964732 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 47.56476228162157 }, "uk": { "accuracy": 0.7142857142857143, "cc": 78.73277592080343, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 68.45639227853934 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 78.73277592080343 }, "zh": { "accuracy": 0.8, "cc": 46.87044056187991, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 7.277988352306409 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 46.87044056187991 } }, "type": { "choice": { "accuracy": 0.8546666666666667, "cc": 81.22811381771561, "cc_by_type": { "choice": 81.22811381771561 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 81.22811381771561 }, "noul": { "accuracy": 0.7973333333333333, "cc": 59.46666666666667, "cc_by_type": { "noul": 59.46666666666667 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 59.46666666666667 }, "score": { "accuracy": 0.7333333333333333, "cc": 60.44867908600179, "cc_by_type": { "score": 60.44867908600179 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 60.44867908600179 } }, "usecase": { "advertising": { "accuracy": 0.5909090909090909, "cc": 33.23171209119251, "cc_by_type": { "choice": 84.0, "noul": -25.0, "score": 40.695136273577546 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 33.23171209119251 }, "code_linting": { "accuracy": 0.9583333333333334, "cc": 94.09748592551544, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 82.29245777654631 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 94.09748592551544 }, "customer_support": { "accuracy": 0.7857142857142857, "cc": 62.92582030524064, "cc_by_type": { "choice": 51.21951219512195, "noul": 100.0, "score": 37.557948720599974 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 62.92582030524064 }, "demand_forecasting": { "accuracy": 0.5294117647058824, "cc": -17.965299307538217, "cc_by_type": { "choice": 64.91228070175438, "noul": -100.0, "score": -18.80817862436903 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5294117647058824, "cc": 24.49212514702329, "cc_by_type": { "choice": 42.02898550724638, "noul": -33.333333333333336, "score": 64.78072326715683 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 24.49212514702329 }, "feature_extraction": { "accuracy": 0.631578947368421, "cc": -13.88384228808446, "cc_by_type": { "choice": 49.685534591194966, "noul": 33.33333333333333, "score": -124.67039478878168 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.6666666666666666, "cc": 35.72504937468765, "cc_by_type": { "choice": 90.19607843137256, "noul": -42.85714285714286, "score": 59.83621254983327 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 35.72504937468765 }, "gaming": { "accuracy": 0.6818181818181818, "cc": 31.77084188905293, "cc_by_type": { "choice": 68.6411149825784, "noul": -50.0, "score": 76.6714106845804 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 31.77084188905293 }, "insurance_claims": { "accuracy": 0.6, "cc": 45.1815821381559, "cc_by_type": { "choice": 58.33333333333333, "noul": 19.999999999999996, "score": 57.211413081134374 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 45.1815821381559 }, "knowledge_graphs": { "accuracy": 0.8636363636363636, "cc": 76.48497795483895, "cc_by_type": { "choice": 100.0, "noul": 66.66666666666667, "score": 62.7882671978502 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 76.48497795483895 }, "lead_generation": { "accuracy": 0.8, "cc": 66.48644556865014, "cc_by_type": { "choice": 88.3720930232558, "noul": 77.77777777777777, "score": 33.309465904916834 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 66.48644556865014 }, "legal_compliance": { "accuracy": 0.3, "cc": -11.843346976830077, "cc_by_type": { "choice": -29.032258064516128, "noul": -33.333333333333336, "score": 26.835550467359237 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.7777777777777778, "cc": 55.32046425195082, "cc_by_type": { "choice": 100.00000000000001, "noul": 19.999999999999996, "score": 45.96139275585243 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 55.32046425195082 }, "model_routing": { "accuracy": 0.9078341013824884, "cc": 79.30093337735978, "cc_by_type": { "choice": 96.49350649350649, "noul": 93.93939393939394, "score": 47.46989969917894 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 79.30093337735978 }, "moderation_trust_safety": { "accuracy": 0.7222222222222222, "cc": 20.596744504671886, "cc_by_type": { "choice": 76.96737044145875, "noul": -100.0, "score": 84.82286307255691 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 20.596744504671886 }, "recruiting": { "accuracy": 0.7222222222222222, "cc": 54.36102452879326, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 63.17592409705759 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 54.36102452879326 }, "risk_assessment": { "accuracy": 0.6111111111111112, "cc": 17.81076588752592, "cc_by_type": { "choice": 85.22167487684729, "noul": -50.0, "score": 18.21062278573048 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 17.81076588752592 }, "scientific_discovery": { "accuracy": 0.6086956521739131, "cc": 24.11187241023282, "cc_by_type": { "choice": 85.5072463768116, "noul": -66.66666666666667, "score": 53.495037520553545 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 24.11187241023282 }, "search_retrieval": { "accuracy": 1.0, "cc": 99.89828165670299, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 99.69484497010896 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 99.89828165670299 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 86.4976368449114, 91.57459849935341 ], "I": [ 65.06454054747739, 73.56568570096134 ], "capability": [ 76.3881588873906, 81.97127395015538 ], "composite": { "A": [ 74.01470401465426, 77.00710380044042 ], "B": [ 72.06197415888809, 76.27993771737268 ], "C": [ 74.01470401465426, 77.00710380044042 ] }, "seed": 30 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 91.25367387086692, "capability": 80.47704277896817, "intelligence": 69.70041168706942 }, "per_type": { "choice": { "accuracy": 0.8546666666666667, "calibration": 92.32514276223817, "cc": 81.22811381771561, "cc_by_type": { "choice": 81.22811381771561 }, "cc_tiered": 83.99682163346372, "cc_tiered_open": 88.89079890008239, "cc_tiered_sealed": 79.10284436684508, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 81.22811381771561 }, "noul": { "accuracy": 0.7973333333333333, "calibration": 87.82874934189933, "cc": 59.46666666666667, "cc_by_type": { "noul": 59.46666666666667 }, "cc_tiered": 64.12951963836969, "cc_tiered_open": 75.17046238785369, "cc_tiered_sealed": 53.0885768888857, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 59.46666666666667 }, "score": { "accuracy": 0.7333333333333333, "calibration": 93.60712950846329, "cc": 60.44867908600179, "cc_by_type": { "score": 60.44867908600179 }, "cc_tiered": 60.97489378937484, "cc_tiered_open": 63.10335426676494, "cc_tiered_sealed": 58.84643331198474, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 60.44867908600179 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 72.11435270662213, "Emphasis on Accuracy 60:20:20": 71.12898591962833, "Emphasis on Cost 20:20:60": 67.90195384485378, "Emphasis on Speed 20:60:20": 78.03652322439497, "Intelligence only": 69.70041168706942 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Messier AI (@agentmessier-ai)", "axes": { "calibration": 72.86180997007926, "cost": 59.65780110751579, "intelligence": 52.02542301918905, "speed": 89.67127141858641 }, "calibration": { "parts": { "choice": { "ece_hard": 0.12514095842876055, "ece_score": 74.9718083142479, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 74.9718083142479 }, "noul": { "abstention_rate": 0.0, "brier": 0.17675601582379594, "ece": 0.15402190415708572, "n": 375, "score": 69.19561916858285 }, "score": { "interval90_coverage": 0.856, "interval90_width": 0.33844444444444444, "n": 375, "nrps": 0.11129842749409077, "score": 74.41800242740706, "top_ece": 0.2001707619788841 } }, "score": 72.86180997007926 }, "capability": 62.443616494634156, "composite_ci95": { "A": [ 54.389784565567226, 67.99837973664832 ], "B": [ 50.72573699938904, 65.42895179636466 ], "C": [ 37.77068372608836, 61.07046762601753 ] }, "cost": { "basis": "ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 dcfcecb0..., the module in force; its base-model references are unchanged since the dae86740... module this row was first priced with - the two later re-freezes each added one hosted-API launch-list key and nothing else): declared base Qwen/Qwen3.5-4B -> deepinfra:Qwen/Qwen3.5-4B USD 0.030000000000000002/M input, USD 0.15/M output, the same basis the live messier-one-v0.2 row carries. The author's server reports NO usage block of any kind, so the input tokens are counted OFFLINE with the author's own Prompter (prompt.py at tag v0.4) at the exact serving settings (--max-state 7500 --max-total 8150, AA labels, qboth layout) and cross-checked against vLLM's own prompt-token counters from the measured pass; output tokens are 0 (the engine is asked for one token restricted to the option labels and nothing is generated for the caller). The reference listing was deprecated after the 25 Sep 2026 cut-off and replaced by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3). Estimated, not charged, not a quote from the author or any host.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.022117700000000004 }, "display": "Messier One v0.4 (Qwen3.5-4B fine-tune, self-hosted)", "endpoint_condition": "Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX A6000 48 GB - different hardware from the H100/RTX 5090 rows of this draw and from the RTX 3090 the author quotes. Started by the author's OWN ./serve_vllm.sh MODEL_DIR 8080 with its defaults, which is what the author insists on for v0.4: vLLM 0.30.0, --max-model-len 8192, chunked prefill and prefix caching OFF (this revision is non-causal), --max-logprobs 256, processed_logprobs, front end --max-state 7500 --max-total 8150 --gate 0.5 --noul-floor 0.801; transformers 5.18.0 exactly as pinned. One request at a time from the unchanged runner. Isolation: the server ran inside a network namespace with loopback only (unshare -n), the cut proven before the sealed input was staged and re-proven immediately before the first sealed request, with the synthetic smoke gate passed in between; a network namespace replaces docker --network none because the pod carried no local image (custody #19335, same property, proven the same way). The pod host kept its network for ssh.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 55.631843562186205, "I_sealed": 48.419002476191885, "base": 52.02542301918905, "excess": -2.626685684361391, "gap": 7.21284108599432, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 72.70061629878168, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 92.21621621621622, "hard": 75.46398238439761, "judge": 64.40342781806197, "standard": 69.8618668899121 } }, "open|noul": { "cc": 59.21394064872326, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 85.71428571428572, "hard": 65.21739130434783, "judge": 30.000000000000004, "standard": 77.77777777777777 } }, "open|score": { "cc": 34.98097373905368, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 43.1256839171765, "hard": 43.01475509215395, "judge": 34.883418864196, "standard": 14.987388256078226 } }, "sealed|choice": { "cc": 60.22807849887394, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 73.3163913595934, "hard": 56.80582861107898, "judge": 54.162585700293846, "standard": 69.62666104197426 } }, "sealed|noul": { "cc": 51.03745559638509, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 68.96551724137932, "hard": 50.90909090909091, "judge": 38.46153846153846, "standard": 61.19402985074627 } }, "sealed|score": { "cc": 33.991473333316634, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 53.18191088973121, "hard": 30.414088756811818, "judge": 29.47815246036215, "standard": 38.32100501755069 } } } }, "jevbench_score": 65.72451272334371, "key": "messier-one-v0.4", "last_measured_on": "2026-10-14", "licence": "Weights CC BY-NC 4.0 (non-commercial: free for research and evaluation, commercial licence on request; listing a measured score is evaluation use); code Apache-2.0; base Qwen/Qwen3.5-4B Apache-2.0", "listing": "ranked", "measured_in": "v1.6.22", "measurement_date_status": "measured", "model_pin": "agentmessier/messier-one@14c84de57b71bf48608e2d886d86ebc5c6f8897e", "noul_decisive": { "acc_among_decisive": 0.7786666666666666, "decisive": 375, "decisive_rate": 1.0, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 9, "ranked": true, "ranks": { "A": 9, "B": 10, "C": 13, "capability": 21 }, "repo": "https://huggingface.co/agentmessier/messier-one", "scores": { "A": 65.72451272334371, "B": 62.43641983896754, "C": 49.41469447240214 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.2828623611479998, "p50_s_raw": 0.06643118057399988, "p95_s_adjusted": 0.3813271221704781, "p95_s_raw": 0.11566356108523905 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.8636363636363636, "cc": 82.16341854240056, "cc_by_type": { "choice": 75.0, "noul": 100.0, "score": 71.49025562720168 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 82.16341854240056 }, "calibration_gen": { "accuracy": 0.5148514851485149, "cc": 29.04494301665566, "cc_by_type": { "choice": 33.19194061505833, "noul": 43.99999999999999, "score": 9.942888434908658 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 29.04494301665566 }, "composite_gen": { "accuracy": 0.6435643564356436, "cc": 46.36869268140659, "cc_by_type": { "choice": 73.98373983739837, "noul": 55.55555555555556, "score": 9.566782651265848 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 46.36869268140659 }, "fairness": { "accuracy": 0.9375, "cc": 83.14562852488994, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 89.4368855746698 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 83.14562852488994 }, "jaggedness": { "accuracy": 0.5611111111111111, "cc": 33.94067092331888, "cc_by_type": { "choice": 49.4706448508181, "noul": 24.32432432432432, "score": 28.027043594814227 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 33.94067092331888 }, "llm_router": { "accuracy": 0.7795527156549521, "cc": 66.83382437622242, "cc_by_type": { "choice": 69.09395086730652, "noul": 75.25773195876289, "score": 56.14979030259788 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 66.83382437622242 }, "long_state": { "accuracy": 0.4, "cc": 5.043826168411282, "cc_by_type": { "choice": 9.677419354838706, "noul": 0.0, "score": 5.454059150395141 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 5.043826168411282 }, "ml_usecase": { "accuracy": 0.48148148148148145, "cc": 10.394403866583607, "cc_by_type": { "choice": 13.402061855670107, "noul": 25.0, "score": -7.218850255919285 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 10.394403866583607 }, "multilingual": { "accuracy": 0.6909090909090909, "cc": 53.25250361364891, "cc_by_type": { "choice": 66.7427453537659, "noul": 55.55555555555556, "score": 37.459209931625296 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 53.25250361364891 }, "negation_pair_gen": { "accuracy": 0.6395348837209303, "cc": 24.730034373509014, "cc_by_type": { "choice": 66.66666666666667, "noul": 26.66666666666666, "score": -19.143230212806284 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 24.730034373509014 }, "option_order": { "accuracy": 0.8541666666666666, "cc": 81.35818908122504, "cc_by_type": { "choice": 81.35818908122504 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 81.35818908122504 }, "perturb_pair": { "accuracy": 0.84375, "cc": 80.45660602129784, "cc_by_type": { "choice": 78.97897897897899, "noul": 100.0, "score": 62.39083908491454 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 80.45660602129784 }, "usecase": { "accuracy": 0.65, "cc": 52.043880329569305, "cc_by_type": { "choice": 62.00980392156864, "noul": 58.33333333333333, "score": 35.788503733805946 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 52.043880329569305 } }, "lang": { "ar": { "accuracy": 0.5, "cc": 17.790541563258085, "cc_by_type": { "choice": 4.76190476190476, "noul": 100.0, "score": -51.39028007213051 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 17.790541563258085 }, "cs": { "accuracy": 0.7142857142857143, "cc": 78.94736842105263, "cc_by_type": { "choice": 57.89473684210526, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 78.94736842105263 }, "da": { "accuracy": 0.8461538461538461, "cc": 59.56093069774181, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 45.34945875989212 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 59.56093069774181 }, "de": { "accuracy": 0.78125, "cc": 72.21420782877351, "cc_by_type": { "choice": 84.07079646017698, "noul": 75.0, "score": 57.57182702614354 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 72.21420782877351 }, "el": { "accuracy": 0.6666666666666666, "cc": 54.75552840589854, "cc_by_type": { "choice": 66.4804469273743, "noul": 0.0, "score": 97.78613829032132 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 54.75552840589854 }, "en": { "accuracy": 0.6828844483058211, "cc": 51.49058721733788, "cc_by_type": { "choice": 63.754408659889414, "noul": 54.67625899280575, "score": 36.04109399931846 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 51.49058721733788 }, "es": { "accuracy": 0.8387096774193549, "cc": 68.24190669801142, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 4.725720094034247 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 68.24190669801142 }, "fi": { "accuracy": 0.375, "cc": 33.934589289450244, "cc_by_type": { "choice": -31.14754098360656, "noul": 100.0, "score": 32.95130885195729 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 33.934589289450244 }, "fr": { "accuracy": 0.5714285714285714, "cc": 33.62990228437019, "cc_by_type": { "choice": 36.47058823529412, "noul": 0.0, "score": 64.41911861781645 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 33.62990228437019 }, "hi": { "accuracy": 0.6190476190476191, "cc": 32.11300719789304, "cc_by_type": { "choice": 66.4804469273743, "noul": 25.0, "score": 4.858574666304816 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 32.11300719789304 }, "id": { "accuracy": 0.25, "cc": -44.44444444444444, "cc_by_type": { "choice": 11.111111111111109, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.7333333333333333, "cc": 64.97069379895709, "cc_by_type": { "choice": 71.94076383476228, "noul": 100.0, "score": 22.971317562109007 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 64.97069379895709 }, "ja": { "accuracy": 0.6, "cc": 39.099214004618965, "cc_by_type": { "choice": 42.81175536139793, "noul": 60.00000000000001, "score": 14.485886652458968 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 39.099214004618965 }, "ko": { "accuracy": 0.6, "cc": 32.83013966262374, "cc_by_type": { "choice": 100.0, "noul": 0.0, "score": -1.509581012128769 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 32.83013966262374 }, "mixed": { "accuracy": 1.0, "cc": 99.18530071199275, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 97.5559021359783 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 99.18530071199275 }, "nl": { "accuracy": 0.6, "cc": 44.550106052514856, "cc_by_type": { "choice": 48.05194805194805, "noul": 33.33333333333333, "score": 52.265036772263194 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 44.550106052514856 }, "no": { "accuracy": 0.3333333333333333, "cc": -14.516129032258064, "cc_by_type": { "choice": -29.032258064516128, "noul": 0.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.7948717948717948, "cc": 69.82411392080512, "cc_by_type": { "choice": 84.76512907321202, "noul": 80.0, "score": 44.70721268920334 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 69.82411392080512 }, "pt": { "accuracy": 0.7142857142857143, "cc": 62.10287688730801, "cc_by_type": { "choice": 46.96523276370065, "noul": 100.0, "score": 39.34339789822337 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 62.10287688730801 }, "sv": { "accuracy": 0.45454545454545453, "cc": 26.944485862981143, "cc_by_type": { "choice": 23.404255319148945, "noul": 19.999999999999996, "score": 37.429202269794494 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 26.944485862981143 }, "tr": { "accuracy": 0.7692307692307693, "cc": 69.43378739054388, "cc_by_type": { "choice": 78.26086956521739, "noul": 100.0, "score": 30.040492606414237 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 69.43378739054388 }, "uk": { "accuracy": 0.7142857142857143, "cc": 55.79040730990699, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": -32.62877807027902 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 55.79040730990699 }, "zh": { "accuracy": 0.7, "cc": 35.76198975772707, "cc_by_type": { "choice": 74.57627118644068, "noul": 33.33333333333333, "score": -0.6236352465927908 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 35.76198975772707 } }, "type": { "choice": { "accuracy": 0.7213333333333334, "cc": 64.00619988901434, "cc_by_type": { "choice": 64.00619988901434 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 64.00619988901434 }, "noul": { "accuracy": 0.7786666666666666, "cc": 55.73333333333332, "cc_by_type": { "noul": 55.73333333333332 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 55.73333333333332 }, "score": { "accuracy": 0.52, "cc": 35.63329245619988, "cc_by_type": { "score": 35.63329245619988 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 35.63329245619988 } }, "usecase": { "advertising": { "accuracy": 0.5454545454545454, "cc": 31.602907151164914, "cc_by_type": { "choice": 36.0, "noul": 50.0, "score": 8.808721453494739 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 31.602907151164914 }, "code_linting": { "accuracy": 0.5416666666666666, "cc": 24.65440579911538, "cc_by_type": { "choice": 44.099378881987576, "noul": 55.55555555555556, "score": -25.69171704019699 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 24.65440579911538 }, "customer_support": { "accuracy": 0.6428571428571429, "cc": 58.85440387413508, "cc_by_type": { "choice": 18.699186991869915, "noul": 100.0, "score": 57.86402463053535 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 58.85440387413508 }, "demand_forecasting": { "accuracy": 0.35294117647058826, "cc": 21.573024553922433, "cc_by_type": { "choice": 6.432748538011691, "noul": 33.33333333333333, "score": 24.952991790422274 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 21.573024553922433 }, "ecommerce_marketplaces": { "accuracy": 0.47058823529411764, "cc": 28.246437211964235, "cc_by_type": { "choice": 27.53623188405797, "noul": 33.33333333333333, "score": 23.869746418501414 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 28.246437211964235 }, "feature_extraction": { "accuracy": 0.5789473684210527, "cc": 0.38882349872808203, "cc_by_type": { "choice": 37.106918238993714, "noul": 33.33333333333333, "score": -69.2737810761428 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.38882349872808203 }, "financial_crime": { "accuracy": 0.5333333333333333, "cc": 42.964161901689614, "cc_by_type": { "choice": 60.784313725490186, "noul": 71.42857142857142, "score": -3.320399448992739 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 42.964161901689614 }, "gaming": { "accuracy": 0.5, "cc": 14.800234981182662, "cc_by_type": { "choice": 47.73519163763067, "noul": -50.0, "score": 46.665513305917315 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 14.800234981182662 }, "insurance_claims": { "accuracy": 0.5333333333333333, "cc": 34.22439466186014, "cc_by_type": { "choice": 58.33333333333333, "noul": -19.999999999999996, "score": 64.33985065224708 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 34.22439466186014 }, "knowledge_graphs": { "accuracy": 0.5, "cc": 35.392748862215676, "cc_by_type": { "choice": 35.76017130620985, "noul": 66.66666666666667, "score": 3.7514086137705016 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 35.392748862215676 }, "lead_generation": { "accuracy": 0.68, "cc": 49.47199754097719, "cc_by_type": { "choice": 76.74418604651163, "noul": 33.33333333333333, "score": 38.338473243086646 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 49.47199754097719 }, "legal_compliance": { "accuracy": 0.5, "cc": 30.394104568777696, "cc_by_type": { "choice": 35.483870967741936, "noul": 33.33333333333333, "score": 22.365109405257822 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 30.394104568777696 }, "llm_guardrails": { "accuracy": 0.7407407407407407, "cc": 56.36973577985058, "cc_by_type": { "choice": 88.48368522072937, "noul": 19.999999999999996, "score": 60.62552211882236 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 56.36973577985058 }, "model_routing": { "accuracy": 0.7972350230414746, "cc": 66.80548991484399, "cc_by_type": { "choice": 78.96103896103895, "noul": 78.78787878787878, "score": 42.66755199561424 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 66.80548991484399 }, "moderation_trust_safety": { "accuracy": 0.7777777777777778, "cc": 48.12446743220059, "cc_by_type": { "choice": 88.48368522072937, "noul": 0.0, "score": 55.889717075872404 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 48.12446743220059 }, "recruiting": { "accuracy": 0.7777777777777778, "cc": 80.53804875918388, "cc_by_type": { "choice": 83.28690807799443, "noul": 100.0, "score": 58.32723819955723 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 80.53804875918388 }, "risk_assessment": { "accuracy": 0.5555555555555556, "cc": 40.26737295094906, "cc_by_type": { "choice": 70.44334975369459, "noul": 50.0, "score": 0.35876909915260047 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 40.26737295094906 }, "scientific_discovery": { "accuracy": 0.6521739130434783, "cc": 41.32801314674797, "cc_by_type": { "choice": 71.01449275362319, "noul": 33.33333333333333, "score": 19.636213353287392 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 41.32801314674797 }, "search_retrieval": { "accuracy": 0.8333333333333334, "cc": 79.25826168668284, "cc_by_type": { "choice": 78.64768683274022, "noul": 100.0, "score": 59.12709822730834 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 79.25826168668284 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 68.3226236790199, 76.55173943269169 ], "I": [ 46.46244328126147, 56.929062865618285 ], "capability": [ 57.84340019966032, 66.10769142886402 ], "composite": { "A": [ 54.389784565567226, 67.99837973664832 ], "B": [ 50.72573699938904, 65.42895179636466 ], "C": [ 37.77068372608836, 61.07046762601753 ] }, "seed": 31 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 72.86180997007926, "capability": 62.443616494634156, "intelligence": 52.02542301918905 }, "per_type": { "choice": { "accuracy": 0.7213333333333334, "calibration": 74.9718083142479, "cc": 64.00619988901434, "cc_by_type": { "choice": 64.00619988901434 }, "cc_tiered": 66.46434739882781, "cc_tiered_open": 72.70061629878168, "cc_tiered_sealed": 60.22807849887394, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 64.00619988901434 }, "noul": { "accuracy": 0.7786666666666666, "calibration": 69.19561916858285, "cc": 55.73333333333332, "cc_by_type": { "noul": 55.73333333333332 }, "cc_tiered": 55.12569812255417, "cc_tiered_open": 59.21394064872326, "cc_tiered_sealed": 51.03745559638509, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 55.73333333333332 }, "score": { "accuracy": 0.52, "calibration": 74.41800242740706, "cc": 35.63329245619988, "cc_by_type": { "score": 35.63329245619988 }, "cc_tiered": 34.48622353618516, "cc_tiered_open": 34.98097373905368, "cc_tiered_sealed": 33.991473333316634, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 35.63329245619988 } }, "raw_on_A": { "calibration": 76.02710478723333, "capability": 64.44192558101349, "intelligence": 52.85674637479366 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 63.64631975796662, "Emphasis on Accuracy 60:20:20": 58.426076553206535, "Emphasis on Cost 20:20:60": 61.98858107407938, "Emphasis on Speed 20:60:20": 72.00546899368237, "Intelligence only": 52.02542301918905 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "MeTask (Hangzhou MeTask Technology Co., Ltd.)", "axes": { "calibration": 87.91805023499681, "cost": 51.488678556745086, "intelligence": 67.8255189476327, "speed": 73.37579625429389 }, "calibration": { "parts": { "choice": { "ece_hard": 0.04579106842955546, "ece_score": 90.8417863140889, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 90.8417863140889 }, "noul": { "abstention_rate": 0.0, "brier": 0.12338284879474153, "ece": 0.05903423423423422, "n": 375, "score": 88.19315315315316 }, "score": { "interval90_coverage": 0.9226666666666666, "interval90_width": 0.28755555555555545, "n": 375, "nrps": 0.07999688319975698, "score": 84.71921123774837, "top_ece": 0.11280944602263793 } }, "score": 87.91805023499681 }, "capability": 77.87178459131476, "composite_ci95": { "A": [ 65.69957159141377, 68.51251191773383 ], "B": [ 65.03695245737931, 69.08121246803393 ], "C": [ 65.69957159141377, 68.51251191773383 ] }, "cost": { "basis": "v1.5-M2 base-model reference: base google/gemma-4-12B-it (author int4 AWQ of google/gemma-4-12B-it-qat-q4_0-unquantized + own LoRA). google/gemma-4-12b-it is not listed on OpenRouter; nearest listed sibling of the same family and size class (fixed Gemma-4-12B precedent) OpenRouter google/gemma-3-12b-it USD 0.05/M input, USD 0.15/M output, read 2026-10-12T00:13:38Z from https://openrouter.ai/api/v1/models x the service's own measured usage tokens; estimated, not charged.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.041404166666666666 }, "display": "metask-jev-rain-12B-trained", "endpoint_condition": "evaluator-owned Lium GPU pod (H100 80GB), offline read-only container (--network none); author's own service (serve.sh -> launch.py -> server.py on vLLM 0.31.0, pinned digest image), unchanged defaults Paid fast-lane order 66c7b6d2 (public). Own LoRA adapter (rank 16) on the author's int4 Gemma 4 12B base, applied unmerged by vLLM. Replaces nothing: the earlier metask_jev_rain_12b row (third-party AWQ wrapper, order f1b81eec) is a different system.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 73.28803588458365, "I_sealed": 62.36300201068175, "base": 67.8255189476327, "excess": 1.0855071035461918, "gap": 10.925033873901903, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 80.46364135488923, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 92.21621621621622, "hard": 79.2387543252595, "judge": 80.22412656558998, "standard": 77.39640016743407 } }, "open|noul": { "cc": 79.59903381642512, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 82.6086956521739, "judge": 70.0, "standard": 77.77777777777777 } }, "open|score": { "cc": 59.80143248243659, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 76.59298352366106, "hard": 57.8351755760549, "judge": 64.35617443717419, "standard": 48.50605784248132 } }, "sealed|choice": { "cc": 69.0735944674903, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 84.75222363405337, "hard": 65.65282757025557, "judge": 62.97747306562194, "standard": 77.2199957814807 } }, "sealed|noul": { "cc": 66.55391237686658, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 82.75862068965519, "hard": 52.727272727272734, "judge": 69.23076923076923, "standard": 82.08955223880596 } }, "sealed|score": { "cc": 51.46149918768837, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 62.57017720623247, "hard": 51.50741442798628, "judge": 47.129277578702755, "standard": 52.31366211129891 } } } }, "jevbench_score": 67.60392832716802, "key": "metask_jev_rain_12b_trained", "last_measured_on": "2026-10-12", "licence": "Weights: Apache-2.0 per model card (base Gemma 4 under the Gemma 4 licence); service code MIT (github.com/chenyuwang166-oss/metask-jev-rain-12B-trained@e36c7a54)", "listing": "ranked", "measured_in": "v1.6.10", "measurement_date_status": "measured", "model_pin": "MeTask-Rain/metask-jev-rain-12B-trained@cf3706fdd57e23025a8c97fbde86bd5b4a547010", "noul_decisive": { "acc_among_decisive": 0.856, "decisive": 375, "decisive_rate": 1.0, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 7, "ranked": true, "ranks": { "A": 7, "B": 5, "C": 6, "capability": 11 }, "repo": "https://huggingface.co/MeTask-Rain/metask-jev-rain-12B-trained", "scores": { "A": 67.60392832716802, "B": 67.64813054324519, "C": 67.60392832716802 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.3790490360464901, "p50_s_raw": 0.11452451802324504, "p95_s_adjusted": 12.126207013113888, "p95_s_raw": 5.988103506556944 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 1.0, "cc": 87.84887353749237, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 63.546620612477135 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 87.84887353749237 }, "calibration_gen": { "accuracy": 0.6435643564356436, "cc": 56.940570475966105, "cc_by_type": { "choice": 45.91728525980912, "noul": 76.0, "score": 48.9044261680892 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 56.940570475966105 }, "composite_gen": { "accuracy": 0.7821782178217822, "cc": 64.2194805783471, "cc_by_type": { "choice": 64.22764227642277, "noul": 55.55555555555556, "score": 72.875243903063 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 64.2194805783471 }, "fairness": { "accuracy": 0.9375, "cc": 89.4731058334587, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 68.41931750037614 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 89.4731058334587 }, "jaggedness": { "accuracy": 0.6555555555555556, "cc": 51.15173057621822, "cc_by_type": { "choice": 53.801732435033685, "noul": 56.75675675675676, "score": 42.8967025368642 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 51.15173057621822 }, "llm_router": { "accuracy": 0.8434504792332268, "cc": 72.24878717449957, "cc_by_type": { "choice": 80.18843004314519, "noul": 87.62886597938144, "score": 48.92906550097207 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 72.24878717449957 }, "long_state": { "accuracy": 0.26666666666666666, "cc": -31.382028062735532, "cc_by_type": { "choice": 9.677419354838706, "noul": -100.0, "score": -3.8235035430453035 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.5555555555555556, "cc": 31.665606480992682, "cc_by_type": { "choice": 38.14432989690721, "noul": 50.0, "score": 6.85248954607085 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 31.665606480992682 }, "multilingual": { "accuracy": 0.8036363636363636, "cc": 70.94558775850952, "cc_by_type": { "choice": 75.54613628953375, "noul": 77.77777777777777, "score": 59.51284920821705 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 70.94558775850952 }, "negation_pair_gen": { "accuracy": 0.7790697674418605, "cc": 55.42410495913857, "cc_by_type": { "choice": 83.33333333333333, "noul": 39.99999999999999, "score": 42.9389815440824 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 55.42410495913857 }, "option_order": { "accuracy": 0.9583333333333334, "cc": 94.67376830892145, "cc_by_type": { "choice": 94.67376830892145 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 94.67376830892145 }, "perturb_pair": { "accuracy": 0.859375, "cc": 80.61760708445547, "cc_by_type": { "choice": 78.97897897897899, "noul": 100.0, "score": 62.873842274387414 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 80.61760708445547 }, "usecase": { "accuracy": 0.740909090909091, "cc": 61.56342140022497, "cc_by_type": { "choice": 69.36274509803921, "noul": 58.33333333333333, "score": 56.994185769302376 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 61.56342140022497 } }, "lang": { "ar": { "accuracy": 0.5, "cc": 24.545761058228774, "cc_by_type": { "choice": 42.857142857142854, "noul": 50.0, "score": -19.21985968245654 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 24.545761058228774 }, "cs": { "accuracy": 0.7142857142857143, "cc": 78.94736842105263, "cc_by_type": { "choice": 57.89473684210526, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 78.94736842105263 }, "da": { "accuracy": 0.7692307692307693, "cc": 79.27519011200793, "cc_by_type": { "choice": 79.26267281105991, "noul": 100.0, "score": 58.56289752496387 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 79.27519011200793 }, "de": { "accuracy": 0.90625, "cc": 85.48969822591152, "cc_by_type": { "choice": 84.07079646017698, "noul": 100.0, "score": 72.39829821755762 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 85.48969822591152 }, "el": { "accuracy": 0.8, "cc": 76.94633377862073, "cc_by_type": { "choice": 66.4804469273743, "noul": 66.66666666666667, "score": 97.6918877418212 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 76.94633377862073 }, "en": { "accuracy": 0.7741094700260643, "cc": 65.19896688108702, "cc_by_type": { "choice": 71.88220793009603, "noul": 68.34532374100719, "score": 55.36936897215781 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 65.19896688108702 }, "es": { "accuracy": 0.8387096774193549, "cc": 73.5487504069601, "cc_by_type": { "choice": 90.2120717781403, "noul": 100.0, "score": 30.434179442740017 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 73.5487504069601 }, "fi": { "accuracy": 0.625, "cc": 65.70550033996419, "cc_by_type": { "choice": 1.6393442622950836, "noul": 100.0, "score": 95.47715675759747 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 65.70550033996419 }, "fr": { "accuracy": 0.8571428571428571, "cc": 83.74288210729817, "cc_by_type": { "choice": 78.82352941176471, "noul": 100.0, "score": 72.40511691012979 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 83.74288210729817 }, "hi": { "accuracy": 0.8095238095238095, "cc": 59.09382926533001, "cc_by_type": { "choice": 83.24022346368716, "noul": 100.0, "score": -5.958735667697135 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 59.09382926533001 }, "id": { "accuracy": 0.75, "cc": 77.77777777777777, "cc_by_type": { "choice": 55.55555555555555, "noul": 100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 77.77777777777777 }, "it": { "accuracy": 0.7333333333333333, "cc": 48.56239452957701, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": -40.28319832865008 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 48.56239452957701 }, "ja": { "accuracy": 0.64, "cc": 45.094346455841844, "cc_by_type": { "choice": 42.81175536139793, "noul": 39.99999999999999, "score": 52.471284006127604 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 45.094346455841844 }, "ko": { "accuracy": 0.5, "cc": 34.600352011262636, "cc_by_type": { "choice": 55.55555555555555, "noul": 0.0, "score": 48.24550047823236 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 34.600352011262636 }, "mixed": { "accuracy": 1.0, "cc": 98.74153284520177, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 96.22459853560528 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 98.74153284520177 }, "nl": { "accuracy": 0.8, "cc": 48.51424476202488, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 12.209400952741323 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 48.51424476202488 }, "no": { "accuracy": 0.8333333333333334, "cc": 67.74193548387098, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 67.74193548387098 }, "pl": { "accuracy": 0.8205128205128205, "cc": 71.6454228057721, "cc_by_type": { "choice": 84.76512907321202, "noul": 80.0, "score": 50.17113934410426 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 71.6454228057721 }, "pt": { "accuracy": 0.8571428571428571, "cc": 81.35337503204035, "cc_by_type": { "choice": 78.78609310548026, "noul": 100.0, "score": 65.27403199064081 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 81.35337503204035 }, "sv": { "accuracy": 0.45454545454545453, "cc": 37.10841544635961, "cc_by_type": { "choice": -27.659574468085104, "noul": 60.00000000000001, "score": 78.98482080716391 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 37.10841544635961 }, "tr": { "accuracy": 0.8461538461538461, "cc": 54.20537342574059, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": -37.38387972277823 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 54.20537342574059 }, "uk": { "accuracy": 1.0, "cc": 96.22347206389003, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 88.67041619167007 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 96.22347206389003 }, "zh": { "accuracy": 0.8, "cc": 37.32160001214934, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": -21.368533296885307 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 37.32160001214934 } }, "type": { "choice": { "accuracy": 0.7853333333333333, "cc": 72.27271857479094, "cc_by_type": { "choice": 72.27271857479094 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 72.27271857479094 }, "noul": { "accuracy": 0.856, "cc": 71.2, "cc_by_type": { "noul": 71.2 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 71.2 }, "score": { "accuracy": 0.6773333333333333, "cc": 54.022025771730455, "cc_by_type": { "score": 54.022025771730455 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.022025771730455 } }, "usecase": { "advertising": { "accuracy": 0.6363636363636364, "cc": 39.183979710642106, "cc_by_type": { "choice": 68.0, "noul": 25.0, "score": 24.551939131926314 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 39.183979710642106 }, "code_linting": { "accuracy": 0.7083333333333334, "cc": 51.97732888297762, "cc_by_type": { "choice": 81.36645962732919, "noul": 77.77777777777777, "score": -3.2122507561741065 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 51.97732888297762 }, "customer_support": { "accuracy": 0.7857142857142857, "cc": 42.34789039430175, "cc_by_type": { "choice": 83.73983739837398, "noul": 50.0, "score": -6.696166215468713 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 42.34789039430175 }, "demand_forecasting": { "accuracy": 0.6470588235294118, "cc": 43.93438993611236, "cc_by_type": { "choice": 53.21637426900585, "noul": 33.33333333333333, "score": 45.253462205997906 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 43.93438993611236 }, "ecommerce_marketplaces": { "accuracy": 0.6470588235294118, "cc": 30.104630020806553, "cc_by_type": { "choice": 56.52173913043478, "noul": -33.333333333333336, "score": 67.12548426531822 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 30.104630020806553 }, "feature_extraction": { "accuracy": 0.5789473684210527, "cc": -40.6229056009301, "cc_by_type": { "choice": 49.685534591194966, "noul": 33.33333333333333, "score": -204.8875847273186 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.7, "cc": 60.81399584070914, "cc_by_type": { "choice": 80.3921568627451, "noul": 42.85714285714286, "score": 59.192687802239476 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 60.81399584070914 }, "gaming": { "accuracy": 0.7727272727272727, "cc": 51.12928904620589, "cc_by_type": { "choice": 89.54703832752612, "noul": 0.0, "score": 63.840828811091555 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 51.12928904620589 }, "insurance_claims": { "accuracy": 0.6666666666666666, "cc": 65.41586316119422, "cc_by_type": { "choice": 16.666666666666668, "noul": 100.0, "score": 79.580922816916 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 65.41586316119422 }, "knowledge_graphs": { "accuracy": 0.6818181818181818, "cc": 53.44925983148553, "cc_by_type": { "choice": 35.76017130620985, "noul": 66.66666666666667, "score": 57.92094152158005 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 53.44925983148553 }, "lead_generation": { "accuracy": 0.84, "cc": 72.23152413124878, "cc_by_type": { "choice": 88.3720930232558, "noul": 100.0, "score": 28.322479370490505 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 72.23152413124878 }, "legal_compliance": { "accuracy": 0.5, "cc": 14.752066065373555, "cc_by_type": { "choice": 35.483870967741936, "noul": 33.33333333333333, "score": -24.561006104954597 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 14.752066065373555 }, "llm_guardrails": { "accuracy": 0.8518518518518519, "cc": 72.19021405060475, "cc_by_type": { "choice": 88.48368522072937, "noul": 60.00000000000001, "score": 68.08695693108486 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 72.19021405060475 }, "model_routing": { "accuracy": 0.8755760368663594, "cc": 74.29999007860982, "cc_by_type": { "choice": 90.64935064935067, "noul": 96.96969696969697, "score": 35.28092261678184 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 74.29999007860982 }, "moderation_trust_safety": { "accuracy": 0.7222222222222222, "cc": 73.75106865956981, "cc_by_type": { "choice": 76.96737044145875, "noul": 100.0, "score": 44.28583553725066 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 73.75106865956981 }, "recruiting": { "accuracy": 0.7777777777777778, "cc": 76.15530393930493, "cc_by_type": { "choice": 49.860724233983284, "noul": 100.0, "score": 78.6051875839315 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 76.15530393930493 }, "risk_assessment": { "accuracy": 0.7222222222222222, "cc": 68.0715834514785, "cc_by_type": { "choice": 55.665024630541865, "noul": 100.0, "score": 48.54972572389362 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 68.0715834514785 }, "scientific_discovery": { "accuracy": 0.4782608695652174, "cc": 30.843083819184248, "cc_by_type": { "choice": -1.4492753623188352, "noul": 33.33333333333333, "score": 60.645193486538254 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 30.843083819184248 }, "search_retrieval": { "accuracy": 0.9583333333333334, "cc": 94.96186264566033, "cc_by_type": { "choice": 89.3238434163701, "noul": 100.0, "score": 95.56174452061086 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 94.96186264566033 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 82.74284328530476, 89.55881014038087 ], "I": [ 62.61412667978783, 71.59774693351402 ], "capability": [ 73.35540902322805, 79.83976612353158 ], "composite": { "A": [ 65.69957159141377, 68.51251191773383 ], "B": [ 65.03695245737931, 69.08121246803393 ], "C": [ 65.69957159141377, 68.51251191773383 ] }, "seed": 32 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 87.91805023499681, "capability": 77.87178459131476, "intelligence": 67.8255189476327 }, "per_type": { "choice": { "accuracy": 0.7853333333333333, "calibration": 90.8417863140889, "cc": 72.27271857479094, "cc_by_type": { "choice": 72.27271857479094 }, "cc_tiered": 74.76861791118976, "cc_tiered_open": 80.46364135488923, "cc_tiered_sealed": 69.0735944674903, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 72.27271857479094 }, "noul": { "accuracy": 0.856, "calibration": 88.19315315315316, "cc": 71.2, "cc_by_type": { "noul": 71.2 }, "cc_tiered": 73.07647309664586, "cc_tiered_open": 79.59903381642512, "cc_tiered_sealed": 66.55391237686658, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 71.2 }, "score": { "accuracy": 0.6773333333333333, "calibration": 84.71921123774837, "cc": 54.022025771730455, "cc_by_type": { "score": 54.022025771730455 }, "cc_tiered": 55.63146583506248, "cc_tiered_open": 59.80143248243659, "cc_tiered_sealed": 51.46149918768837, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.022025771730455 } }, "raw_on_A": { "calibration": 86.30134459903176, "capability": 77.02938857798543, "intelligence": 67.75743255693911 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 62.76947628204991, "Emphasis on Accuracy 60:20:20": 64.69865676810258, "Emphasis on Cost 20:20:60": 57.71178234790809, "Emphasis on Speed 20:60:20": 66.62147498913741, "Intelligence only": 67.8255189476327 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "MeTask (Hangzhou MeTask Technology Co., Ltd.)", "axes": { "calibration": 87.09471194592824, "cost": 49.238192537895735, "intelligence": 70.71905450578407, "speed": 83.17884674032656 }, "calibration": { "parts": { "choice": { "ece_hard": 0.057130297017528, "ece_score": 88.5739405964944, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 88.5739405964944 }, "noul": { "abstention_rate": 0.0, "brier": 0.1123272461650836, "ece": 0.06861261261261234, "n": 375, "score": 86.27747747747753 }, "score": { "interval90_coverage": 0.9226666666666666, "interval90_width": 0.2508888888888888, "n": 375, "nrps": 0.07405460121549541, "score": 86.43271776381279, "top_ece": 0.09864552175412443 } }, "score": 87.09471194592824 }, "capability": 78.90688322585615, "composite_ci95": { "A": [ 65.30821255607486, 67.94674409767933 ], "B": [ 64.96105873997377, 68.76565711110239 ], "C": [ 65.30821255607486, 67.94674409767933 ] }, "cost": { "basis": "v1.5-M2 base-model reference: exact base google/gemma-4-31B-it (served as Google's unchanged google/gemma-4-31B-it-qat-w4a16-ct@52f3f65b + own LoRA). OpenRouter google/gemma-4-31b-it list price USD 0.09/M input, USD 0.34/M output (listed since Apr 2026; same reference as the published Jeff, deck31b and Quyet-1.0-Large rows), read 2026-10-12T00:13:38Z from https://openrouter.ai/api/v1/models x the service's own measured usage tokens; estimated, not charged.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.04921078 }, "display": "metask-jev-rain-31B-trained", "endpoint_condition": "evaluator-owned Lium GPU pod (H100 80GB), offline read-only container (--network none); author's own service (service/serve.sh -> launch.py -> server.py on vLLM 0.31.0, pinned digest image), unchanged defaults Paid fast-lane order 147c5e18 (public). Own LoRA adapter (rank 16) on Google's official W4A16 Gemma 4 31B QAT checkpoint, applied unmerged by vLLM; max_model_len 6656, longer inputs are shortened by the service's documented extraction rule (author-tested only on H200; measured on H100 80GB with unchanged settings).", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 72.5520469583762, "I_sealed": 68.88606205319194, "base": 70.71905450578407, "excess": -6.173541865171451, "gap": 3.66598490518426, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 80.81042171537538, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 100.00000000000001, "hard": 81.12614029569046, "judge": 76.26895187870798, "standard": 77.39640016743407 } }, "open|noul": { "cc": 73.59903381642512, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 82.6086956521739, "judge": 50.0, "standard": 77.77777777777777 } }, "open|score": { "cc": 63.24668534332811, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 70.41765078820325, "hard": 69.57893658072337, "judge": 60.72068725487829, "standard": 50.785697278774734 } }, "sealed|choice": { "cc": 80.73173886349481, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 94.28208386277001, "hard": 78.142708453799, "judge": 78.84427032321254, "standard": 81.96582999367222 } }, "sealed|noul": { "cc": 74.61818469744357, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 86.20689655172413, "hard": 67.27272727272727, "judge": 81.53846153846153, "standard": 73.13432835820895 } }, "sealed|score": { "cc": 51.30826259863744, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 70.15351391347656, "hard": 51.34670276898963, "judge": 46.83933293506929, "standard": 48.51215109586572 } } } }, "jevbench_score": 66.93314450921662, "key": "metask_jev_rain_31b_trained", "last_measured_on": "2026-10-12", "licence": "Weights: Apache-2.0 per model card (base Gemma 4 under the Gemma 4 licence); service code (service/) MIT", "listing": "ranked", "measured_in": "v1.6.10", "measurement_date_status": "measured", "model_pin": "MeTask-Rain/metask-jev-rain-31B-trained@da56acc0bb607cc214c184ad9f1e9231a0d63dee", "noul_decisive": { "acc_among_decisive": 0.8773333333333333, "decisive": 375, "decisive_rate": 1.0, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 8, "ranked": true, "ranks": { "A": 8, "B": 7, "C": 7, "capability": 10 }, "repo": "https://huggingface.co/MeTask-Rain/metask-jev-rain-31B-trained", "scores": { "A": 66.93314450921662, "B": 67.25625355613948, "C": 66.93314450921662 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.5687712740385905, "p50_s_raw": 0.20938563701929525, "p95_s_adjusted": 0.8456247233552858, "p95_s_raw": 0.3478123616776429 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.8181818181818182, "cc": 63.16886779201999, "cc_by_type": { "choice": 75.0, "noul": 100.0, "score": 14.506603376059957 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 63.16886779201999 }, "calibration_gen": { "accuracy": 0.7722772277227723, "cc": 64.53799979415957, "cc_by_type": { "choice": 77.7306468716861, "noul": 60.00000000000001, "score": 55.88335251079264 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 64.53799979415957 }, "composite_gen": { "accuracy": 0.8811881188118812, "cc": 83.81438823065723, "cc_by_type": { "choice": 96.7479674796748, "noul": 92.59259259259258, "score": 62.102604619704316 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 83.81438823065723 }, "fairness": { "accuracy": 0.9375, "cc": 86.04066368360208, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 98.12199105080622 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 86.04066368360208 }, "jaggedness": { "accuracy": 0.6666666666666666, "cc": 49.85525524192105, "cc_by_type": { "choice": 62.463907603464875, "noul": 51.35135135135136, "score": 35.750506770946934 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 49.85525524192105 }, "llm_router": { "accuracy": 0.9297124600638977, "cc": 83.7793054809374, "cc_by_type": { "choice": 93.66029761380645, "noul": 95.8762886597938, "score": 61.801330169211944 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 83.7793054809374 }, "long_state": { "accuracy": 0.3333333333333333, "cc": 9.832941925502283, "cc_by_type": { "choice": 9.677419354838706, "noul": 0.0, "score": 19.821406421668144 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 9.832941925502283 }, "ml_usecase": { "accuracy": 0.7407407407407407, "cc": 54.59598157228413, "cc_by_type": { "choice": 62.886597938144334, "noul": 75.0, "score": 25.901346778708078 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 54.59598157228413 }, "multilingual": { "accuracy": 0.7963636363636364, "cc": 68.20002344704231, "cc_by_type": { "choice": 75.54613628953375, "noul": 69.44444444444444, "score": 59.60948960714872 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 68.20002344704231 }, "negation_pair_gen": { "accuracy": 0.872093023255814, "cc": 68.5323347023584, "cc_by_type": { "choice": 93.33333333333333, "noul": 60.00000000000001, "score": 52.263670773741836 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 68.5323347023584 }, "option_order": { "accuracy": 0.9791666666666666, "cc": 97.3368841544607, "cc_by_type": { "choice": 97.3368841544607 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 97.3368841544607 }, "perturb_pair": { "accuracy": 0.875, "cc": 81.60522267243225, "cc_by_type": { "choice": 84.98498498498498, "noul": 100.0, "score": 59.83068303231177 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 81.60522267243225 }, "usecase": { "accuracy": 0.7363636363636363, "cc": 64.54982362341306, "cc_by_type": { "choice": 76.7156862745098, "noul": 70.83333333333333, "score": 46.100451262396035 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 64.54982362341306 } }, "lang": { "ar": { "accuracy": 0.6875, "cc": 36.07646009918423, "cc_by_type": { "choice": 61.904761904761905, "noul": 50.0, "score": -3.675381607209216 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 36.07646009918423 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 0.8461538461538461, "cc": 73.84108691654622, "cc_by_type": { "choice": 79.26267281105991, "noul": 100.0, "score": 42.260587938578766 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 73.84108691654622 }, "de": { "accuracy": 0.90625, "cc": 90.42477813573423, "cc_by_type": { "choice": 84.07079646017698, "noul": 100.0, "score": 87.20353794702571 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 90.42477813573423 }, "el": { "accuracy": 0.8, "cc": 77.579631749179, "cc_by_type": { "choice": 66.4804469273743, "noul": 66.66666666666667, "score": 99.59178165349604 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 77.579631749179 }, "en": { "accuracy": 0.8227628149435273, "cc": 71.3569930395452, "cc_by_type": { "choice": 83.0853907079484, "noul": 75.53956834532374, "score": 55.44602006536346 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 71.3569930395452 }, "es": { "accuracy": 0.8064516129032258, "cc": 69.16556628426598, "cc_by_type": { "choice": 80.42414355628058, "noul": 100.0, "score": 27.07255529651733 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 69.16556628426598 }, "fi": { "accuracy": 0.875, "cc": 88.9244647500369, "cc_by_type": { "choice": 67.21311475409836, "noul": 100.0, "score": 99.56027949601237 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 88.9244647500369 }, "fr": { "accuracy": 0.9285714285714286, "cc": 93.75128961226879, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 81.25386883680636 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 93.75128961226879 }, "hi": { "accuracy": 0.8571428571428571, "cc": 73.31989019858385, "cc_by_type": { "choice": 99.99999999999999, "noul": 100.0, "score": 19.959670595751543 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 73.31989019858385 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8, "cc": 56.92598694485033, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": -15.192421082830165 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 56.92598694485033 }, "ja": { "accuracy": 0.72, "cc": 46.34982921310412, "cc_by_type": { "choice": 57.108816521048446, "noul": 80.0, "score": 1.9406711182639036 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 46.34982921310412 }, "ko": { "accuracy": 0.9, "cc": 60.502271657899314, "cc_by_type": { "choice": 100.0, "noul": 0.0, "score": 81.50681497369794 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 60.502271657899314 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 77.73180568607792, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 99.86208372490044 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 77.73180568607792 }, "nl": { "accuracy": 0.5, "cc": 8.642558761906725, "cc_by_type": { "choice": 48.05194805194805, "noul": -33.333333333333336, "score": 11.209061567105461 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 8.642558761906725 }, "no": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0, "noul": 100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 100.0 }, "pl": { "accuracy": 0.8205128205128205, "cc": 63.392816051625765, "cc_by_type": { "choice": 92.382564536606, "noul": 39.99999999999999, "score": 57.79588361827128 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 63.392816051625765 }, "pt": { "accuracy": 0.7142857142857143, "cc": 67.80504865797501, "cc_by_type": { "choice": 46.96523276370065, "noul": 100.0, "score": 56.44991321022436 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 67.80504865797501 }, "sv": { "accuracy": 0.6363636363636364, "cc": 46.19054683365891, "cc_by_type": { "choice": 48.93617021276595, "noul": 60.00000000000001, "score": 29.63547028821076 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 46.19054683365891 }, "tr": { "accuracy": 0.7692307692307693, "cc": 68.75012962289753, "cc_by_type": { "choice": 78.26086956521739, "noul": 100.0, "score": 27.989519303475184 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 68.75012962289753 }, "uk": { "accuracy": 0.5714285714285714, "cc": 61.68878465176133, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 49.582482987542065 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 61.68878465176133 }, "zh": { "accuracy": 0.9, "cc": 69.21114832796756, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 7.63344498390266 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 69.21114832796756 } }, "type": { "choice": { "accuracy": 0.8573333333333333, "cc": 81.57255209628963, "cc_by_type": { "choice": 81.57255209628963 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 81.57255209628963 }, "noul": { "accuracy": 0.8773333333333333, "cc": 75.46666666666665, "cc_by_type": { "noul": 75.46666666666665 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 75.46666666666665 }, "score": { "accuracy": 0.6746666666666666, "cc": 54.7684541832536, "cc_by_type": { "score": 54.7684541832536 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.7684541832536 } }, "usecase": { "advertising": { "accuracy": 0.7272727272727273, "cc": 48.45861881975295, "cc_by_type": { "choice": 84.0, "noul": 25.0, "score": 36.375856459258834 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 48.45861881975295 }, "code_linting": { "accuracy": 0.7916666666666666, "cc": 76.75345303484892, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 30.26035910454675 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 76.75345303484892 }, "customer_support": { "accuracy": 0.8571428571428571, "cc": 69.6944643957582, "cc_by_type": { "choice": 83.73983739837398, "noul": 100.0, "score": 25.343555788900616 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 69.6944643957582 }, "demand_forecasting": { "accuracy": 0.23529411764705882, "cc": -24.108890369968993, "cc_by_type": { "choice": -5.263157894736844, "noul": -33.333333333333336, "score": -33.7301798818368 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.7647058823529411, "cc": 57.255300551831404, "cc_by_type": { "choice": 71.0144927536232, "noul": 33.33333333333333, "score": 67.41807556853769 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 57.255300551831404 }, "feature_extraction": { "accuracy": 0.7894736842105263, "cc": 35.41674609079076, "cc_by_type": { "choice": 74.84276729559748, "noul": 66.66666666666667, "score": -35.25919568989187 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 35.41674609079076 }, "financial_crime": { "accuracy": 0.6333333333333333, "cc": 56.61757692065063, "cc_by_type": { "choice": 70.58823529411765, "noul": 71.42857142857142, "score": 27.835924039262792 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 56.61757692065063 }, "gaming": { "accuracy": 0.6818181818181818, "cc": 43.298609533332666, "cc_by_type": { "choice": 68.6411149825784, "noul": 0.0, "score": 61.2547136174196 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 43.298609533332666 }, "insurance_claims": { "accuracy": 0.4666666666666667, "cc": 40.88581737462815, "cc_by_type": { "choice": 37.5, "noul": 60.00000000000001, "score": 25.15745212388444 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 40.88581737462815 }, "knowledge_graphs": { "accuracy": 0.7727272727272727, "cc": 70.06051887452556, "cc_by_type": { "choice": 74.30406852248395, "noul": 100.0, "score": 35.877488101092766 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 70.06051887452556 }, "lead_generation": { "accuracy": 0.8, "cc": 64.06806981529785, "cc_by_type": { "choice": 100.0, "noul": 77.77777777777777, "score": 14.426431668115768 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 64.06806981529785 }, "legal_compliance": { "accuracy": 0.6, "cc": 49.78201560348273, "cc_by_type": { "choice": 3.2258064516129026, "noul": 100.0, "score": 46.1202403588353 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 49.78201560348273 }, "llm_guardrails": { "accuracy": 0.8148148148148148, "cc": 72.7024042480826, "cc_by_type": { "choice": 100.00000000000001, "noul": 60.00000000000001, "score": 58.107212744247796 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 72.7024042480826 }, "model_routing": { "accuracy": 0.9032258064516129, "cc": 80.21287919965458, "cc_by_type": { "choice": 91.81818181818183, "noul": 100.0, "score": 48.82045578078191 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 80.21287919965458 }, "moderation_trust_safety": { "accuracy": 0.7222222222222222, "cc": 48.98671618683705, "cc_by_type": { "choice": 76.96737044145875, "noul": 0.0, "score": 69.9927781190524 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 48.98671618683705 }, "recruiting": { "accuracy": 0.7777777777777778, "cc": 78.10700850764788, "cc_by_type": { "choice": 66.57381615598885, "noul": 100.0, "score": 67.74720936695476 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 78.10700850764788 }, "risk_assessment": { "accuracy": 0.7222222222222222, "cc": 56.67924976273154, "cc_by_type": { "choice": 70.44334975369459, "noul": 100.0, "score": -0.40560046549995565 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 56.67924976273154 }, "scientific_discovery": { "accuracy": 0.6521739130434783, "cc": 45.24701491441865, "cc_by_type": { "choice": 56.52173913043478, "noul": 33.33333333333333, "score": 45.88597227948784 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 45.24701491441865 }, "search_retrieval": { "accuracy": 1.0, "cc": 99.61559815097014, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 98.84679445291042 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 99.61559815097014 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 82.97114923360294, 88.56755270121242 ], "I": [ 65.82052260869564, 74.74298970281055 ], "capability": [ 74.80359370984546, 81.42461710858433 ], "composite": { "A": [ 65.30821255607486, 67.94674409767933 ], "B": [ 64.96105873997377, 68.76565711110239 ], "C": [ 65.30821255607486, 67.94674409767933 ] }, "seed": 33 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 87.09471194592824, "capability": 78.90688322585615, "intelligence": 70.71905450578407 }, "per_type": { "choice": { "accuracy": 0.8573333333333333, "calibration": 88.5739405964944, "cc": 81.57255209628963, "cc_by_type": { "choice": 81.57255209628963 }, "cc_tiered": 80.7710802894351, "cc_tiered_open": 80.81042171537538, "cc_tiered_sealed": 80.73173886349481, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 81.57255209628963 }, "noul": { "accuracy": 0.8773333333333333, "calibration": 86.27747747747753, "cc": 75.46666666666665, "cc_by_type": { "noul": 75.46666666666665 }, "cc_tiered": 74.10860925693434, "cc_tiered_open": 73.59903381642512, "cc_tiered_sealed": 74.61818469744357, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 75.46666666666665 }, "score": { "accuracy": 0.6746666666666666, "calibration": 86.43271776381279, "cc": 54.7684541832536, "cc_by_type": { "score": 54.7684541832536 }, "cc_tiered": 57.277473970982776, "cc_tiered_open": 63.24668534332811, "cc_tiered_sealed": 51.30826259863744, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.7684541832536 } }, "raw_on_A": { "calibration": 83.28617618857919, "capability": 76.3197068643228, "intelligence": 69.35323754006639 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 62.60259745903769, "Emphasis on Accuracy 60:20:20": 64.86418326296348, "Emphasis on Cost 20:20:60": 55.67505050802875, "Emphasis on Speed 20:60:20": 68.76094684861118, "Intelligence only": 68.58049910691462 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "MeTask (Hangzhou MeTask Technology Co., Ltd.)", "axes": { "calibration": 86.60202339920977, "cost": 47.944910134589236, "intelligence": 72.59243765084337, "speed": 84.66561996644248 }, "calibration": { "parts": { "choice": { "ece_hard": 0.06613866866964949, "ece_score": 86.7722662660701, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 86.7722662660701 }, "noul": { "abstention_rate": 0.0, "brier": 0.1016786666666672, "ece": 0.06551999999999943, "n": 375, "score": 86.89600000000011 }, "score": { "interval90_coverage": 0.92, "interval90_width": 0.20799999999999996, "n": 375, "nrps": 0.06738985445027339, "score": 86.13780393155909, "top_ece": 0.10492703345927233 } }, "score": 86.60202339920977 }, "capability": 79.59723052502656, "composite_ci95": { "A": [ 62.030779246541385, 64.3767232323197 ], "B": [ 62.06051651483721, 65.46093434113322 ], "C": [ 62.030779246541385, 64.3767232323197 ] }, "cost": { "basis": "v1.5-M2 base-model reference: exact base google/gemma-4-31B-it, OpenRouter google/gemma-4-31b-it list price USD 0.09/M input, USD 0.34/M output (the same reference as the published metask_jev_rain_31b_trained row; snapshot read 2026-10-13T12:10:25Z by the order's public-docs receipt) x the service's own measured usage tokens (840,049 in / 17,397 out); estimated, not charged.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.05434626 }, "display": "metask-jev-rain-31B-trained service v3", "endpoint_condition": "evaluator-owned Lium GPU pod (RTX PRO 6000 96 GB; the reviewed recipe needs 96 GB), offline read-only container (--network none, verified by docker inspect before the input upload); the author's own service at revision 2feff076 (service v3), unchanged defaults. Paid public fast-lane order 4d9fef10. RE-SUBMISSION, DISCLOSED: the same LoRA weights as the published metask_jev_rain_31b_trained row (order 147c5e18, v1.6.10) with a changed service/ layer (e.g. its yes/no constant); those changes may have been tuned on that row's published v1.6.10 aggregates (custodian #18945). Both rows are shown; no third same-weights row is admitted on this draw. Weights file shas: model 1b9b1d62..., adapter 2b956899..., code tree 6a1234ee...", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 72.52301075937709, "I_sealed": 72.66186454230964, "base": 72.59243765084337, "excess": -9.978380553288268, "gap": -0.13885378293255712, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 80.81042171537538, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 100.00000000000001, "hard": 81.12614029569046, "judge": 76.26895187870798, "standard": 77.39640016743407 } }, "open|noul": { "cc": 73.59903381642512, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 82.6086956521739, "judge": 50.0, "standard": 77.77777777777777 } }, "open|score": { "cc": 63.15957674633078, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 73.16584682233966, "hard": 69.17688425857045, "judge": 60.01699079087648, "standard": 50.83570561702846 } }, "sealed|choice": { "cc": 82.58817768397486, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 94.28208386277001, "hard": 80.2243552677229, "judge": 79.72575905974536, "standard": 85.76249736342545 } }, "sealed|noul": { "cc": 76.53948727546258, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 86.20689655172413, "hard": 69.0909090909091, "judge": 81.53846153846153, "standard": 79.10447761194031 } }, "sealed|score": { "cc": 58.85792866749151, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 69.80503007296836, "hard": 56.95507418481911, "judge": 61.45515272353761, "standard": 53.2942508460287 } } } }, "jevbench_score": 63.422592654952346, "key": "metask_jev_rain_31b_trained_v3", "last_measured_on": "2026-10-13", "licence": "Weights: Apache-2.0 per model card (base Gemma 4 under the Gemma 4 licence); service code (service/) MIT", "listing": "ranked", "measured_in": "v1.6.17", "measurement_date_status": "measured", "model_pin": "MeTask-Rain/metask-jev-rain-31B-trained@2feff076f07711fe2fefe7a6fdcedfe4132b773e", "noul_decisive": { "acc_among_decisive": 0.8853333333333333, "decisive": 375, "decisive_rate": 1.0, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 10, "ranked": true, "ranks": { "A": 10, "B": 9, "C": 8, "capability": 8 }, "repo": "https://huggingface.co/MeTask-Rain/metask-jev-rain-31B-trained", "scores": { "A": 63.422592654952346, "B": 64.06084679489565, "C": 63.422592654952346 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.5115797939710319, "p50_s_raw": 0.18078989698551595, "p95_s_adjusted": 0.6676127476617694, "p95_s_raw": 0.2588063738308847 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.9090909090909091, "cc": 89.03475546368588, "cc_by_type": { "choice": 83.33333333333333, "noul": 100.0, "score": 83.77093305772432 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 89.03475546368588 }, "calibration_gen": { "accuracy": 0.7920792079207921, "cc": 68.68645561782115, "cc_by_type": { "choice": 77.7306468716861, "noul": 68.0, "score": 60.328719981777354 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 68.68645561782115 }, "composite_gen": { "accuracy": 0.8910891089108911, "cc": 84.77793164693051, "cc_by_type": { "choice": 96.7479674796748, "noul": 92.59259259259258, "score": 64.99323486852417 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 84.77793164693051 }, "fairness": { "accuracy": 0.9375, "cc": 86.04372227473641, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 98.13116682420925 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 86.04372227473641 }, "jaggedness": { "accuracy": 0.7, "cc": 55.63506291104909, "cc_by_type": { "choice": 69.68238691049086, "noul": 56.75675675675676, "score": 40.46604506589963 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 55.63506291104909 }, "llm_router": { "accuracy": 0.9297124600638977, "cc": 83.60538459908456, "cc_by_type": { "choice": 94.45276041208066, "noul": 95.8762886597938, "score": 60.48710472537918 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 83.60538459908456 }, "long_state": { "accuracy": 0.3333333333333333, "cc": 8.720258887926093, "cc_by_type": { "choice": 9.677419354838706, "noul": 0.0, "score": 16.483357308939574 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 8.720258887926093 }, "ml_usecase": { "accuracy": 0.8148148148148148, "cc": 62.009380987023235, "cc_by_type": { "choice": 62.886597938144334, "noul": 75.0, "score": 48.14154502292535 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 62.009380987023235 }, "multilingual": { "accuracy": 0.8145454545454546, "cc": 70.77751409077383, "cc_by_type": { "choice": 77.50244538637104, "noul": 72.22222222222223, "score": 62.60787466372824 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 70.77751409077383 }, "negation_pair_gen": { "accuracy": 0.8837209302325582, "cc": 71.91781286399173, "cc_by_type": { "choice": 93.33333333333333, "noul": 60.00000000000001, "score": 62.420105258641826 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 71.91781286399173 }, "option_order": { "accuracy": 0.9791666666666666, "cc": 97.3368841544607, "cc_by_type": { "choice": 97.3368841544607 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 97.3368841544607 }, "perturb_pair": { "accuracy": 0.875, "cc": 82.80777431471158, "cc_by_type": { "choice": 84.98498498498498, "noul": 100.0, "score": 63.43833795914975 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 82.80777431471158 }, "usecase": { "accuracy": 0.7772727272727272, "cc": 68.49729252167786, "cc_by_type": { "choice": 76.7156862745098, "noul": 70.83333333333333, "score": 57.942857957190455 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 68.49729252167786 } }, "lang": { "ar": { "accuracy": 0.75, "cc": 43.24855438365077, "cc_by_type": { "choice": 61.904761904761905, "noul": 50.0, "score": 17.84090124619041 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 43.24855438365077 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 0.8461538461538461, "cc": 74.01835450650829, "cc_by_type": { "choice": 79.26267281105991, "noul": 100.0, "score": 42.79239070846494 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 74.01835450650829 }, "de": { "accuracy": 0.90625, "cc": 81.83435068683548, "cc_by_type": { "choice": 84.07079646017698, "noul": 100.0, "score": 61.432255600329455 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 81.83435068683548 }, "el": { "accuracy": 0.8, "cc": 77.60222006784046, "cc_by_type": { "choice": 66.4804469273743, "noul": 66.66666666666667, "score": 99.65954660948044 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 77.60222006784046 }, "en": { "accuracy": 0.8410078192875761, "cc": 74.40408858051423, "cc_by_type": { "choice": 84.40341221122515, "noul": 76.97841726618707, "score": 61.83043626413047 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 74.40408858051423 }, "es": { "accuracy": 0.8064516129032258, "cc": 68.2680286109703, "cc_by_type": { "choice": 80.42414355628058, "noul": 100.0, "score": 24.379942276630317 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 68.2680286109703 }, "fi": { "accuracy": 0.875, "cc": 88.93547668805367, "cc_by_type": { "choice": 67.21311475409836, "noul": 100.0, "score": 99.59331531006269 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 88.93547668805367 }, "fr": { "accuracy": 0.9285714285714286, "cc": 93.7369976649519, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 81.2109929948557 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 93.7369976649519 }, "hi": { "accuracy": 0.9047619047619048, "cc": 80.07784582404595, "cc_by_type": { "choice": 99.99999999999999, "noul": 100.0, "score": 40.23353747213787 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 80.07784582404595 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.7333333333333333, "cc": 54.53760923768931, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": -22.357554204313203 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 54.53760923768931 }, "ja": { "accuracy": 0.84, "cc": 64.88034315630443, "cc_by_type": { "choice": 85.70293884034949, "noul": 80.0, "score": 28.938090628563796 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 64.88034315630443 }, "ko": { "accuracy": 0.9, "cc": 60.26417092788616, "cc_by_type": { "choice": 100.0, "noul": 0.0, "score": 80.79251278365848 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 60.26417092788616 }, "mixed": { "accuracy": 1.0, "cc": 99.93609912586727, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 99.80829737760182 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 99.93609912586727 }, "nl": { "accuracy": 0.5, "cc": 8.53215978329243, "cc_by_type": { "choice": 48.05194805194805, "noul": -33.333333333333336, "score": 10.877864631262579 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 8.53215978329243 }, "no": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0, "noul": 100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 100.0 }, "pl": { "accuracy": 0.8205128205128205, "cc": 63.14426530245589, "cc_by_type": { "choice": 92.382564536606, "noul": 39.99999999999999, "score": 57.050231370761686 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 63.14426530245589 }, "pt": { "accuracy": 0.7857142857142857, "cc": 74.38219774143825, "cc_by_type": { "choice": 57.57218621096051, "noul": 100.0, "score": 65.57440701335422 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 74.38219774143825 }, "sv": { "accuracy": 0.6363636363636364, "cc": 46.19054710826382, "cc_by_type": { "choice": 48.93617021276595, "noul": 60.00000000000001, "score": 29.635471112025513 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 46.19054710826382 }, "tr": { "accuracy": 0.7692307692307693, "cc": 69.0713909756119, "cc_by_type": { "choice": 78.26086956521739, "noul": 100.0, "score": 28.95330336161831 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 69.0713909756119 }, "uk": { "accuracy": 0.5714285714285714, "cc": 61.68232554763859, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 49.56310567517384 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 61.68232554763859 }, "zh": { "accuracy": 0.9, "cc": 69.21240428354913, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 7.637212850647435 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 69.21240428354913 } }, "type": { "choice": { "accuracy": 0.8693333333333333, "cc": 83.12252434987275, "cc_by_type": { "choice": 83.12252434987275 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 83.12252434987275 }, "noul": { "accuracy": 0.8853333333333333, "cc": 77.06666666666666, "cc_by_type": { "noul": 77.06666666666666 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 77.06666666666666 }, "score": { "accuracy": 0.7173333333333334, "cc": 60.018133778302385, "cc_by_type": { "score": 60.018133778302385 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 60.018133778302385 } }, "usecase": { "advertising": { "accuracy": 0.7272727272727273, "cc": 49.15314922895124, "cc_by_type": { "choice": 84.0, "noul": 25.0, "score": 38.45944768685374 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 49.15314922895124 }, "code_linting": { "accuracy": 0.8333333333333334, "cc": 80.079808393905, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 40.239425181714985 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 80.079808393905 }, "customer_support": { "accuracy": 0.8571428571428571, "cc": 84.23287684117527, "cc_by_type": { "choice": 67.47967479674797, "noul": 100.0, "score": 85.21895572677785 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 84.23287684117527 }, "demand_forecasting": { "accuracy": 0.23529411764705882, "cc": -20.319424506293327, "cc_by_type": { "choice": -5.263157894736844, "noul": -33.333333333333336, "score": -22.361782290809806 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.7647058823529411, "cc": 57.24850688656365, "cc_by_type": { "choice": 71.0144927536232, "noul": 33.33333333333333, "score": 67.39769457273441 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 57.24850688656365 }, "feature_extraction": { "accuracy": 0.7894736842105263, "cc": 30.523095261874122, "cc_by_type": { "choice": 74.84276729559748, "noul": 66.66666666666667, "score": -49.940148176641785 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 30.523095261874122 }, "financial_crime": { "accuracy": 0.6666666666666666, "cc": 60.33576603025934, "cc_by_type": { "choice": 70.58823529411765, "noul": 71.42857142857142, "score": 38.99049136808895 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 60.33576603025934 }, "gaming": { "accuracy": 0.7272727272727273, "cc": 46.327230880637806, "cc_by_type": { "choice": 68.6411149825784, "noul": 0.0, "score": 70.340577659335 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 46.327230880637806 }, "insurance_claims": { "accuracy": 0.6, "cc": 54.927785267620116, "cc_by_type": { "choice": 37.5, "noul": 60.00000000000001, "score": 67.28335580286033 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 54.927785267620116 }, "knowledge_graphs": { "accuracy": 0.8181818181818182, "cc": 78.22201831947774, "cc_by_type": { "choice": 87.15203426124197, "noul": 100.0, "score": 47.51402069719127 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 78.22201831947774 }, "lead_generation": { "accuracy": 0.8, "cc": 63.998217861086424, "cc_by_type": { "choice": 100.0, "noul": 77.77777777777777, "score": 14.216875805481488 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 63.998217861086424 }, "legal_compliance": { "accuracy": 0.6, "cc": 51.14449916437249, "cc_by_type": { "choice": 3.2258064516129026, "noul": 100.0, "score": 50.207691041504575 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 51.14449916437249 }, "llm_guardrails": { "accuracy": 0.8518518518518519, "cc": 75.39671275196183, "cc_by_type": { "choice": 100.00000000000001, "noul": 60.00000000000001, "score": 66.19013825588543 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 75.39671275196183 }, "model_routing": { "accuracy": 0.9078341013824884, "cc": 80.45531443691877, "cc_by_type": { "choice": 92.98701298701299, "noul": 100.0, "score": 48.37893032374332 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 80.45531443691877 }, "moderation_trust_safety": { "accuracy": 0.8333333333333334, "cc": 56.46101690387395, "cc_by_type": { "choice": 76.96737044145875, "noul": 0.0, "score": 92.4156802701631 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 56.46101690387395 }, "recruiting": { "accuracy": 0.8333333333333334, "cc": 82.04193370234395, "cc_by_type": { "choice": 66.57381615598885, "noul": 100.0, "score": 79.55198495104301 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 82.04193370234395 }, "risk_assessment": { "accuracy": 0.7222222222222222, "cc": 56.88699086990223, "cc_by_type": { "choice": 70.44334975369459, "noul": 100.0, "score": 0.21762285601210074 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 56.88699086990223 }, "scientific_discovery": { "accuracy": 0.6956521739130435, "cc": 47.859636984518865, "cc_by_type": { "choice": 56.52173913043478, "noul": 33.33333333333333, "score": 53.7238384897885 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 47.859636984518865 }, "search_retrieval": { "accuracy": 1.0, "cc": 99.6261843392683, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 98.87855301780489 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 99.6261843392683 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 83.07016607530771, 89.06520423655694 ], "I": [ 67.49111579069434, 76.40434090096831 ], "capability": [ 75.94084118220704, 82.19002318852692 ], "composite": { "A": [ 62.030779246541385, 64.3767232323197 ], "B": [ 62.06051651483721, 65.46093434113322 ], "C": [ 62.030779246541385, 64.3767232323197 ] }, "seed": 34 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 86.60202339920977, "capability": 79.59723052502656, "intelligence": 72.59243765084337 }, "per_type": { "choice": { "accuracy": 0.8693333333333333, "calibration": 86.7722662660701, "cc": 83.12252434987275, "cc_by_type": { "choice": 83.12252434987275 }, "cc_tiered": 81.69929969967512, "cc_tiered_open": 80.81042171537538, "cc_tiered_sealed": 82.58817768397486, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 83.12252434987275 }, "noul": { "accuracy": 0.8853333333333333, "calibration": 86.89600000000011, "cc": 77.06666666666666, "cc_by_type": { "noul": 77.06666666666666 }, "cc_tiered": 75.06926054594385, "cc_tiered_open": 73.59903381642512, "cc_tiered_sealed": 76.53948727546258, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 77.06666666666666 }, "score": { "accuracy": 0.7173333333333334, "calibration": 86.13780393155909, "cc": 60.018133778302385, "cc_by_type": { "score": 60.018133778302385 }, "cc_tiered": 61.00875270691114, "cc_tiered_open": 63.15957674633078, "cc_tiered_sealed": 58.85792866749151, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 60.018133778302385 } }, "raw_on_A": { "calibration": 81.92265838998793, "capability": 76.06423959692178, "intelligence": 70.20582080385563 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 59.393214247919566, "Emphasis on Accuracy 60:20:20": 62.13156429902127, "Emphasis on Cost 20:20:60": 52.14956700153696, "Emphasis on Speed 20:60:20": 65.61538874423533, "Intelligence only": 66.74771293052854 } }, { "addendum": null, "api_flag": true, "api_subset": true, "author": "Microsoft", "axes": { "calibration": 85.15566582345811, "cost": 62.19876038790379, "intelligence": 59.226098571054024, "speed": 88.69804989142763 }, "calibration": { "parts": { "choice": { "ece_hard": 0.1108223199674765, "ece_score": 77.8355360065047, "mean_tvd": null, "n": 296, "n_ece_hard": 128, "n_tvd": 0, "score": 77.8355360065047 }, "noul": { "abstention_rate": 0.22972972972972971, "brier": 0.08789565767934152, "ece": 0.06454285728875721, "n": 148, "score": 87.09142854224856 }, "score": { "interval90_coverage": 0.9194630872483222, "interval90_width": 0.1784116331096197, "n": 149, "nrps": 0.06306645857440242, "score": 87.1325461475575, "top_ece": 0.09714130923722374 } }, "score": 84.01983689877025 }, "canonical_system_key": "microsoft-decision-1", "capability": 72.19088219725606, "composite_ci95": { "A": [ 60.33591615493133, 74.53742586189195 ], "B": [ 56.31152174249009, 73.1804420130518 ], "C": [ 41.89994177425786, 74.53742586189195 ] }, "cost": { "basis": "LEAD DECISION r3: USD 0.042/M input and USD 0/M output, basis of the live Microsoft-Decision-1 (Azure Foundry) row on https://benchmarkheaven.com/jev-models/api. Reported usage across all 600 decisions, including seven HTTP422 refusals retained in denominator; frozen parent all-item cost basis. Invoice, warmup and refusal charges unknown.", "common_basis": null, "kind": "tariff", "usage_estimated_rows": 0, "usd_per_1000": 0.01819874 }, "display": "Microsoft-Decision-1 (Azure Foundry) — A2 re-run", "endpoint_condition": "Vendor-hosted Azure Foundry API; latency measured from Sandy (Germany). Seven of 600 HTTP 422 responses scored wrong. Four self-hosted systems re-run on H200, versus H100 parents; wide equating interval. MeTask 31B and Torchcast not completed after four failed pod attempts; see measurement notes.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 67.00640482707561, "I_sealed": 46.29756060035449, "base": 56.65198271371505, "excess": 3.8882152491360813, "gap": 20.708844226721126, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 82.78833309663467, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 96.10810810810811, "hard": 73.57659641396666, "judge": 84.17930125247197, "standard": 92.46546672247803 } }, "open|noul": { "cc": 54.94202898550725, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 56.52173913043479, "judge": 30.000000000000004, "standard": 66.66666666666667 } }, "open|score": { "cc": 63.288852399084945, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 52.860607370724196, "hard": 67.50339329867514, "judge": 74.36852911363773, "standard": 43.45437804225577 } }, "sealed|choice": { "cc": 63.30695607434754, "n": { "easy": 15, "hard": 64, "judge": 35, "standard": 36 }, "tiers": { "easy": 35.48387096774193, "hard": 52.901023890784984, "judge": 76.86375321336762, "standard": 77.69516728624535 } }, "sealed|noul": { "cc": 23.62820512820513, "n": { "easy": 13, "hard": 32, "judge": 15, "standard": 15 }, "tiers": { "easy": 84.61538461538463, "hard": 6.25, "judge": -6.666666666666665, "standard": 73.33333333333334 } }, "sealed|score": { "cc": 51.95752059851082, "n": { "easy": 9, "hard": 33, "judge": 20, "standard": 13 }, "tiers": { "easy": 76.44302715293294, "hard": 49.562049785657535, "judge": 44.71869741001873, "standard": 55.363943729744456 } } } }, "jevbench_score": 71.4549280926844, "key": "microsoft-decision-1-a2-20261012", "last_measured_on": "2026-10-12", "listing": "preliminary", "measured_in": "v1.6.14", "measurement_date_status": "measured", "not_ranked_because": "A2 supplement measurement with an incomplete equating pool (four of six); excluded from every ranking.", "planned_weight_denominator": 1.0, "preliminary": { "equating_pool_size": 4, "equating_pool_target": 6, "label": "PRELIMINARY — API supplement A2 (300 sealed + 300 public), not the S1200 full set; equating pool 4 of 6", "measured_on": "2026-10-12", "n_items": 600, "n_ok": 593, "scope": "API supplement A2" }, "provider": "microsoft", "rank": null, "ranked": false, "ranks": { "A": null, "B": null, "C": null, "capability": null }, "repo": "https://commandline.microsoft.com/microsoft-decision-1-model-foundry/", "scores": { "A": 71.4549280926844, "B": 68.62118813181577, "C": 69.62351351359915 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.3374342759998399, "p50_s_raw": 0.3374342759998399, "p95_s_adjusted": 0.3999501053016501, "p95_s_raw": 0.3999501053016501 }, "status": { "answered_ok": 593, "missing": 0, "rows": 600, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": true, "api_subset_tag": "A2", "breakdowns": { "family": { "calibration": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0, "noul": 100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 100.0 }, "calibration_gen": { "accuracy": 0.7692307692307693, "cc": 63.29111148533698, "cc_by_type": { "choice": 60.8695652173913, "noul": 75.0, "score": 54.003769238619604 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 63.29111148533698 }, "composite_gen": { "accuracy": 0.7741935483870968, "cc": 55.04557875346427, "cc_by_type": { "choice": 100.0, "noul": 19.999999999999996, "score": 45.136736260392794 }, "low_n": false, "n": 31, "n_by_type": { "choice": 10, "noul": 10, "score": 11 }, "score": 55.04557875346427 }, "fairness": { "accuracy": 1.0, "cc": 99.45663234640188, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 98.36989703920564 }, "low_n": true, "n": 18, "n_by_type": { "choice": 6, "noul": 6, "score": 6 }, "score": 99.45663234640188 }, "jaggedness": { "accuracy": 0.6, "cc": 36.18096361245757, "cc_by_type": { "choice": 72.78911564625851, "noul": -28.57142857142857, "score": 64.32520376254277 }, "low_n": false, "n": 45, "n_by_type": { "choice": 20, "noul": 14, "score": 11 }, "score": 36.18096361245757 }, "llm_router": { "accuracy": 0.8155339805825242, "cc": 77.29773373392734, "cc_by_type": { "choice": 66.38655462184875, "noul": 100.0, "score": 65.50664657993329 }, "low_n": false, "n": 103, "n_by_type": { "choice": 50, "noul": 30, "score": 23 }, "score": 77.29773373392734 }, "long_state": { "accuracy": 0.0, "cc": -110.3825136612022, "cc_by_type": { "choice": -31.14754098360656, "noul": -100.0, "score": -200.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 2, "score": 1 }, "score": 0.0 }, "multilingual": { "accuracy": 0.7405660377358491, "cc": 56.84729346177679, "cc_by_type": { "choice": 67.37084291989123, "noul": 39.99999999999999, "score": 63.17103746543915 }, "low_n": false, "n": 212, "n_by_type": { "choice": 107, "noul": 40, "score": 65 }, "score": 56.84729346177679 }, "negation_pair_gen": { "accuracy": 0.8, "cc": 60.0, "cc_by_type": { "choice": 100.0, "noul": 19.999999999999996 }, "low_n": true, "n": 20, "n_by_type": { "choice": 10, "noul": 10 }, "score": 60.0 }, "option_order": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0 }, "low_n": true, "n": 21, "n_by_type": { "choice": 21 }, "score": 100.0 }, "perturb_pair": { "accuracy": 1.0, "cc": 97.86434741302423, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 93.59304223907267 }, "low_n": true, "n": 26, "n_by_type": { "choice": 14, "noul": 8, "score": 4 }, "score": 97.86434741302423 }, "usecase": { "accuracy": 0.7058823529411765, "cc": 37.19296114334664, "cc_by_type": { "choice": 67.7891654465593, "noul": 10.000000000000009, "score": 33.78971798348063 }, "low_n": false, "n": 85, "n_by_type": { "choice": 44, "noul": 20, "score": 21 }, "score": 37.19296114334664 } }, "lang": { "ar": { "accuracy": 0.8888888888888888, "cc": 92.43674893442142, "cc_by_type": { "choice": 77.77777777777779, "noul": 100.0, "score": 99.53246902548645 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 2, "score": 1 }, "score": 92.43674893442142 }, "cs": { "accuracy": 0.625, "cc": 66.1654135338346, "cc_by_type": { "choice": 32.330827067669176, "noul": 100.0 }, "low_n": true, "n": 8, "n_by_type": { "choice": 6, "noul": 2 }, "score": 66.1654135338346 }, "da": { "accuracy": 0.6666666666666666, "cc": 29.010593471104, "cc_by_type": { "choice": 80.83067092651757, "noul": 0.0, "score": 6.2011094867944205 }, "low_n": true, "n": 12, "n_by_type": { "choice": 7, "noul": 4, "score": 1 }, "score": 29.010593471104 }, "de": { "accuracy": 0.9230769230769231, "cc": 90.62759668123668, "cc_by_type": { "choice": 74.64788732394366, "noul": 100.0, "score": 97.23490271976637 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 4, "score": 4 }, "score": 90.62759668123668 }, "el": { "accuracy": 0.3333333333333333, "cc": -21.42857142857143, "cc_by_type": { "choice": 57.14285714285714, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 3, "noul": 3 }, "score": 0.0 }, "en": { "accuracy": 0.7753424657534247, "cc": 58.59078985584421, "cc_by_type": { "choice": 76.21638426861493, "noul": 42.574257425742566, "score": 56.981727873175124 }, "low_n": false, "n": 365, "n_by_type": { "choice": 185, "noul": 101, "score": 79 }, "score": 58.59078985584421 }, "es": { "accuracy": 0.9166666666666666, "cc": 70.0797903785218, "cc_by_type": { "choice": 100.00000000000001, "noul": 100.0, "score": 10.239371135565412 }, "low_n": true, "n": 12, "n_by_type": { "choice": 7, "noul": 4, "score": 1 }, "score": 70.0797903785218 }, "fi": { "accuracy": 1.0, "cc": 94.53482291929028, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 83.60446875787086 }, "low_n": true, "n": 10, "n_by_type": { "choice": 2, "noul": 2, "score": 6 }, "score": 94.53482291929028 }, "fr": { "accuracy": 0.75, "cc": 83.59037174371464, "cc_by_type": { "choice": 51.99999999999999, "noul": 100.0, "score": 98.77111523114391 }, "low_n": true, "n": 8, "n_by_type": { "choice": 6, "noul": 1, "score": 1 }, "score": 83.59037174371464 }, "hi": { "accuracy": 1.0, "cc": 92.81199164662956, "cc_by_type": { "choice": 100.00000000000001, "noul": 100.0, "score": 78.43597493988868 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 1, "score": 1 }, "score": 92.81199164662956 }, "id": { "accuracy": 0.7272727272727273, "cc": 70.7560754894387, "cc_by_type": { "choice": 73.09417040358746, "score": 68.41798057528993 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "score": 6 }, "score": 70.7560754894387 }, "it": { "accuracy": 0.7272727272727273, "cc": 80.04349357771646, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 40.13048073314939 }, "low_n": true, "n": 11, "n_by_type": { "choice": 2, "noul": 2, "score": 7 }, "score": 80.04349357771646 }, "ja": { "accuracy": 0.6363636363636364, "cc": 47.61381032028706, "cc_by_type": { "choice": 1.6393442622950836, "noul": 50.0, "score": 91.20208669856609 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 4, "score": 3 }, "score": 47.61381032028706 }, "ko": { "accuracy": 0.8181818181818182, "cc": 77.78079668122632, "cc_by_type": { "choice": 100.0, "score": 55.561593362452655 }, "low_n": true, "n": 11, "n_by_type": { "choice": 3, "score": 8 }, "score": 77.78079668122632 }, "mixed": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0, "noul": 100.0 }, "low_n": true, "n": 13, "n_by_type": { "choice": 11, "noul": 2 }, "score": 100.0 }, "nl": { "accuracy": 0.5714285714285714, "cc": 1.7832893135850607, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": 5.349867940755182 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 2, "score": 1 }, "score": 1.7832893135850607 }, "no": { "accuracy": 0.6363636363636364, "cc": 39.253029666918, "cc_by_type": { "choice": 14.893617021276594, "noul": 33.33333333333333, "score": 69.53213864614409 }, "low_n": true, "n": 11, "n_by_type": { "choice": 3, "noul": 3, "score": 5 }, "score": 39.253029666918 }, "pl": { "accuracy": 0.7, "cc": 52.957577540994386, "cc_by_type": { "choice": 67.97153024911032, "noul": 50.0, "score": 40.901202373872835 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 4, "score": 8 }, "score": 52.957577540994386 }, "pt": { "accuracy": 0.6923076923076923, "cc": 74.25511482979587, "cc_by_type": { "choice": 26.530612244897956, "noul": 100.0, "score": 96.23473224448966 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 4, "score": 2 }, "score": 74.25511482979587 }, "sv": { "accuracy": 0.6363636363636364, "cc": 3.3459357774096503, "cc_by_type": { "choice": 33.70165745856353, "noul": -100.0, "score": 76.33614987366542 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 1, "score": 6 }, "score": 3.3459357774096503 }, "tr": { "accuracy": 0.7142857142857143, "cc": 77.61194029850746, "cc_by_type": { "choice": 55.22388059701492, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 77.61194029850746 }, "uk": { "accuracy": 0.5384615384615384, "cc": -2.908915160021811, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0, "score": 35.71769896437901 }, "low_n": true, "n": 13, "n_by_type": { "choice": 3, "noul": 1, "score": 9 }, "score": 0.0 }, "zh": { "accuracy": 0.8333333333333334, "cc": 89.63050869746054, "cc_by_type": { "choice": 70.58823529411765, "noul": 100.0, "score": 98.30329079826399 }, "low_n": true, "n": 12, "n_by_type": { "choice": 9, "noul": 2, "score": 1 }, "score": 89.63050869746054 } }, "type": { "choice": { "accuracy": 0.7933333333333333, "cc": 73.01414581066375, "cc_by_type": { "choice": 73.01414581066375 }, "low_n": false, "n": 300, "n_by_type": { "choice": 300 }, "score": 73.01414581066375 }, "noul": { "accuracy": 0.7266666666666667, "cc": 45.333333333333336, "cc_by_type": { "noul": 45.333333333333336 }, "low_n": false, "n": 150, "n_by_type": { "noul": 150 }, "score": 45.333333333333336 }, "score": { "accuracy": 0.7466666666666667, "cc": 59.17272659737858, "cc_by_type": { "score": 59.17272659737858 }, "low_n": false, "n": 150, "n_by_type": { "score": 150 }, "score": 59.17272659737858 } }, "usecase": { "advertising": { "accuracy": 0.6153846153846154, "cc": 22.655509813036304, "cc_by_type": { "choice": 73.7991266375546, "noul": -33.333333333333336, "score": 27.50073613488765 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 3, "score": 5 }, "score": 22.655509813036304 }, "code_linting": { "accuracy": 0.5555555555555556, "cc": 30.103576043627953, "cc_by_type": { "choice": 56.83453237410072, "noul": 0.0, "score": 33.47619575678314 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 30.103576043627953 }, "customer_support": { "accuracy": 0.875, "cc": 62.22727451527351, "cc_by_type": { "choice": 100.0, "noul": 0.0, "score": 86.68182354582052 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 2, "score": 1 }, "score": 62.22727451527351 }, "demand_forecasting": { "accuracy": 0.2857142857142857, "cc": -25.73258663586017, "cc_by_type": { "choice": 16.89750692520776, "noul": -33.333333333333336, "score": -60.76193349945493 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.6875, "cc": 37.128494536040996, "cc_by_type": { "choice": 51.61290322580645, "noul": -33.333333333333336, "score": 93.10591371564988 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 3, "score": 5 }, "score": 37.128494536040996 }, "feature_extraction": { "accuracy": 0.875, "cc": 79.10347860874892, "cc_by_type": { "choice": 85.94847775175644, "noul": 100.0, "score": 51.361958074490275 }, "low_n": true, "n": 16, "n_by_type": { "choice": 9, "noul": 1, "score": 6 }, "score": 79.10347860874892 }, "financial_crime": { "accuracy": 0.8666666666666667, "cc": 86.22232602398712, "cc_by_type": { "choice": 80.70739549839227, "noul": 100.0, "score": 77.95958257356911 }, "low_n": true, "n": 15, "n_by_type": { "choice": 7, "noul": 2, "score": 6 }, "score": 86.22232602398712 }, "gaming": { "accuracy": 0.5333333333333333, "cc": 12.129649732040141, "cc_by_type": { "choice": 45.454545454545446, "noul": -33.333333333333336, "score": 24.267737074908315 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 3, "score": 7 }, "score": 12.129649732040141 }, "insurance_claims": { "accuracy": 0.5454545454545454, "cc": 11.15157429187248, "cc_by_type": { "choice": 43.573667711598745, "noul": 0.0, "score": -10.118944835981303 }, "low_n": true, "n": 11, "n_by_type": { "choice": 7, "noul": 2, "score": 2 }, "score": 11.15157429187248 }, "knowledge_graphs": { "accuracy": 0.7272727272727273, "cc": 65.92165664860643, "cc_by_type": { "choice": 56.83453237410071, "noul": 50.0, "score": 90.93043757171858 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 65.92165664860643 }, "lead_generation": { "accuracy": 0.5833333333333334, "cc": 32.27917770928797, "cc_by_type": { "choice": 52.56916996047431, "noul": 0.0, "score": 44.2683631673896 }, "low_n": true, "n": 12, "n_by_type": { "choice": 6, "noul": 4, "score": 2 }, "score": 32.27917770928797 }, "legal_compliance": { "accuracy": 1.0, "cc": 94.12770453740406, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 82.38311361221218 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 1, "score": 4 }, "score": 94.12770453740406 }, "llm_guardrails": { "accuracy": 0.8888888888888888, "cc": 87.81390014088669, "cc_by_type": { "choice": 78.14207650273225, "noul": 100.0, "score": 85.29962391992782 }, "low_n": true, "n": 18, "n_by_type": { "choice": 12, "noul": 1, "score": 5 }, "score": 87.81390014088669 }, "model_routing": { "accuracy": 0.8588235294117647, "cc": 81.49062517209573, "cc_by_type": { "choice": 79.13561847988078, "noul": 100.0, "score": 65.33625703640644 }, "low_n": false, "n": 85, "n_by_type": { "choice": 40, "noul": 24, "score": 21 }, "score": 81.49062517209573 }, "moderation_trust_safety": { "accuracy": 0.5833333333333334, "cc": 1.4715857588347916, "cc_by_type": { "choice": 61.29032258064516, "noul": -100.0, "score": 43.124434695859215 }, "low_n": true, "n": 12, "n_by_type": { "choice": 7, "noul": 1, "score": 4 }, "score": 1.4715857588347916 }, "other": { "accuracy": 0.4, "cc": 15.494864573175938, "cc_by_type": { "choice": 9.774436090225558, "score": 21.215293056126317 }, "low_n": true, "n": 5, "n_by_type": { "choice": 3, "score": 2 }, "score": 15.494864573175938 }, "recruiting": { "accuracy": 0.6666666666666666, "cc": 30.434646255259782, "cc_by_type": { "choice": 49.999999999999986, "noul": 0.0, "score": 41.30393876577936 }, "low_n": true, "n": 9, "n_by_type": { "choice": 3, "noul": 2, "score": 4 }, "score": 30.434646255259782 }, "risk_assessment": { "accuracy": 0.75, "cc": 10.1937605340201, "cc_by_type": { "choice": 52.94117647058823, "noul": -100.0, "score": 77.64010513147205 }, "low_n": true, "n": 12, "n_by_type": { "choice": 6, "noul": 1, "score": 5 }, "score": 10.1937605340201 }, "scientific_discovery": { "accuracy": 0.5714285714285714, "cc": 27.722363031181995, "cc_by_type": { "choice": 63.07692307692309, "noul": 0.0, "score": 20.09016601662289 }, "low_n": true, "n": 14, "n_by_type": { "choice": 7, "noul": 2, "score": 5 }, "score": 27.722363031181995 }, "search_retrieval": { "accuracy": 1.0, "cc": 98.185292235786, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 94.55587670735798 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 5, "score": 4 }, "score": 98.185292235786 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 77.61094153455542, 91.6237504581693 ], "I": [ 47.90571478644313, 68.3083574651847 ], "capability": [ 64.56492363928368, 77.67236999065625 ], "composite": { "A": [ 60.33591615493133, 74.53742586189195 ], "B": [ 56.31152174249009, 73.1804420130518 ], "C": [ 41.89994177425786, 74.53742586189195 ] }, "seed": 22 }, "ci95_basis": "bootstrap over the API system's items (n = A sealed items) AND over the equating offset (pool systems resampled with replacement, items resampled; B_offset = 200)", "ci95_n": 300, "complete": true, "equated": { "calibration": 85.15566582345811, "capability": 72.19088219725606, "intelligence": 59.226098571054024 }, "equating_flag": null, "equating_offset": { "C": 1.1358289246878535, "I": 2.574115857338972 }, "full_set_api": false, "g_med_used": 16.820628977585045, "lane": "api", "metric_basis": "A2 u P (sealed side = supplementary API subset A2, n=300)", "n_items": 600, "n_public": 300, "n_sealed_side": 300, "offset_ci95": { "C": [ -2.4463102425519985, 9.397802125135494 ], "I": [ -4.427206040125616, 10.076945418608098 ] }, "on_S": null, "per_type": { "choice": { "accuracy": 0.7933333333333333, "calibration": 77.8355360065047, "cc": 73.01414581066375, "cc_by_type": { "choice": 73.01414581066375 }, "cc_tiered": 73.0476445854911, "cc_tiered_open": 82.78833309663467, "cc_tiered_sealed": 63.30695607434754, "low_n": false, "n": 300, "n_by_type": { "choice": 300 }, "score": 73.01414581066375 }, "noul": { "accuracy": 0.7266666666666667, "calibration": 87.09142854224856, "cc": 45.333333333333336, "cc_by_type": { "noul": 45.333333333333336 }, "cc_tiered": 39.285117056856194, "cc_tiered_open": 54.94202898550725, "cc_tiered_sealed": 23.62820512820513, "low_n": false, "n": 150, "n_by_type": { "noul": 150 }, "score": 45.333333333333336 }, "score": { "accuracy": 0.7466666666666667, "calibration": 87.1325461475575, "cc": 59.17272659737858, "cc_by_type": { "score": 59.17272659737858 }, "cc_tiered": 57.62318649879788, "cc_tiered_open": 63.288852399084945, "cc_tiered_sealed": 51.95752059851082, "low_n": false, "n": 150, "n_by_type": { "score": 150 }, "score": 59.17272659737858 } }, "raw_on_A": { "calibration": 84.01983689877025, "capability": 70.33590980624265, "intelligence": 56.65198271371505 }, "set": "A2+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 300 }, "noul": { "invalid_rate": 0.013333333333333308, "n": 150 }, "score": { "invalid_rate": 0.00666666666666671, "n": 150 } }, "views": { "Balanced 33:33:33": 67.81784570375109, "Emphasis on Accuracy 60:20:20": 64.0984217053166, "Emphasis on Cost 20:20:60": 65.4526287582965, "Emphasis on Speed 20:60:20": 74.86761060072513, "Intelligence only": 59.22609857105403 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "pst2154", "axes": { "calibration": 89.03920345557668, "cost": null, "intelligence": 49.60038985253669, "speed": 87.70433528726323 }, "calibration": { "parts": { "choice": { "ece_hard": 0.031840143420803706, "ece_score": 93.63197131583925, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 93.63197131583925 }, "noul": { "abstention_rate": 0.30400000000000005, "brier": 0.10770403214312142, "ece": 0.06945449416016128, "n": 375, "score": 86.10910116796775 }, "score": { "interval90_coverage": 0.928, "interval90_width": 0.37866666666666676, "n": 375, "nrps": 0.09148110518890483, "score": 87.37653788292307, "top_ece": 0.0804940685763168 } }, "score": 89.03920345557668 }, "capability": 69.31979665405669, "composite_ci95": { "A": null, "B": null, "C": null }, "cost": { "basis": "UNPRICED. The frozen pricing_v15 module RAISES on the declared base nvidia/Nemotron-Labs-Diffusion-8B ('base model lacks an explicit market-listing decision') and the dated snapshot carries no listing of that family, so this row carries no pricing_base and is listed, measured and shown WITHOUT a cost number and WITHOUT a rank - the pre-declared default (Sage 1.4, Celeris Option A). No rate is invented. The predecessor nemotron-diffusion-8b row's usd_per_1000_override is a v1.5.4 historical carry on a different draw and is deliberately NOT reused. Asked of the deputy on board #19427; the server reports both token fields, so a later pricing decision needs no re-measurement.", "common_basis": null, "kind": "unpriced", "usage_estimated_rows": 0, "usd_per_1000": null }, "display": "Nemotron Deciffusion 8B v2 (pst2154)", "endpoint_condition": "Evaluator-owned rented Lium GPU pod (United States; measurement receipts on file), 1x NVIDIA RTX A6000 48 GB - different hardware from the L40S and GB200 the author quotes and from the H100/RTX 5090 rows of this draw. Started from the author's OWN published container, by DIGEST ghcr.io/pst2154/nemotron-deciffusion@sha256:944c0af4967ebd1e461a3b1faa9d16db197288af24393b21c917617461f35276 (image config sha256:0a4bd2d039b5c2da9ff8b181232ee3714f6b38e31bd9a3100a29326068d6f802), in the author's own documented sandboxed form: docker run --gpus all --shm-size=8g --network none --read-only --tmpfs /tmp:rw,exec,size=16g --user 1000:1000 --cap-drop ALL --security-opt no-new-privileges -v :/models/checkpoint:ro, with QUANTIZATION=none (BF16) and MAX_MODEL_LEN=16384 and every other default of the image (ATTENTION=noncausal, COMPACT_OVER=8192, COMPACT_BUDGET=8192, MAX_BATCHED_TOKENS=65536, --no-enable-prefix-caching, --max-logprobs 1052, --logprobs-mode processed_logits, vLLM nightly fe4df3df0eb7 with the author's overlay). /health reported back quantization none, attention noncausal, max_prompt_tokens 16382. All ten checkpoint files verified against the author's own SHA256SUMS (sha256sum -c, 10/10 OK) BEFORE the container started. One request at a time from the unchanged runner, which reached the server through the container's own network namespace with nsenter - the container published no port and had no network at all. The cut was proven before the sealed input was staged (only lo, no routes, a name probe and a bare-IP probe both failed) and re-proven immediately before the first sealed request, with a synthetic smoke gate passed in between (custody #19428). 1,500/1,500 answered, HTTP 200 on every item, raw p50 105 ms / p95 180 ms / max 1.82 s.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 55.54041453254173, "I_sealed": 43.660365172531655, "base": 49.60038985253669, "excess": 2.0405225896543655, "gap": 11.880049360010076, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 75.4454905515748, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 88.32432432432431, "hard": 84.90091223655237, "judge": 72.31377719182598, "standard": 54.79280033486815 } }, "open|noul": { "cc": 44.98550724637681, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 39.13043478260869, "judge": 19.999999999999996, "standard": 66.66666666666667 } }, "open|score": { "cc": 46.19024579967359, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 71.27224719217729, "hard": 41.00474552808914, "judge": 52.73987117942826, "standard": 34.19580757695866 } }, "sealed|choice": { "cc": 66.84585672814276, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 92.37611181702668, "hard": 65.65282757025557, "judge": 56.80705190989228, "standard": 71.52499472685088 } }, "sealed|noul": { "cc": 23.96819855389078, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 44.827586206896555, "hard": 27.27272727272727, "judge": 7.692307692307687, "standard": 31.343283582089555 } }, "sealed|score": { "cc": 40.167040235561444, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 62.908566475040885, "hard": 39.62773652825137, "judge": 34.70949534264651, "standard": 38.061201869814276 } } } }, "jevbench_score": null, "key": "nemotron-deciffusion-8b-v2", "last_measured_on": "2026-10-15", "licence": "NVIDIA Nemotron Open Model License (https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/); base nvidia/Nemotron-Labs-Diffusion-8B", "listing": "listed", "measured_in": "v1.6.22", "measurement_date_status": "measured", "model_pin": "nemotron-deciffusion/Nemotron-Deciffusion-8B-v2@1dea38015e86e101eb978a33fa493818fc71e366", "not_ranked_because": "unpriced: the frozen pricing path has no market-listing decision for the declared base nvidia/Nemotron-Labs-Diffusion-8B and raises rather than estimating; the deputy's answer to #19427 decides whether an entry is added", "noul_decisive": { "acc_among_decisive": 0.9501915708812261, "decisive": 261, "decisive_rate": 0.696, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": null, "ranked": false, "ranks": { "A": null, "B": null, "C": null, "capability": null }, "repo": "https://huggingface.co/nemotron-deciffusion/Nemotron-Deciffusion-8B-v2", "scores": { "A": null, "B": null, "C": null }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.35293248631060126, "p50_s_raw": 0.10146624315530062, "p95_s_adjusted": 0.48070078976452346, "p95_s_raw": 0.16535039488226172 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.9090909090909091, "cc": 69.81508925384576, "cc_by_type": { "choice": 91.66666666666667, "noul": 100.0, "score": 17.778601094870574 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 69.81508925384576 }, "calibration_gen": { "accuracy": 0.5247524752475248, "cc": 24.130548030105405, "cc_by_type": { "choice": 55.461293743372224, "noul": -19.999999999999996, "score": 36.930350346943996 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 24.130548030105405 }, "composite_gen": { "accuracy": 0.8118811881188119, "cc": 62.586444350419164, "cc_by_type": { "choice": 90.2439024390244, "noul": 40.74074074074075, "score": 56.77468987149234 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 62.586444350419164 }, "fairness": { "accuracy": 0.8125, "cc": 76.48515065368318, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 69.45545196104952 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 76.48515065368318 }, "jaggedness": { "accuracy": 0.5611111111111111, "cc": 22.292630536155855, "cc_by_type": { "choice": 63.90760346487007, "noul": -18.918918918918916, "score": 21.889207062516412 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 22.292630536155855 }, "llm_router": { "accuracy": 0.7635782747603834, "cc": 64.81045782636264, "cc_by_type": { "choice": 65.13163687593556, "noul": 79.38144329896907, "score": 49.91829330418326 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 64.81045782636264 }, "long_state": { "accuracy": 0.5333333333333333, "cc": -14.49955948045043, "cc_by_type": { "choice": 48.387096774193544, "noul": -100.0, "score": 8.114224784455171 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.37037037037037035, "cc": -14.464526914190303, "cc_by_type": { "choice": 13.402061855670107, "noul": -25.0, "score": -31.795642598241013 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.6727272727272727, "cc": 43.55075461566889, "cc_by_type": { "choice": 65.76459080534725, "noul": 16.666666666666675, "score": 48.22100637499275 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 43.55075461566889 }, "negation_pair_gen": { "accuracy": 0.7209302325581395, "cc": 40.54640345029721, "cc_by_type": { "choice": 86.66666666666667, "noul": 6.666666666666665, "score": 28.305877017558302 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 40.54640345029721 }, "option_order": { "accuracy": 0.9375, "cc": 92.01065246338216, "cc_by_type": { "choice": 92.01065246338216 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 92.01065246338216 }, "perturb_pair": { "accuracy": 0.828125, "cc": 76.80928525816542, "cc_by_type": { "choice": 72.97297297297297, "noul": 100.0, "score": 57.45488280152327 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 76.80928525816542 }, "usecase": { "accuracy": 0.6772727272727272, "cc": 46.174276523358934, "cc_by_type": { "choice": 70.58823529411767, "noul": 25.0, "score": 42.934594275959135 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 46.174276523358934 } }, "lang": { "ar": { "accuracy": 0.4375, "cc": 7.324936282948684, "cc_by_type": { "choice": 23.809523809523807, "noul": 100.0, "score": -101.83471496067776 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 7.324936282948684 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 0.6923076923076923, "cc": 43.555049861840054, "cc_by_type": { "choice": 79.26267281105991, "noul": 33.33333333333333, "score": 18.06914344112691 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 43.555049861840054 }, "de": { "accuracy": 0.78125, "cc": 60.20540912770944, "cc_by_type": { "choice": 68.14159292035399, "noul": 75.0, "score": 37.474634462774326 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 60.20540912770944 }, "el": { "accuracy": 0.5333333333333333, "cc": 32.087424203928286, "cc_by_type": { "choice": 66.4804469273743, "noul": -66.66666666666667, "score": 96.44849235107722 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 32.087424203928286 }, "en": { "accuracy": 0.7046046915725456, "cc": 49.66247129995006, "cc_by_type": { "choice": 72.10187818064216, "noul": 33.81294964028776, "score": 43.07258607892028 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 49.66247129995006 }, "es": { "accuracy": 0.8709677419354839, "cc": 67.89447005427888, "cc_by_type": { "choice": 90.2120717781403, "noul": 75.0, "score": 38.47133838469635 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 67.89447005427888 }, "fi": { "accuracy": 0.625, "cc": 46.07913376963507, "cc_by_type": { "choice": 34.42622950819673, "noul": 33.33333333333333, "score": 70.47783846737516 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 46.07913376963507 }, "fr": { "accuracy": 0.42857142857142855, "cc": 23.839275340930822, "cc_by_type": { "choice": 15.294117647058824, "noul": 0.0, "score": 56.223708375733636 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 23.839275340930822 }, "hi": { "accuracy": 0.6190476190476191, "cc": 27.037410682068597, "cc_by_type": { "choice": 49.720670391061454, "noul": 0.0, "score": 31.39156165514434 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 27.037410682068597 }, "id": { "accuracy": 0.5, "cc": -22.222222222222225, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8, "cc": 67.26700899902467, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": 15.830645079692863 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 67.26700899902467 }, "ja": { "accuracy": 0.56, "cc": 22.223201710577673, "cc_by_type": { "choice": 42.81175536139793, "noul": 19.999999999999996, "score": 3.857849770335098 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 22.223201710577673 }, "ko": { "accuracy": 0.7, "cc": 49.120207596142166, "cc_by_type": { "choice": 100.0, "noul": 0.0, "score": 47.36062278842651 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 49.120207596142166 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 76.3181021434982, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 95.62097309716124 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 76.3181021434982 }, "nl": { "accuracy": 0.5, "cc": 0.6113352439164336, "cc_by_type": { "choice": 74.02597402597404, "noul": -100.0, "score": 27.80803170577527 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.6113352439164336 }, "no": { "accuracy": 0.6666666666666666, "cc": 50.0, "cc_by_type": { "choice": 100.0, "noul": 0.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 50.0 }, "pl": { "accuracy": 0.7435897435897436, "cc": 53.9772539570996, "cc_by_type": { "choice": 84.76512907321202, "noul": 19.999999999999996, "score": 57.16663279808676 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 53.9772539570996 }, "pt": { "accuracy": 0.7142857142857143, "cc": 67.38198878819895, "cc_by_type": { "choice": 46.96523276370065, "noul": 100.0, "score": 55.18073360089622 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 67.38198878819895 }, "sv": { "accuracy": 0.45454545454545453, "cc": 26.13951124323292, "cc_by_type": { "choice": 23.404255319148945, "noul": 19.999999999999996, "score": 35.01427841054983 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 26.13951124323292 }, "tr": { "accuracy": 0.6153846153846154, "cc": 29.199516858326863, "cc_by_type": { "choice": 56.52173913043478, "noul": 0.0, "score": 31.076811444545815 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 29.199516858326863 }, "uk": { "accuracy": 0.7142857142857143, "cc": 72.32623665099804, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 81.4948389852522 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 72.32623665099804 }, "zh": { "accuracy": 0.7, "cc": 31.384878910972457, "cc_by_type": { "choice": 74.57627118644068, "noul": 33.33333333333333, "score": -13.754967786856632 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 31.384878910972457 } }, "type": { "choice": { "accuracy": 0.7693333333333333, "cc": 70.2060889033468, "cc_by_type": { "choice": 70.2060889033468 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 70.2060889033468 }, "noul": { "accuracy": 0.6613333333333333, "cc": 32.266666666666666, "cc_by_type": { "noul": 32.266666666666666 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 32.266666666666666 }, "score": { "accuracy": 0.5893333333333334, "cc": 42.502298232284154, "cc_by_type": { "score": 42.502298232284154 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 42.502298232284154 } }, "usecase": { "advertising": { "accuracy": 0.5, "cc": 19.398592732098425, "cc_by_type": { "choice": 36.0, "noul": 0.0, "score": 22.19577819629528 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 19.398592732098425 }, "code_linting": { "accuracy": 0.4166666666666667, "cc": 4.560003558944312, "cc_by_type": { "choice": 25.465838509316768, "noul": 11.111111111111116, "score": -22.896938943594947 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 4.560003558944312 }, "customer_support": { "accuracy": 0.7142857142857143, "cc": 54.984605803255306, "cc_by_type": { "choice": 51.21951219512195, "noul": 50.0, "score": 63.73430521464396 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 54.984605803255306 }, "demand_forecasting": { "accuracy": 0.29411764705882354, "cc": -32.94550967102398, "cc_by_type": { "choice": 29.82456140350877, "noul": -100.0, "score": -28.66109041658069 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5882352941176471, "cc": 22.84201437357555, "cc_by_type": { "choice": 56.52173913043478, "noul": -33.333333333333336, "score": 45.337637323625216 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 22.84201437357555 }, "feature_extraction": { "accuracy": 0.5789473684210527, "cc": 12.742920575752285, "cc_by_type": { "choice": 24.528301886792455, "noul": 33.33333333333333, "score": -19.63287349286893 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 12.742920575752285 }, "financial_crime": { "accuracy": 0.6, "cc": 36.722035568569, "cc_by_type": { "choice": 70.58823529411765, "noul": 14.28571428571428, "score": 25.29215712587507 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 36.722035568569 }, "gaming": { "accuracy": 0.5454545454545454, "cc": 28.13714675856338, "cc_by_type": { "choice": 47.73519163763067, "noul": 0.0, "score": 36.67624863805947 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 28.13714675856338 }, "insurance_claims": { "accuracy": 0.4, "cc": 18.23892289455222, "cc_by_type": { "choice": 58.33333333333333, "noul": -60.0, "score": 56.38343535032333 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 18.23892289455222 }, "knowledge_graphs": { "accuracy": 0.6363636363636364, "cc": 43.4296215704276, "cc_by_type": { "choice": 61.4561027837259, "noul": 33.33333333333333, "score": 35.49942859422358 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 43.4296215704276 }, "lead_generation": { "accuracy": 0.68, "cc": 50.10373799055933, "cc_by_type": { "choice": 88.3720930232558, "noul": 33.33333333333333, "score": 28.60578761508887 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 50.10373799055933 }, "legal_compliance": { "accuracy": 0.5, "cc": 15.263876228821458, "cc_by_type": { "choice": 67.74193548387096, "noul": -33.333333333333336, "score": 11.383026535926744 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 15.263876228821458 }, "llm_guardrails": { "accuracy": 0.8148148148148148, "cc": 78.76271298376912, "cc_by_type": { "choice": 88.48368522072937, "noul": 100.0, "score": 47.804453730577976 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 78.76271298376912 }, "model_routing": { "accuracy": 0.7880184331797235, "cc": 67.78953178167247, "cc_by_type": { "choice": 71.94805194805195, "noul": 93.93939393939394, "score": 37.48114945757148 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 67.78953178167247 }, "moderation_trust_safety": { "accuracy": 0.6666666666666666, "cc": 11.731104036301266, "cc_by_type": { "choice": 65.4510556621881, "noul": -100.0, "score": 69.74225644671569 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 11.731104036301266 }, "recruiting": { "accuracy": 0.8888888888888888, "cc": 85.90646185176752, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 57.719385555302516 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 85.90646185176752 }, "risk_assessment": { "accuracy": 0.4444444444444444, "cc": 7.194103790460911, "cc_by_type": { "choice": 40.88669950738917, "noul": -50.0, "score": 30.695611863993566 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 7.194103790460911 }, "scientific_discovery": { "accuracy": 0.43478260869565216, "cc": -11.791006340242546, "cc_by_type": { "choice": 42.02898550724638, "noul": -100.0, "score": 22.59799547202598 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.9583333333333334, "cc": 79.5102568122854, "cc_by_type": { "choice": 100.0, "noul": 50.0, "score": 88.53077043685622 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 79.5102568122854 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 83.18591676737714, 89.23178634534294 ], "I": [ 44.370616071367344, 53.69165422867726 ], "capability": [ 64.6443564356608, 70.75508199572516 ], "composite": { "A": null, "B": null, "C": null }, "seed": 35 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 89.03920345557668, "capability": 69.31979665405669, "intelligence": 49.60038985253669 }, "per_type": { "choice": { "accuracy": 0.7693333333333333, "calibration": 93.63197131583925, "cc": 70.2060889033468, "cc_by_type": { "choice": 70.2060889033468 }, "cc_tiered": 71.14567363985879, "cc_tiered_open": 75.4454905515748, "cc_tiered_sealed": 66.84585672814276, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 70.2060889033468 }, "noul": { "accuracy": 0.6613333333333333, "calibration": 86.10910116796775, "cc": 32.266666666666666, "cc_by_type": { "noul": 32.266666666666666 }, "cc_tiered": 34.47685290013379, "cc_tiered_open": 44.98550724637681, "cc_tiered_sealed": 23.96819855389078, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 32.266666666666666 }, "score": { "accuracy": 0.5893333333333334, "calibration": 87.37653788292307, "cc": 42.502298232284154, "cc_by_type": { "score": 42.502298232284154 }, "cc_tiered": 43.17864301761752, "cc_tiered_open": 46.19024579967359, "cc_tiered_sealed": 40.167040235561444, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 42.502298232284154 } }, "raw_on_A": { "calibration": 84.05380304593929, "capability": 67.14117186884195, "intelligence": 50.228540691744605 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": null, "Emphasis on Accuracy 60:20:20": null, "Emphasis on Cost 20:20:60": null, "Emphasis on Speed 20:60:20": null, "Intelligence only": null } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "EryriLabs", "axes": { "calibration": 79.60239191738147, "cost": 71.168725487126, "intelligence": 20.23194846537782, "speed": 54.82852093556197 }, "calibration": { "parts": { "choice": { "ece_hard": 0.12475073319463456, "ece_score": 75.04985336107308, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 75.04985336107308 }, "noul": { "abstention_rate": 0.4879356568364611, "brier": 0.19795608560606281, "ece": 0.11196353518746716, "n": 373, "score": 77.60729296250656 }, "score": { "interval90_coverage": 0.9301075268817204, "interval90_width": 0.6832437275985659, "n": 372, "nrps": 0.1405294657671992, "score": 86.15002942856476, "top_ece": 0.06823497283075278 } }, "score": 79.60239191738147 }, "capability": 49.91717019137964, "composite_ci95": { "A": [ 3.9551183728636277, 10.025383814298642 ], "B": [ 3.1283345268811145, 8.438645773279612 ], "C": [ 2.746609981155297, 6.962072093262948 ] }, "cost": { "basis": "ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 dcfcecb0..., the module in force; its base-model references are unchanged since the dae86740... module this row was first priced with - the two later re-freezes each added one hosted-API launch-list key and nothing else): declared base Qwen/Qwen3.5-2B-Base -> deepinfra:Qwen/Qwen3.5-2B USD 0.02/M input, USD 0.1/M output. The author's shim reports usage {calls, permutations, backend} and NO token fields at all, so the input tokens are counted OFFLINE with the author's own prompt code (nev.layout.single_prompts over hev.decider_layout.render_row) at the exact serving settings, and cross-checked against llama-server's own prompt-token counter delta from the measured pass; output tokens are 0, because one readout position per call is restricted to the option letters and nothing is returned as text (the pjev-2b convention and the 9 Oct CoS rule for native option-probability models). Estimated, not charged, and not a quote from the author.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.009142026666666666 }, "display": "Nev-2B (EryriLabs)", "endpoint_condition": "Sandy, the evaluator's own SHARED CPU host (AMD Ryzen 5 3600, 6 cores / 12 threads, Zen 2, AVX2, no AVX-512), NOT a dedicated machine and NOT a GPU: other jobs ran beside it throughout and the 1-minute load average was sampled every 60 s (measurement receipts on file) - over this row's window of 199 samples it ran 6.89 to 35.91, mean 13.06. The author's own two documented processes, unedited: llama-server -m nev-Q8_0.gguf -c 8192 --port 8080 --metrics (llama.cpp b11372, the one build the author names; --metrics is the only addition and changes no model behaviour), and the author's shim PYTHONPATH=python python -m nev.serve --backend llama-server:http://127.0.0.1:8080 --gguf nev-Q8_0.gguf on port 8090, with --backend-arg timeout=900 (the author's own documented flag, raised from the 300 s default because a loaded shared box can otherwise turn an answerable item into a false refusal). llama.cpp chose n_threads=6, its own default; the author passes none. One request at a time from the unchanged runner. Isolation: a user+network namespace (unshare -Urn, loopback only - Sandy permits no privileged unshare -n), and the author's two processes additionally inside bwrap with a cleared environment, a read-only root, tmpfs /home and /tmp and read-only binds of only the GGUF, the llama.cpp build, the venv, its interpreter and the author's own python/ tree, so the sealed item directory does not exist inside that sandbox; the runner stays outside it and the author's code never sees the file. All three cuts were PROVEN before the sealed input was read (only lo, no routes, a name probe and a bare-IP probe both failing, and an ls of the sealed tree failing inside the sandbox) and the network cut was re-proven immediately before the first sealed request, with a synthetic smoke gate passed in between (custody #19428). 1,485/1,500 answered, the 15 refusals all HTTP 502 relaying llama-server's own 400 exceed_context_size_error. Raw p50 7.02 s / p95 15.29 s / max 56.88 s - this is the shared box, not the model, and it is not comparable to the GPU and API rows of this draw.", "endpoint_kind": "cpu", "full_coverage": true, "intelligence": { "I_open": 23.36677716577721, "I_sealed": 17.09711976497843, "base": 20.23194846537782, "excess": -3.5698693695569297, "gap": 6.269657400798781, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 44.315480566561476, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 64.97297297297297, "hard": 43.37842088707141, "judge": 36.71720500988794, "standard": 47.25826705734617 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 14.28571428571428, "hard": -4.347826086956519, "judge": -30.000000000000004, "standard": -22.22222222222222 } }, "open|score": { "cc": 25.784850930770162, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 55.23566077335427, "hard": 18.910462877818656, "judge": 28.909830224325873, "standard": 20.120753175047547 } }, "sealed|choice": { "cc": 42.28746532429039, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 61.88055908513342, "hard": 31.30565514051115, "judge": 47.992164544564176, "standard": 45.89748998101666 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": -13.793103448275868, "hard": -14.54545454545455, "judge": -32.30769230769231, "standard": -4.477611940298509 } }, "sealed|score": { "cc": 9.003893970644905, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 8.190773300943444, "hard": 5.502957884946635, "judge": 6.841648353006702, "standard": 19.655694903349474 } } } }, "jevbench_score": 6.94672938401131, "key": "nev-2b", "last_measured_on": "2026-10-15", "licence": "Apache-2.0 (weights and shim); base Qwen/Qwen3.5-2B-Base Apache-2.0", "listing": "ranked", "measured_in": "v1.6.22", "measurement_date_status": "measured", "model_pin": "EryriLabs/Nev-2B-GGUF@65d329f676d75cc538346e2687bed71187ae76b2", "noul_decisive": { "acc_among_decisive": 0.8324607329842932, "decisive": 191, "decisive_rate": 0.5120643431635389, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": 29, "ranked": true, "ranks": { "A": 29, "B": 31, "C": 31, "capability": 30 }, "repo": "https://huggingface.co/EryriLabs/Nev-2B-GGUF", "scores": { "A": 6.94672938401131, "B": 5.696802372255928, "C": 4.8241176277856335 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 12.836194814974442, "p50_s_raw": 6.343097407487221, "p95_s_adjusted": 25.627816557881303, "p95_s_raw": 12.738908278940652 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.7727272727272727, "cc": 46.938355311096586, "cc_by_type": { "choice": 66.66666666666667, "noul": 100.0, "score": -25.851600733376934 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 46.938355311096586 }, "calibration_gen": { "accuracy": 0.4158415841584158, "cc": 6.186350760967346, "cc_by_type": { "choice": 33.19194061505833, "noul": -12.0, "score": -2.6328883321562913 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 6.186350760967346 }, "composite_gen": { "accuracy": 0.4158415841584158, "cc": 0.4670486607000252, "cc_by_type": { "choice": 34.959349593495936, "noul": -33.333333333333336, "score": -0.2248702780625278 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 0.4670486607000252 }, "fairness": { "accuracy": 0.5625, "cc": 29.939866445263462, "cc_by_type": { "choice": 100.0, "noul": -19.999999999999996, "score": 9.819599335790386 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 29.939866445263462 }, "jaggedness": { "accuracy": 0.29444444444444445, "cc": -15.553356623470975, "cc_by_type": { "choice": 14.82194417709337, "noul": -62.16216216216216, "score": 0.6801481146558652 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.5814696485623003, "cc": 35.257392136280366, "cc_by_type": { "choice": 48.48991811217751, "noul": 21.649484536082465, "score": 35.632773760581124 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 35.257392136280366 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.37037037037037035, "cc": -10.080419559021554, "cc_by_type": { "choice": 25.773195876288657, "noul": -50.0, "score": -6.014454553353321 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.5854545454545454, "cc": 27.40677901913386, "cc_by_type": { "choice": 57.93935441799805, "noul": -2.777777777777779, "score": 27.058760417181315 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 27.40677901913386 }, "negation_pair_gen": { "accuracy": 0.29069767441860467, "cc": -24.37540474381098, "cc_by_type": { "choice": 23.333333333333332, "noul": -86.66666666666667, "score": -9.792880898099598 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.0 }, "option_order": { "accuracy": 0.5208333333333334, "cc": 38.748335552596544, "cc_by_type": { "choice": 38.748335552596544 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 38.748335552596544 }, "perturb_pair": { "accuracy": 0.578125, "cc": 29.40536694686776, "cc_by_type": { "choice": 45.94594594594596, "noul": 19.999999999999996, "score": 22.270154894657324 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 29.40536694686776 }, "usecase": { "accuracy": 0.4909090909090909, "cc": 15.280162860434373, "cc_by_type": { "choice": 53.43137254901962, "noul": -25.0, "score": 17.4091160322835 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 15.280162860434373 } }, "lang": { "ar": { "accuracy": 0.4375, "cc": -1.277013365806873, "cc_by_type": { "choice": 23.809523809523807, "noul": 0.0, "score": -27.640563906944426 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.7142857142857143, "cc": -10.526315789473685, "cc_by_type": { "choice": 78.94736842105263, "noul": -100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 0.0 }, "da": { "accuracy": 0.6923076923076923, "cc": 48.03437833196489, "cc_by_type": { "choice": 79.26267281105991, "noul": 33.33333333333333, "score": 31.50712885150143 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 48.03437833196489 }, "de": { "accuracy": 0.78125, "cc": 64.63827546988519, "cc_by_type": { "choice": 76.10619469026548, "noul": 75.0, "score": 42.8086317193901 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 64.63827546988519 }, "el": { "accuracy": 0.3333333333333333, "cc": 12.633202749033721, "cc_by_type": { "choice": 16.201117318435756, "noul": -66.66666666666667, "score": 88.36515759533208 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 12.633202749033721 }, "en": { "accuracy": 0.45438748913987836, "cc": 8.999756883752474, "cc_by_type": { "choice": 38.93167034817731, "noul": -21.58273381294964, "score": 9.650334116029747 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 8.999756883752474 }, "es": { "accuracy": 0.6774193548387096, "cc": 41.361890062571156, "cc_by_type": { "choice": 80.42414355628058, "noul": 50.0, "score": -6.338473368567121 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 41.361890062571156 }, "fi": { "accuracy": 0.25, "cc": -12.72873038858315, "cc_by_type": { "choice": 1.6393442622950836, "noul": -33.333333333333336, "score": -6.492202094711197 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 0.0 }, "fr": { "accuracy": 0.5, "cc": 31.924464844536768, "cc_by_type": { "choice": 36.47058823529412, "noul": 0.0, "score": 59.30280629831618 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 31.924464844536768 }, "hi": { "accuracy": 0.5238095238095238, "cc": 10.330713957506195, "cc_by_type": { "choice": 66.4804469273743, "noul": -25.0, "score": -10.488305054855719 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 10.330713957506195 }, "id": { "accuracy": 0.75, "cc": 77.77777777777777, "cc_by_type": { "choice": 55.55555555555555, "noul": 100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 77.77777777777777 }, "it": { "accuracy": 0.8, "cc": 68.38283399191003, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": 19.178120058348945 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 68.38283399191003 }, "ja": { "accuracy": 0.6, "cc": 29.951924796051298, "cc_by_type": { "choice": 42.81175536139793, "noul": 19.999999999999996, "score": 27.04401902675596 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 29.951924796051298 }, "ko": { "accuracy": 0.7, "cc": 26.885351406385215, "cc_by_type": { "choice": 55.55555555555555, "noul": 0.0, "score": 25.100498663600103 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 26.885351406385215 }, "mixed": { "accuracy": 0.7142857142857143, "cc": 61.38675762936513, "cc_by_type": { "choice": 66.66666666666667, "noul": 33.33333333333333, "score": 84.1602728880954 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 61.38675762936513 }, "nl": { "accuracy": 0.5, "cc": 5.740053731306311, "cc_by_type": { "choice": 48.05194805194805, "noul": -33.333333333333336, "score": 2.501546475304217 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 5.740053731306311 }, "no": { "accuracy": 0.6666666666666666, "cc": 50.0, "cc_by_type": { "choice": 100.0, "noul": 0.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 50.0 }, "pl": { "accuracy": 0.6410256410256411, "cc": 35.205060203589944, "cc_by_type": { "choice": 84.76512907321202, "noul": 0.0, "score": 20.850051537557825 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 35.205060203589944 }, "pt": { "accuracy": 0.5, "cc": 8.889197461537425, "cc_by_type": { "choice": 25.75132586918091, "noul": -19.999999999999996, "score": 20.91626651543136 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 8.889197461537425 }, "sv": { "accuracy": 0.2727272727272727, "cc": -10.185723298219905, "cc_by_type": { "choice": 23.404255319148945, "noul": -60.0, "score": 6.038574786191342 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.3076923076923077, "cc": -19.83194292150323, "cc_by_type": { "choice": 13.043478260869565, "noul": -50.0, "score": -22.53930702537925 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 0.0 }, "uk": { "accuracy": 0.8571428571428571, "cc": 86.98909963559849, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 60.967298906795484 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 86.98909963559849 }, "zh": { "accuracy": 0.6, "cc": -3.638185596880632, "cc_by_type": { "choice": 74.57627118644068, "noul": -33.333333333333336, "score": -52.15749464374924 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 } }, "type": { "choice": { "accuracy": 0.56, "cc": 43.1676840352858, "cc_by_type": { "choice": 43.1676840352858 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 43.1676840352858 }, "noul": { "accuracy": 0.424, "cc": -15.200000000000003, "cc_by_type": { "noul": -15.200000000000003 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.4, "cc": 13.212269569733326, "cc_by_type": { "score": 13.212269569733326 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 13.212269569733326 } }, "usecase": { "advertising": { "accuracy": 0.4090909090909091, "cc": 6.118021937826218, "cc_by_type": { "choice": 52.0, "noul": -50.0, "score": 16.35406581347866 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 6.118021937826218 }, "code_linting": { "accuracy": 0.375, "cc": -7.611128605515098, "cc_by_type": { "choice": 62.73291925465839, "noul": -77.77777777777779, "score": -7.788527293425895 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 0.0 }, "customer_support": { "accuracy": 0.7857142857142857, "cc": 45.204629089096905, "cc_by_type": { "choice": 67.47967479674797, "noul": 50.0, "score": 18.134212470542753 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 45.204629089096905 }, "demand_forecasting": { "accuracy": 0.47058823529411764, "cc": 11.222937880038742, "cc_by_type": { "choice": 29.82456140350877, "noul": -33.333333333333336, "score": 37.177585569940796 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 11.222937880038742 }, "ecommerce_marketplaces": { "accuracy": 0.6470588235294118, "cc": 27.98629504757422, "cc_by_type": { "choice": 85.50724637681158, "noul": -33.333333333333336, "score": 31.784972099244424 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 27.98629504757422 }, "feature_extraction": { "accuracy": 0.42105263157894735, "cc": -25.92255453318933, "cc_by_type": { "choice": 24.528301886792455, "noul": -33.333333333333336, "score": -68.96263215302712 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5, "cc": 20.969729542778605, "cc_by_type": { "choice": 80.3921568627451, "noul": -42.85714285714286, "score": 25.37417462273358 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 20.969729542778605 }, "gaming": { "accuracy": 0.22727272727272727, "cc": -20.360993700444038, "cc_by_type": { "choice": 16.37630662020906, "noul": -100.0, "score": 22.54071227845883 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.2, "cc": -2.256037962010261, "cc_by_type": { "choice": -4.166666666666665, "noul": -19.999999999999996, "score": 17.398552780635878 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.3181818181818182, "cc": 3.402882822738425, "cc_by_type": { "choice": 10.064239828693788, "noul": 0.0, "score": 0.1444086395214872 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 3.402882822738425 }, "lead_generation": { "accuracy": 0.4, "cc": 5.5660152900999575, "cc_by_type": { "choice": 41.860465116279066, "noul": -11.111111111111116, "score": -14.05130813486808 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 5.5660152900999575 }, "legal_compliance": { "accuracy": 0.4, "cc": -12.359465939103076, "cc_by_type": { "choice": 67.74193548387096, "noul": -100.0, "score": -4.820333301180191 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.5925925925925926, "cc": 14.8266510341226, "cc_by_type": { "choice": 76.96737044145875, "noul": -60.0, "score": 27.512582660909047 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 14.8266510341226 }, "model_routing": { "accuracy": 0.5852534562211982, "cc": 32.503636070558116, "cc_by_type": { "choice": 54.41558441558442, "noul": 18.181818181818187, "score": 24.91350561427175 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 32.503636070558116 }, "moderation_trust_safety": { "accuracy": 0.5555555555555556, "cc": 26.903819630623776, "cc_by_type": { "choice": 53.93474088291747, "noul": 0.0, "score": 26.77671800895386 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 26.903819630623776 }, "recruiting": { "accuracy": 0.5, "cc": 5.445115065352193, "cc_by_type": { "choice": 49.860724233983284, "noul": -33.333333333333336, "score": -0.19204570459336878 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 5.445115065352193 }, "risk_assessment": { "accuracy": 0.3888888888888889, "cc": -8.059756459304968, "cc_by_type": { "choice": 40.88669950738917, "noul": -50.0, "score": -15.06596888530407 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.5652173913043478, "cc": 23.630586887314262, "cc_by_type": { "choice": 56.52173913043478, "noul": -33.333333333333336, "score": 47.70335486484133 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 23.630586887314262 }, "search_retrieval": { "accuracy": 0.625, "cc": 48.89395421647513, "cc_by_type": { "choice": 67.97153024911032, "noul": 50.0, "score": 28.710332400315085 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 48.89395421647513 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 73.84590078514807, 82.44210784130463 ], "I": [ 16.20990292645624, 23.40150138631245 ], "capability": [ 45.91651814476041, 52.159459655207804 ], "composite": { "A": [ 3.9551183728636277, 10.025383814298642 ], "B": [ 3.1283345268811145, 8.438645773279612 ], "C": [ 2.746609981155297, 6.962072093262948 ] }, "seed": 36 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 79.60239191738147, "capability": 49.91717019137964, "intelligence": 20.23194846537782 }, "per_type": { "choice": { "accuracy": 0.56, "calibration": 75.04985336107308, "cc": 43.1676840352858, "cc_by_type": { "choice": 43.1676840352858 }, "cc_tiered": 43.30147294542593, "cc_tiered_open": 44.315480566561476, "cc_tiered_sealed": 42.28746532429039, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 43.1676840352858 }, "noul": { "accuracy": 0.424, "calibration": 77.60729296250656, "cc": -15.200000000000003, "cc_by_type": { "noul": -15.200000000000003 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.4, "calibration": 86.15002942856476, "cc": 13.212269569733326, "cc_by_type": { "score": 13.212269569733326 }, "cc_tiered": 17.394372450707532, "cc_tiered_open": 25.784850930770162, "cc_tiered_sealed": 9.003893970644905, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 13.212269569733326 } }, "raw_on_A": { "calibration": 77.01332672896311, "capability": 47.71286883151212, "intelligence": 18.412410934061132 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 6.010996970991734, "Emphasis on Accuracy 60:20:20": 4.5337702474967685, "Emphasis on Cost 20:20:60": 7.4546746716932075, "Emphasis on Speed 20:60:20": 6.926442013882268, "Intelligence only": 3.3126314570949202 } }, { "addendum": null, "api_flag": true, "api_subset": false, "author": "Bespoke Labs", "axes": { "calibration": 86.63426567315432, "cost": 48.61814579553861, "intelligence": 41.405096245385266, "speed": 88.39856712562789 }, "calibration": { "parts": { "choice": { "ece_hard": 0.08196727615205182, "ece_score": 83.60654476958965, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 83.60654476958965 }, "noul": { "abstention_rate": 0.42666666666666664, "brier": 0.11036389023557593, "ece": 0.06483675752018682, "n": 375, "score": 87.03264849596263 }, "score": { "interval90_coverage": 0.984, "interval90_width": 0.5255555555555554, "n": 375, "nrps": 0.08473015745164805, "score": 89.26360375391067, "top_ece": 0.0649988837350692 } }, "score": 86.63426567315432 }, "capability": 64.01968095926979, "composite_ci95": { "A": [ 28.981226595250288, 49.821188749622806 ], "B": [ 26.165454132173615, 46.72119177956524 ], "C": [ 20.12585180225714, 34.598047742793604 ] }, "cost": { "basis": "Bespoke Labs public list price, USD 0.04/M input, output free (fetched 16 Oct 2026): Bespoke's Nimble pricing page (docs.bespokelabs.ai/nimble/pricing, saved copy sha256 03b06261...) says \"Nimble charges for input tokens only. Output tokens are free.\" and lists nimble-v4 at USD 0.04 per 1M input tokens. Counted as the manufacturer's standard, non-promotional launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops, SubQ Snap and d1-ultra), through a scoped re-freeze of the frozen pricing module that adds exactly three keys (sage-1.4.0, nimble-v4, nimble-v4-max) to its launch-list-price set and nothing else, with a zero-difference proof for every other registry entry and the independent release review; it is the separate pricing change CoS deputy decision #19084 made the cost axis wait for. Free credit for new accounts is a trial credit and is not used: the row is priced at the published tariff, never as cost 0. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (1,935,351 input tokens over all 1,500 answered items; the 3,000 reported output tokens are free under the tariff), divided by the full 1,500-row basis the scorer uses. USD 0.051609 per 1,000 decisions, 0.799x the official cost cap USD 0.064595. Not a bill and not a quote.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.05160936000000001 }, "display": "Bespoke Nimble-v4 (Bespoke Labs)", "endpoint_condition": "Vendor-hosted api.bespokelabs.ai /v1/systemone API, model `nimble-v4`, through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) on the official per-item latency subset without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `bespoke` before the first request (CoS deputy #18998/#19084, custodian #18999; every sealed item of this draw was already retired). Native probabilities on all three question types. Hosting implementation and deployed weights were not independently inspected. Measured as our own add after Bespoke Labs' public announcement of 13 Oct 2026 (no paid order); key on Benchmark Heaven's own account with Bespoke's free credit.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 46.227471634307484, "I_sealed": 36.582720856463055, "base": 41.405096245385266, "excess": -0.19477599251128197, "gap": 9.644750777844429, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 74.38459954164753, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 96.10810810810811, "hard": 69.80182447310474, "judge": 76.26895187870798, "standard": 69.8618668899121 } }, "open|noul": { "cc": 13.326432022084196, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 71.42857142857142, "hard": 4.347826086956519, "judge": 0.0, "standard": 22.222222222222232 } }, "open|score": { "cc": 50.97138333919072, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 49.922961492763804, "hard": 58.84686036423604, "judge": 45.799744250773585, "standard": 43.502098844939226 } }, "sealed|choice": { "cc": 66.57364668901572, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 82.84625158831004, "hard": 66.69365097721753, "judge": 57.68854064642509, "standard": 71.52499472685088 } }, "sealed|noul": { "cc": 3.5399803490385073, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 24.13793103448276, "hard": 3.6363636363636376, "judge": -19.999999999999996, "standard": 28.358208955223873 } }, "sealed|score": { "cc": 39.63453553133493, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 63.47008501291492, "hard": 32.35700189778908, "judge": 34.84728157786855, "standard": 49.4527089878362 } } } }, "jevbench_score": 38.379270878182446, "key": "nimble-v4", "last_measured_on": "2026-10-14", "licence": "Proprietary hosted API (no weights release)", "listing": "ranked", "measured_in": "v1.6.18", "measurement_date_status": "measured", "model_pin": "hosted-api:api.bespokelabs.ai/v1/systemone#nimble-v4", "noul_decisive": { "acc_among_decisive": 0.9627906976744186, "decisive": 215, "decisive_rate": 0.5733333333333334, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 14, "ranked": true, "ranks": { "A": 14, "B": 15, "C": 15, "capability": 18 }, "repo": "https://bespokelabs.ai", "repriced_in": "v1.6.23", "scores": { "A": 38.379270878182446, "B": 35.34254582825073, "C": 26.652271443182258 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.23237984799197875, "p50_s_raw": 0.23237984799197875, "p95_s_adjusted": 0.6222212287582807, "p95_s_raw": 0.6222212287582807 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.8636363636363636, "cc": 62.87504268080306, "cc_by_type": { "choice": 83.33333333333333, "noul": 100.0, "score": 5.291794709075859 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 62.87504268080306 }, "calibration_gen": { "accuracy": 0.5544554455445545, "cc": 36.71840534927043, "cc_by_type": { "choice": 52.27995758218452, "noul": 36.00000000000001, "score": 21.875258465626757 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 36.71840534927043 }, "composite_gen": { "accuracy": 0.6831683168316832, "cc": 38.05576935546986, "cc_by_type": { "choice": 93.4959349593496, "noul": -25.92592592592593, "score": 46.597299032985916 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 38.05576935546986 }, "fairness": { "accuracy": 0.8333333333333334, "cc": 68.76166602198055, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 72.95166473260831 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 68.76166602198055 }, "jaggedness": { "accuracy": 0.5444444444444444, "cc": 22.243715441838727, "cc_by_type": { "choice": 46.58325312800771, "noul": -29.729729729729726, "score": 49.877622927238185 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 22.243715441838727 }, "llm_router": { "accuracy": 0.7603833865814696, "cc": 62.50547658592752, "cc_by_type": { "choice": 66.71656247248393, "noul": 62.88659793814433, "score": 57.91326934715432 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 62.50547658592752 }, "long_state": { "accuracy": 0.5333333333333333, "cc": 4.871919127970649, "cc_by_type": { "choice": 61.29032258064516, "noul": 0.0, "score": -46.67456519673321 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 4.871919127970649 }, "ml_usecase": { "accuracy": 0.4074074074074074, "cc": -12.631364067994236, "cc_by_type": { "choice": 50.51546391752578, "noul": -75.0, "score": -13.409556121508492 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.6581818181818182, "cc": 36.79679245547057, "cc_by_type": { "choice": 73.58982719269645, "noul": -11.111111111111116, "score": 47.91166128482639 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 36.79679245547057 }, "negation_pair_gen": { "accuracy": 0.627906976744186, "cc": 17.910040456548206, "cc_by_type": { "choice": 93.33333333333333, "noul": -46.666666666666664, "score": 7.063454702977956 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 17.910040456548206 }, "option_order": { "accuracy": 0.9375, "cc": 92.01065246338216, "cc_by_type": { "choice": 92.01065246338216 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 92.01065246338216 }, "perturb_pair": { "accuracy": 0.78125, "cc": 71.46060305149378, "cc_by_type": { "choice": 69.96996996996997, "noul": 100.0, "score": 44.41183918451137 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 71.46060305149378 }, "usecase": { "accuracy": 0.5727272727272728, "cc": 30.241095001794505, "cc_by_type": { "choice": 62.00980392156864, "noul": -8.333333333333337, "score": 37.04681441714821 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 30.241095001794505 } }, "lang": { "ar": { "accuracy": 0.5, "cc": -0.4063065270671255, "cc_by_type": { "choice": 61.904761904761905, "noul": 0.0, "score": -63.12368148596328 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.7142857142857143, "cc": 78.94736842105263, "cc_by_type": { "choice": 57.89473684210526, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 78.94736842105263 }, "da": { "accuracy": 0.6153846153846154, "cc": 17.33676891316524, "cc_by_type": { "choice": 79.26267281105991, "noul": -33.333333333333336, "score": 6.0809672617691435 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 17.33676891316524 }, "de": { "accuracy": 0.8125, "cc": 72.84971754245485, "cc_by_type": { "choice": 76.10619469026548, "noul": 75.0, "score": 67.44295793709905 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 72.84971754245485 }, "el": { "accuracy": 0.5333333333333333, "cc": 33.173708716426326, "cc_by_type": { "choice": 66.4804469273743, "noul": -66.66666666666667, "score": 99.70734588857135 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 33.173708716426326 }, "en": { "accuracy": 0.6637706342311034, "cc": 42.06591187806426, "cc_by_type": { "choice": 68.58715417190416, "noul": 13.669064748201443, "score": 43.94151671408719 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 42.06591187806426 }, "es": { "accuracy": 0.8709677419354839, "cc": 67.31271336713421, "cc_by_type": { "choice": 90.2120717781403, "noul": 100.0, "score": 11.726068323262318 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 67.31271336713421 }, "fi": { "accuracy": 0.5, "cc": 25.707953232257534, "cc_by_type": { "choice": 67.21311475409836, "noul": -33.333333333333336, "score": 43.24407827600757 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 25.707953232257534 }, "fr": { "accuracy": 0.7857142857142857, "cc": 72.26504569240879, "cc_by_type": { "choice": 36.47058823529412, "noul": 100.0, "score": 80.32454884193226 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 72.26504569240879 }, "hi": { "accuracy": 0.5714285714285714, "cc": 22.236011151779042, "cc_by_type": { "choice": 66.4804469273743, "noul": -25.0, "score": 25.22758652796282 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 22.236011151779042 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8, "cc": 71.53128312800176, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": 28.623467466624174 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 71.53128312800176 }, "ja": { "accuracy": 0.48, "cc": 14.784765883400851, "cc_by_type": { "choice": 42.81175536139793, "noul": -19.999999999999996, "score": 21.542542288804622 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 14.784765883400851 }, "ko": { "accuracy": 0.5, "cc": 10.452983327342158, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": 31.358949982026463 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 10.452983327342158 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 77.30841577705657, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 98.59191399783639 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 77.30841577705657 }, "nl": { "accuracy": 0.4, "cc": -6.49782092272594, "cc_by_type": { "choice": 48.05194805194805, "noul": -100.0, "score": 32.45458917987413 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.0, "cc": -64.51612903225806, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.6666666666666666, "cc": 38.61167387134184, "cc_by_type": { "choice": 84.76512907321202, "noul": -19.999999999999996, "score": 51.06989254081351 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 38.61167387134184 }, "pt": { "accuracy": 0.75, "cc": 50.52489539329753, "cc_by_type": { "choice": 68.1791396582204, "noul": 19.999999999999996, "score": 63.39554652167219 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 50.52489539329753 }, "sv": { "accuracy": 0.36363636363636365, "cc": 13.221329392617008, "cc_by_type": { "choice": 23.404255319148945, "noul": -19.999999999999996, "score": 36.25973285870208 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 13.221329392617008 }, "tr": { "accuracy": 0.6923076923076923, "cc": 37.8682340701448, "cc_by_type": { "choice": 78.26086956521739, "noul": 50.0, "score": -14.65616735478299 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 37.8682340701448 }, "uk": { "accuracy": 0.8571428571428571, "cc": 80.86133582192467, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 42.584007465774 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 80.86133582192467 }, "zh": { "accuracy": 0.7, "cc": 11.860785606137426, "cc_by_type": { "choice": 100.0, "noul": -33.333333333333336, "score": -31.08430984825439 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 11.860785606137426 } }, "type": { "choice": { "accuracy": 0.7626666666666667, "cc": 69.34499320691174, "cc_by_type": { "choice": 69.34499320691174 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 69.34499320691174 }, "noul": { "accuracy": 0.552, "cc": 10.40000000000001, "cc_by_type": { "noul": 10.40000000000001 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 10.40000000000001 }, "score": { "accuracy": 0.576, "cc": 43.476768622445704, "cc_by_type": { "score": 43.476768622445704 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 43.476768622445704 } }, "usecase": { "advertising": { "accuracy": 0.45454545454545453, "cc": 2.2786515839975863, "cc_by_type": { "choice": 68.0, "noul": -75.0, "score": 13.835954751992762 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 2.2786515839975863 }, "code_linting": { "accuracy": 0.4583333333333333, "cc": 16.217833358888537, "cc_by_type": { "choice": 62.73291925465839, "noul": -11.111111111111116, "score": -2.968308066881664 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 16.217833358888537 }, "customer_support": { "accuracy": 0.6428571428571429, "cc": 30.601303418460095, "cc_by_type": { "choice": 51.21951219512195, "noul": 0.0, "score": 40.58439806025833 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 30.601303418460095 }, "demand_forecasting": { "accuracy": 0.23529411764705882, "cc": -29.26143997524909, "cc_by_type": { "choice": 6.432748538011691, "noul": -100.0, "score": 5.782931536241042 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.5294117647058824, "cc": 22.322772125960967, "cc_by_type": { "choice": 42.02898550724638, "noul": -33.333333333333336, "score": 58.27266420396986 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 22.322772125960967 }, "feature_extraction": { "accuracy": 0.3684210526315789, "cc": -22.96464494240405, "cc_by_type": { "choice": 24.528301886792455, "noul": -66.66666666666667, "score": -26.75557004733793 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5, "cc": 17.40028973466517, "cc_by_type": { "choice": 80.3921568627451, "noul": -42.85714285714286, "score": 14.665855198393274 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 17.40028973466517 }, "gaming": { "accuracy": 0.5, "cc": 2.0356283791569916, "cc_by_type": { "choice": 68.6411149825784, "noul": -100.0, "score": 37.465770154892574 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 2.0356283791569916 }, "insurance_claims": { "accuracy": 0.26666666666666666, "cc": 5.2997131864851355, "cc_by_type": { "choice": 37.5, "noul": -60.0, "score": 38.399139559455406 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 5.2997131864851355 }, "knowledge_graphs": { "accuracy": 0.5909090909090909, "cc": 41.33594739962141, "cc_by_type": { "choice": 61.4561027837259, "noul": 33.33333333333333, "score": 29.218406081805014 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 41.33594739962141 }, "lead_generation": { "accuracy": 0.6, "cc": 24.531777745700577, "cc_by_type": { "choice": 88.3720930232558, "noul": -33.333333333333336, "score": 18.556573547179266 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 24.531777745700577 }, "legal_compliance": { "accuracy": 0.6, "cc": 12.053690294805428, "cc_by_type": { "choice": 67.74193548387096, "noul": -33.333333333333336, "score": 1.7524687338786538 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 12.053690294805428 }, "llm_guardrails": { "accuracy": 0.7037037037037037, "cc": 41.77608289103801, "cc_by_type": { "choice": 88.48368522072937, "noul": -19.999999999999996, "score": 56.84456345238465 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 41.77608289103801 }, "model_routing": { "accuracy": 0.815668202764977, "cc": 68.58454886246625, "cc_by_type": { "choice": 78.96103896103895, "noul": 81.81818181818181, "score": 44.974425808178 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 68.58454886246625 }, "moderation_trust_safety": { "accuracy": 0.5555555555555556, "cc": 8.881552486989458, "cc_by_type": { "choice": 65.4510556621881, "noul": -100.0, "score": 61.193601798780286 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 8.881552486989458 }, "recruiting": { "accuracy": 0.6666666666666666, "cc": 50.38160882884504, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 51.23767699721296 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 50.38160882884504 }, "risk_assessment": { "accuracy": 0.3333333333333333, "cc": -0.19458016484627372, "cc_by_type": { "choice": 40.88669950738917, "noul": -50.0, "score": 8.529559998072012 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.5217391304347826, "cc": 13.047825366682751, "cc_by_type": { "choice": 42.02898550724638, "noul": -33.333333333333336, "score": 30.447823926135207 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 13.047825366682751 }, "search_retrieval": { "accuracy": 0.9166666666666666, "cc": 93.19059202541666, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 79.57177607624996 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 93.19059202541666 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 81.49834439906063, 87.74439164047466 ], "I": [ 36.79661812447763, 46.318323239071304 ], "capability": [ 60.26162450986514, 65.86137375483418 ], "composite": { "A": [ 28.981226595250288, 49.821188749622806 ], "B": [ 26.165454132173615, 46.72119177956524 ], "C": [ 20.12585180225714, 34.598047742793604 ] }, "seed": 37 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 86.63426567315432, "capability": 64.01968095926979, "intelligence": 41.405096245385266 }, "per_type": { "choice": { "accuracy": 0.7626666666666667, "calibration": 83.60654476958965, "cc": 69.34499320691174, "cc_by_type": { "choice": 69.34499320691174 }, "cc_tiered": 70.47912311533162, "cc_tiered_open": 74.38459954164753, "cc_tiered_sealed": 66.57364668901572, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 69.34499320691174 }, "noul": { "accuracy": 0.552, "calibration": 87.03264849596263, "cc": 10.40000000000001, "cc_by_type": { "noul": 10.40000000000001 }, "cc_tiered": 8.433206185561351, "cc_tiered_open": 13.326432022084196, "cc_tiered_sealed": 3.5399803490385073, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 10.40000000000001 }, "score": { "accuracy": 0.576, "calibration": 89.26360375391067, "cc": 43.476768622445704, "cc_by_type": { "score": 43.476768622445704 }, "cc_tiered": 45.30295943526283, "cc_tiered_open": 50.97138333919072, "cc_tiered_sealed": 39.63453553133493, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 43.476768622445704 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 34.714095863764875, "Emphasis on Accuracy 60:20:20": 31.071443227703835, "Emphasis on Cost 20:20:60": 33.36299010939205, "Emphasis on Speed 20:60:20": 41.215018376860975, "Intelligence only": 26.8459119487892 } }, { "addendum": null, "api_flag": true, "api_subset": false, "author": "Bespoke Labs", "axes": { "calibration": 91.24998737212707, "cost": 38.05267025219774, "intelligence": 68.18491883484498, "speed": 91.45091520080334 }, "calibration": { "parts": { "choice": { "ece_hard": 0.04168013967750483, "ece_score": 91.66397206449903, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 91.66397206449903 }, "noul": { "abstention_rate": 0.21599999999999997, "brier": 0.060518846834681644, "ece": 0.04479113991582226, "n": 375, "score": 91.04177201683554 }, "score": { "interval90_coverage": 0.9973333333333333, "interval90_width": 0.3084444444444445, "n": 375, "nrps": 0.04645325205782863, "score": 91.04421803504661, "top_ece": 0.06633119362061955 } }, "score": 91.24998737212707 }, "capability": 79.71745310348602, "composite_ci95": { "A": [ 36.04430568066157, 37.26919772955478 ], "B": [ 36.16295569603835, 38.05449200685788 ], "C": [ 36.04430568066157, 37.26919772955478 ] }, "cost": { "basis": "Bespoke Labs public list price, USD 0.09/M input, output free (fetched 16 Oct 2026): Bespoke's Nimble pricing page (docs.bespokelabs.ai/nimble/pricing, saved copy sha256 03b06261...) says \"Nimble charges for input tokens only. Output tokens are free.\" and lists nimble-v4-max at USD 0.09 per 1M input tokens. Counted as the manufacturer's standard, non-promotional launch list price under interpretation I-1 (the reading that admitted kushal-gemma4-31b-it-autoloops, SubQ Snap and d1-ultra), through a scoped re-freeze of the frozen pricing module that adds exactly three keys (sage-1.4.0, nimble-v4, nimble-v4-max) to its launch-list-price set and nothing else, with a zero-difference proof for every other registry entry and the independent release review; it is the separate pricing change CoS deputy decision #19084 made the cost axis wait for. Free credit for new accounts is a trial credit and is not used: the row is priced at the published tariff, never as cost 0. ESTIMATE in the scorer's sense: the tariff x this row's OWN provider-reported tokens (1,935,351 input tokens over all 1,500 answered items; the 3,000 reported output tokens are free under the tariff), divided by the full 1,500-row basis the scorer uses. Not a bill and not a quote. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.116121 per 1,000 decisions is 1.798x the official cap USD 0.064595; the row is ranked (the v1.6.16 vllm-sr-d3 and v1.6.20 SubQ Snap precedent) and falls outside the official-caps board views by construction.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.11612106 }, "display": "Bespoke Nimble-v4-max (Bespoke Labs)", "endpoint_condition": "Vendor-hosted api.bespokelabs.ai /v1/systemone API, model `nimble-v4-max`, through the unchanged TypeSafe adapter; latency measured from Sandy (Germany) on the official per-item latency subset without API adjustment. Full S1200+P300 draw of v1.6-regular-20261012 through the unchanged run_api_full_v161.py; the full-set exposure gate logged all 1,200 sealed items to provider `bespoke` before the first request (CoS deputy #18998/#19084, custodian #18999; every sealed item of this draw was already retired). Native probabilities on all three question types. Hosting implementation and deployed weights were not independently inspected. Measured as our own add after Bespoke Labs' public announcement of 13 Oct 2026 (no paid order); key on Benchmark Heaven's own account with Bespoke's free credit.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 72.20451669265262, "I_sealed": 64.16532097703733, "base": 68.18491883484498, "excess": -1.8003310547404183, "gap": 8.039195715615293, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 86.67735092678937, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 96.10810810810811, "hard": 84.90091223655237, "judge": 92.089650626236, "standard": 77.39640016743407 } }, "open|noul": { "cc": 63.948240165631475, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 85.71428571428572, "hard": 82.6086956521739, "judge": 30.000000000000004, "standard": 66.66666666666667 } }, "open|score": { "cc": 65.98795898553702, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 68.01753536790163, "hard": 72.99899229354084, "judge": 67.76239404368843, "standard": 48.28945159111995 } }, "sealed|choice": { "cc": 80.44759269360293, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 94.28208386277001, "hard": 74.49982652943217, "judge": 81.48873653281098, "standard": 83.86416367854883 } }, "sealed|noul": { "cc": 46.774787744422326, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 79.3103448275862, "hard": 43.636363636363626, "judge": 38.46153846153846, "standard": 49.25373134328359 } }, "sealed|score": { "cc": 65.27358249308672, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 79.33978396612804, "hard": 64.40573294358967, "judge": 63.511594612062325, "standard": 62.619162677096796 } } } }, "jevbench_score": 36.868911623241004, "key": "nimble-v4-max", "last_measured_on": "2026-10-14", "licence": "Proprietary hosted API (no weights release)", "listing": "ranked", "measured_in": "v1.6.18", "measurement_date_status": "measured", "model_pin": "hosted-api:api.bespokelabs.ai/v1/systemone#nimble-v4-max", "noul_decisive": { "acc_among_decisive": 0.9795918367346939, "decisive": 294, "decisive_rate": 0.784, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 15, "ranked": true, "ranks": { "A": 15, "B": 14, "C": 14, "capability": 7 }, "repo": "https://bespokelabs.ai", "repriced_in": "v1.6.23", "scores": { "A": 36.868911623241004, "B": 37.36543171468674, "C": 36.868911623241004 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.254514531508903, "p50_s_raw": 0.254514531508903, "p95_s_adjusted": 0.28131694759504167, "p95_s_raw": 0.28131694759504167 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.9545454545454546, "cc": 77.93302244321468, "cc_by_type": { "choice": 91.66666666666667, "noul": 100.0, "score": 42.13240066297739 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 77.93302244321468 }, "calibration_gen": { "accuracy": 0.693069306930693, "cc": 48.46720156444257, "cc_by_type": { "choice": 61.82396606574761, "noul": 28.000000000000004, "score": 55.57763862758007 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 48.46720156444257 }, "composite_gen": { "accuracy": 0.900990099009901, "cc": 85.97656131741992, "cc_by_type": { "choice": 93.4959349593496, "noul": 92.59259259259258, "score": 71.8411564003176 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 85.97656131741992 }, "fairness": { "accuracy": 0.9375, "cc": 85.3952829117887, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 96.18584873536608 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 85.3952829117887 }, "jaggedness": { "accuracy": 0.7444444444444445, "cc": 52.85440527377147, "cc_by_type": { "choice": 68.23869104908566, "noul": 13.513513513513509, "score": 76.81101125871524 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 52.85440527377147 }, "llm_router": { "accuracy": 0.9297124600638977, "cc": 83.04124579094008, "cc_by_type": { "choice": 94.45276041208066, "noul": 93.81443298969072, "score": 60.85654397104885 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 83.04124579094008 }, "long_state": { "accuracy": 0.6, "cc": 23.69771087111538, "cc_by_type": { "choice": 61.29032258064516, "noul": 0.0, "score": 9.802810032700982 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 23.69771087111538 }, "ml_usecase": { "accuracy": 0.5925925925925926, "cc": 18.542786785637407, "cc_by_type": { "choice": 50.51546391752578, "noul": -25.0, "score": 30.11289643938645 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 18.542786785637407 }, "multilingual": { "accuracy": 0.7345454545454545, "cc": 54.53849518745235, "cc_by_type": { "choice": 76.5242908379524, "noul": 25.0, "score": 62.091194724404616 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 54.53849518745235 }, "negation_pair_gen": { "accuracy": 0.9302325581395349, "cc": 78.91939008748496, "cc_by_type": { "choice": 100.0, "noul": 66.66666666666667, "score": 70.09150359578821 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 78.91939008748496 }, "option_order": { "accuracy": 0.9375, "cc": 92.01065246338216, "cc_by_type": { "choice": 92.01065246338216 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 92.01065246338216 }, "perturb_pair": { "accuracy": 0.890625, "cc": 82.73195100155357, "cc_by_type": { "choice": 84.98498498498498, "noul": 100.0, "score": 63.21086801967573 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 82.73195100155357 }, "usecase": { "accuracy": 0.7454545454545455, "cc": 56.032678027680525, "cc_by_type": { "choice": 74.26470588235296, "noul": 29.166666666666675, "score": 64.66666153402196 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 56.032678027680525 } }, "lang": { "ar": { "accuracy": 0.8125, "cc": 58.740127058626875, "cc_by_type": { "choice": 80.95238095238095, "noul": 100.0, "score": -4.73199977650034 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 58.740127058626875 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 0.6153846153846154, "cc": 41.85206203924473, "cc_by_type": { "choice": 79.26267281105991, "noul": 33.33333333333333, "score": 12.960179973340946 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 41.85206203924473 }, "de": { "accuracy": 0.84375, "cc": 82.27735534531733, "cc_by_type": { "choice": 68.14159292035399, "noul": 100.0, "score": 78.69047311559798 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 82.27735534531733 }, "el": { "accuracy": 0.7333333333333333, "cc": 61.060090115548206, "cc_by_type": { "choice": 83.24022346368716, "noul": 0.0, "score": 99.94004688295746 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 61.060090115548206 }, "en": { "accuracy": 0.8392701998262381, "cc": 71.15704384183026, "cc_by_type": { "choice": 83.52473120904064, "noul": 60.431654676258994, "score": 69.51474564019114 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 71.15704384183026 }, "es": { "accuracy": 0.8387096774193549, "cc": 69.66571860471659, "cc_by_type": { "choice": 90.2120717781403, "noul": 100.0, "score": 18.78508403600947 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 69.66571860471659 }, "fi": { "accuracy": 0.875, "cc": 76.57541393410838, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 96.39290846899185 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 76.57541393410838 }, "fr": { "accuracy": 0.8571428571428571, "cc": 89.21722025064541, "cc_by_type": { "choice": 78.82352941176471, "noul": 100.0, "score": 88.8281313401715 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 89.21722025064541 }, "hi": { "accuracy": 0.6666666666666666, "cc": 34.367393690812165, "cc_by_type": { "choice": 66.4804469273743, "noul": 0.0, "score": 36.62173414506219 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 34.367393690812165 }, "id": { "accuracy": 0.5, "cc": -22.222222222222225, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8666666666666667, "cc": 70.95270534511762, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": 26.887734117971718 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 70.95270534511762 }, "ja": { "accuracy": 0.6, "cc": 27.327090862326045, "cc_by_type": { "choice": 71.40587768069896, "noul": 0.0, "score": 10.575394906279168 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 27.327090862326045 }, "ko": { "accuracy": 0.5, "cc": 3.596678785557382, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0, "score": 55.23448080111659 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 3.596678785557382 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 77.40723626765454, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 98.88837546963032 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 77.40723626765454 }, "nl": { "accuracy": 0.4, "cc": -26.53328200822546, "cc_by_type": { "choice": 48.05194805194805, "noul": -100.0, "score": -27.651794076624437 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.5, "cc": 25.0, "cc_by_type": { "choice": 100.0, "noul": -50.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 25.0 }, "pl": { "accuracy": 0.8461538461538461, "cc": 73.1717555920901, "cc_by_type": { "choice": 92.382564536606, "noul": 60.00000000000001, "score": 67.13270223966425 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 73.1717555920901 }, "pt": { "accuracy": 0.75, "cc": 65.04851477203407, "cc_by_type": { "choice": 68.1791396582204, "noul": 60.00000000000001, "score": 66.96640465788181 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 65.04851477203407 }, "sv": { "accuracy": 0.5454545454545454, "cc": 40.482391585259116, "cc_by_type": { "choice": 48.93617021276595, "noul": 19.999999999999996, "score": 52.51100454301141 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 40.482391585259116 }, "tr": { "accuracy": 0.7692307692307693, "cc": 60.52275499920906, "cc_by_type": { "choice": 100.0, "noul": 50.0, "score": 31.568264997627193 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 60.52275499920906 }, "uk": { "accuracy": 0.7142857142857143, "cc": 72.19627963052231, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 81.10496792382497 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 72.19627963052231 }, "zh": { "accuracy": 0.7, "cc": 33.346706726269964, "cc_by_type": { "choice": 100.0, "noul": -33.333333333333336, "score": 33.373453512143236 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 33.346706726269964 } }, "type": { "choice": { "accuracy": 0.8613333333333333, "cc": 82.08920951415067, "cc_by_type": { "choice": 82.08920951415067 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 82.08920951415067 }, "noul": { "accuracy": 0.768, "cc": 53.6, "cc_by_type": { "noul": 53.6 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 53.6 }, "score": { "accuracy": 0.776, "cc": 66.16378952885489, "cc_by_type": { "score": 66.16378952885489 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 66.16378952885489 } }, "usecase": { "advertising": { "accuracy": 0.45454545454545453, "cc": 11.284240243103335, "cc_by_type": { "choice": 68.0, "noul": -75.0, "score": 40.85272072931001 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 11.284240243103335 }, "code_linting": { "accuracy": 0.875, "cc": 81.22933598842444, "cc_by_type": { "choice": 81.36645962732919, "noul": 100.0, "score": 62.321548337944144 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 81.22933598842444 }, "customer_support": { "accuracy": 0.6428571428571429, "cc": 21.699250411759156, "cc_by_type": { "choice": 51.21951219512195, "noul": 0.0, "score": 13.878239040155515 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 21.699250411759156 }, "demand_forecasting": { "accuracy": 0.35294117647058826, "cc": -20.13552623419365, "cc_by_type": { "choice": 18.128654970760234, "noul": -33.333333333333336, "score": -45.20190034000786 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.7647058823529411, "cc": 40.00228808667219, "cc_by_type": { "choice": 71.0144927536232, "noul": -33.333333333333336, "score": 82.32570483972668 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 40.00228808667219 }, "feature_extraction": { "accuracy": 0.7368421052631579, "cc": -25.10701220211857, "cc_by_type": { "choice": 87.42138364779873, "noul": 33.33333333333333, "score": -196.07575358748778 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.6, "cc": 31.913037340162806, "cc_by_type": { "choice": 90.19607843137256, "noul": -42.85714285714286, "score": 48.400176446258726 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 31.913037340162806 }, "gaming": { "accuracy": 0.7272727272727273, "cc": 52.57354917823071, "cc_by_type": { "choice": 100.0, "noul": 0.0, "score": 57.72064753469211 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 52.57354917823071 }, "insurance_claims": { "accuracy": 0.4666666666666667, "cc": 15.471706603552159, "cc_by_type": { "choice": 37.5, "noul": -60.0, "score": 68.91511981065648 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 15.471706603552159 }, "knowledge_graphs": { "accuracy": 0.7272727272727273, "cc": 64.00816563621142, "cc_by_type": { "choice": 61.4561027837259, "noul": 66.66666666666667, "score": 63.901727458241695 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 64.00816563621142 }, "lead_generation": { "accuracy": 0.8, "cc": 57.96643471148216, "cc_by_type": { "choice": 88.3720930232558, "noul": 33.33333333333333, "score": 52.19387777785735 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 57.96643471148216 }, "legal_compliance": { "accuracy": 0.4, "cc": 19.212456270641866, "cc_by_type": { "choice": 3.2258064516129026, "noul": 33.33333333333333, "score": 21.07822902697937 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 19.212456270641866 }, "llm_guardrails": { "accuracy": 0.8148148148148148, "cc": 55.60780315989738, "cc_by_type": { "choice": 88.48368522072937, "noul": 19.999999999999996, "score": 58.33972425896277 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 55.60780315989738 }, "model_routing": { "accuracy": 0.9078341013824884, "cc": 79.1817133930041, "cc_by_type": { "choice": 94.15584415584415, "noul": 93.93939393939394, "score": 49.4499020837742 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 79.1817133930041 }, "moderation_trust_safety": { "accuracy": 0.6666666666666666, "cc": 15.930238569976515, "cc_by_type": { "choice": 65.4510556621881, "noul": -100.0, "score": 82.33966004774146 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 15.930238569976515 }, "recruiting": { "accuracy": 0.7222222222222222, "cc": 57.68500793305302, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 73.1478743098369 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 57.68500793305302 }, "risk_assessment": { "accuracy": 0.6666666666666666, "cc": 36.52975459820843, "cc_by_type": { "choice": 85.22167487684729, "noul": 0.0, "score": 24.367588917777994 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 36.52975459820843 }, "scientific_discovery": { "accuracy": 0.4782608695652174, "cc": 6.286419441145905, "cc_by_type": { "choice": 27.536231884057973, "noul": -66.66666666666667, "score": 57.98969310604642 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 6.286419441145905 }, "search_retrieval": { "accuracy": 1.0, "cc": 99.53112849752597, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 98.59338549257794 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 99.53112849752597 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 86.97624644729329, 91.79030186697042 ], "I": [ 63.21651836088954, 72.04116776498421 ], "capability": [ 75.86375358445287, 81.18305744681227 ], "composite": { "A": [ 36.04430568066157, 37.26919772955478 ], "B": [ 36.16295569603835, 38.05449200685788 ], "C": [ 36.04430568066157, 37.26919772955478 ] }, "seed": 38 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 91.24998737212707, "capability": 79.71745310348602, "intelligence": 68.18491883484498 }, "per_type": { "choice": { "accuracy": 0.8613333333333333, "calibration": 91.66397206449903, "cc": 82.08920951415067, "cc_by_type": { "choice": 82.08920951415067 }, "cc_tiered": 83.56247181019614, "cc_tiered_open": 86.67735092678937, "cc_tiered_sealed": 80.44759269360293, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 82.08920951415067 }, "noul": { "accuracy": 0.768, "calibration": 91.04177201683554, "cc": 53.6, "cc_by_type": { "noul": 53.6 }, "cc_tiered": 55.3615139550269, "cc_tiered_open": 63.948240165631475, "cc_tiered_sealed": 46.774787744422326, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 53.6 }, "score": { "accuracy": 0.776, "calibration": 91.04421803504661, "cc": 66.16378952885489, "cc_by_type": { "score": 66.16378952885489 }, "cc_tiered": 65.63077073931187, "cc_tiered_open": 65.98795898553702, "cc_tiered_sealed": 65.27358249308672, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 66.16378952885489 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 33.49268352634633, "Emphasis on Accuracy 60:20:20": 35.65981131100044, "Emphasis on Cost 20:20:60": 27.729235265471015, "Emphasis on Speed 20:60:20": 39.268032743453034, "Intelligence only": 39.49286081411445 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "BlackdromeAILabs", "axes": { "calibration": 83.63708212673413, "cost": 61.22227091890348, "intelligence": 25.051442411072966, "speed": 93.34244428975083 }, "calibration": { "parts": { "choice": { "ece_hard": 0.07120275435939666, "ece_score": 85.75944912812066, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 85.75944912812066 }, "noul": { "abstention_rate": 0.4906666666666667, "brier": 0.17048542395103217, "ece": 0.10058815024855235, "n": 375, "score": 79.88236995028953 }, "score": { "interval90_coverage": 0.9013333333333333, "interval90_width": 0.5231111111111111, "n": 375, "nrps": 0.13612432038172226, "score": 85.26942730179223, "top_ece": 0.07924356679121652 } }, "score": 83.63708212673413 }, "capability": 54.34426226890355, "composite_ci95": { "A": [ 7.82303975781986, 17.560274631336252 ], "B": [ 6.3119492196141485, 14.918494362183155 ], "C": [ 5.4326664984860145, 12.194635160650176 ] }, "cost": { "basis": "ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module: declared base Qwen/Qwen3.5-4B-Base -> deepinfra:Qwen/Qwen3.5-4B USD 0.030000000000000002/M input, USD 0.15/M output (frozen at the 25 Sep cut-off, I-3). Estimated, not charged, not a quote from the author or any host. Frozen pricing_v15: base-model references unchanged since sha256 dcfcecb0...2835, the module this row was first priced with (the later re-freezes only add hosted-API launch-list keys), USD per 1,000 decisions over the full 1,500-item denominator (refused items cost nothing but stay in the denominator). Tokens as reported by the author's server (output_tokens 0, nothing generated).", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.019615120000000003 }, "display": "Noma (Blackdrome AI Labs, Qwen3.5-4B-Base derivative)", "endpoint_condition": "Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's own `noma serve` with the default fast path (128-token buckets + CUDA graphs).", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 29.756564202105558, "I_sealed": 20.346320620040373, "base": 25.051442411072966, "excess": -0.42928318829052614, "gap": 9.410243582065185, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 53.03396058840887, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 76.64864864864865, "hard": 41.491034916640444, "judge": 64.40342781806197, "standard": 47.25826705734617 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 42.85714285714286, "hard": -30.434782608695656, "judge": -30.000000000000004, "standard": 0.0 } }, "open|score": { "cc": 36.23573201790782, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 49.562287908823535, "hard": 33.84744031581148, "judge": 44.6284633982232, "standard": 21.759940406169587 } }, "sealed|choice": { "cc": 48.83838252279794, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 67.5984752223634, "hard": 43.79553602405458, "judge": 50.6366307541626, "standard": 46.84665682345498 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 10.344827586206895, "hard": -12.727272727272732, "judge": -38.46153846153846, "standard": 4.477611940298498 } }, "sealed|score": { "cc": 12.200579337323186, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 26.122391760234898, "hard": 11.720567435668894, "judge": 1.0388793398912077, "standard": 22.942246925323882 } } } }, "jevbench_score": 12.723026319141203, "key": "noma", "last_measured_on": "2026-10-16", "licence": "MPL-2.0 (weights and code)", "listing": "ranked", "measured_in": "v1.6.24", "measurement_date_status": "measured", "model_pin": "BlackdromeAILabs/noma@fac0429688a137ce451a27ad34d05e9fd47fb624", "noul_decisive": { "acc_among_decisive": 0.8848167539267016, "decisive": 191, "decisive_rate": 0.5093333333333333, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 24, "ranked": true, "ranks": { "A": 24, "B": 25, "C": 25, "capability": 28 }, "repo": "https://huggingface.co/BlackdromeAILabs/noma", "scores": { "A": 12.723026319141203, "B": 10.561737274193275, "C": 8.835434943848057 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.2009935579902958, "p50_s_raw": 0.025496778995147906, "p95_s_adjusted": 0.23044825975957792, "p95_s_raw": 0.04022412987978896 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.45454545454545453, "cc": 17.41950772702694, "cc_by_type": { "choice": 25.0, "noul": 50.0, "score": -22.741476818919182 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 17.41950772702694 }, "calibration_gen": { "accuracy": 0.2871287128712871, "cc": -16.632615001550132, "cc_by_type": { "choice": 14.103923647932142, "noul": -43.99999999999999, "score": -20.00176865258254 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.5247524752475248, "cc": 7.918370014291946, "cc_by_type": { "choice": 67.47967479674797, "noul": -18.518518518518523, "score": -25.206046235353607 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 7.918370014291946 }, "fairness": { "accuracy": 0.75, "cc": 55.05881191793984, "cc_by_type": { "choice": 100.0, "noul": 19.999999999999996, "score": 45.17643575381953 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 55.05881191793984 }, "jaggedness": { "accuracy": 0.4388888888888889, "cc": -1.015847243318935, "cc_by_type": { "choice": 33.58999037536093, "noul": -40.54054054054054, "score": 3.90300843522281 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.7188498402555911, "cc": 52.54172250879653, "cc_by_type": { "choice": 65.13163687593556, "noul": 46.39175257731958, "score": 46.10177807313447 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 52.54172250879653 }, "long_state": { "accuracy": 0.13333333333333333, "cc": -29.786333031651964, "cc_by_type": { "choice": -29.032258064516128, "noul": 0.0, "score": -60.326741030439756 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.2962962962962963, "cc": -27.07773849436275, "cc_by_type": { "choice": 13.402061855670107, "noul": -100.0, "score": 5.364722661241627 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.5163636363636364, "cc": 17.772590134559422, "cc_by_type": { "choice": 49.135963482230196, "noul": -30.555555555555557, "score": 34.73736247700363 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 17.772590134559422 }, "negation_pair_gen": { "accuracy": 0.4186046511627907, "cc": -14.28969892052313, "cc_by_type": { "choice": 43.33333333333332, "noul": -53.333333333333336, "score": -32.869096761569374 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.0 }, "option_order": { "accuracy": 0.7708333333333334, "cc": 70.70572569906791, "cc_by_type": { "choice": 70.70572569906791 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 70.70572569906791 }, "perturb_pair": { "accuracy": 0.765625, "cc": 70.18476184672892, "cc_by_type": { "choice": 63.96396396396397, "noul": 80.0, "score": 66.59032157622278 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 70.18476184672892 }, "usecase": { "accuracy": 0.4409090909090909, "cc": 10.091614151059419, "cc_by_type": { "choice": 44.85294117647061, "noul": -37.5, "score": 22.921901276707647 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 10.091614151059419 } }, "lang": { "ar": { "accuracy": 0.25, "cc": -27.048739826395394, "cc_by_type": { "choice": -14.285714285714286, "noul": -50.0, "score": -16.86050519347191 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.2857142857142857, "cc": 47.368421052631575, "cc_by_type": { "choice": -5.263157894736845, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 47.368421052631575 }, "da": { "accuracy": 0.6153846153846154, "cc": 10.925357399486423, "cc_by_type": { "choice": 79.26267281105991, "noul": -33.333333333333336, "score": -13.153267279267311 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 10.925357399486423 }, "de": { "accuracy": 0.59375, "cc": 30.560566461628643, "cc_by_type": { "choice": 60.176991150442475, "noul": 0.0, "score": 31.504708234443456 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 30.560566461628643 }, "el": { "accuracy": 0.6, "cc": 44.32658194414518, "cc_by_type": { "choice": 66.4804469273743, "noul": -33.333333333333336, "score": 99.83263223839458 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 44.32658194414518 }, "en": { "accuracy": 0.5369244135534318, "cc": 19.628069056743954, "cc_by_type": { "choice": 49.69551262493742, "noul": -5.755395683453235, "score": 14.944090228747676 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 19.628069056743954 }, "es": { "accuracy": 0.7419354838709677, "cc": 57.38676236262902, "cc_by_type": { "choice": 80.42414355628058, "noul": 75.0, "score": 16.736143531606473 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 57.38676236262902 }, "fi": { "accuracy": 0.375, "cc": -5.7500144985855615, "cc_by_type": { "choice": 67.21311475409836, "noul": -100.0, "score": 15.536841750144958 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 0.0 }, "fr": { "accuracy": 0.6428571428571429, "cc": 37.1638514750788, "cc_by_type": { "choice": 57.64705882352941, "noul": 0.0, "score": 53.84449560170699 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 37.1638514750788 }, "hi": { "accuracy": 0.47619047619047616, "cc": 1.2012258532770967, "cc_by_type": { "choice": 49.720670391061454, "noul": -75.0, "score": 28.883007168769836 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 1.2012258532770967 }, "id": { "accuracy": 0.5, "cc": -22.222222222222225, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8666666666666667, "cc": 78.87962593695318, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": 50.66849589347838 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 78.87962593695318 }, "ja": { "accuracy": 0.56, "cc": 24.526026775767367, "cc_by_type": { "choice": 42.81175536139793, "noul": 19.999999999999996, "score": 10.766324965904172 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 24.526026775767367 }, "ko": { "accuracy": 0.5, "cc": 3.947561358412102, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": 11.84268407523631 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 3.947561358412102 }, "mixed": { "accuracy": 0.6428571428571429, "cc": 55.142380630907915, "cc_by_type": { "choice": 83.33333333333333, "noul": 33.33333333333333, "score": 48.760475226057075 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 55.142380630907915 }, "nl": { "accuracy": 0.4, "cc": -2.1042632801837207, "cc_by_type": { "choice": 22.077922077922082, "noul": -100.0, "score": 71.60928808152676 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.16666666666666666, "cc": -32.25806451612903, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.5641025641025641, "cc": 18.96793208728958, "cc_by_type": { "choice": 77.14769360981802, "noul": -60.0, "score": 39.756102652050714 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 18.96793208728958 }, "pt": { "accuracy": 0.5, "cc": 21.53686440182103, "cc_by_type": { "choice": 36.35827931644077, "noul": -19.999999999999996, "score": 48.25231388902231 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 21.53686440182103 }, "sv": { "accuracy": 0.18181818181818182, "cc": 0.8117545513000266, "cc_by_type": { "choice": -2.127659574468081, "noul": -60.0, "score": 64.56292322836816 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 0.8117545513000266 }, "tr": { "accuracy": 0.3076923076923077, "cc": -8.704425914860943, "cc_by_type": { "choice": -8.695652173913041, "noul": 0.0, "score": -17.41762557066979 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 0.0 }, "uk": { "accuracy": 0.5714285714285714, "cc": 62.31405497077973, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 51.458293944597244 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 62.31405497077973 }, "zh": { "accuracy": 0.4, "cc": -15.528851182756002, "cc_by_type": { "choice": 23.72881355932204, "noul": -33.333333333333336, "score": -36.982033774256706 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 } }, "type": { "choice": { "accuracy": 0.6133333333333333, "cc": 50.0564496067663, "cc_by_type": { "choice": 50.0564496067663 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 50.0564496067663 }, "noul": { "accuracy": 0.45066666666666666, "cc": -9.866666666666667, "cc_by_type": { "noul": -9.866666666666667 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.464, "cc": 18.86027982285219, "cc_by_type": { "score": 18.86027982285219 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 18.86027982285219 } }, "usecase": { "advertising": { "accuracy": 0.22727272727272727, "cc": -37.645864323073745, "cc_by_type": { "choice": 20.0, "noul": -75.0, "score": -57.93759296922125 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.4583333333333333, "cc": 8.8939425965539, "cc_by_type": { "choice": 44.099378881987576, "noul": -33.333333333333336, "score": 15.91578224100746 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 8.8939425965539 }, "customer_support": { "accuracy": 0.6428571428571429, "cc": 36.874031663183274, "cc_by_type": { "choice": 51.21951219512195, "noul": 0.0, "score": 59.40258279442787 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 36.874031663183274 }, "demand_forecasting": { "accuracy": 0.17647058823529413, "cc": -22.785404404592622, "cc_by_type": { "choice": -5.263157894736844, "noul": -100.0, "score": 36.906944680958986 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.35294117647058826, "cc": -18.60576964001473, "cc_by_type": { "choice": 42.02898550724638, "noul": -100.0, "score": 2.153705572709419 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 0.0 }, "feature_extraction": { "accuracy": 0.2631578947368421, "cc": -32.66952881898021, "cc_by_type": { "choice": -0.62893081761006, "noul": -66.66666666666667, "score": -30.7129889726639 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.4666666666666667, "cc": 10.016228768867249, "cc_by_type": { "choice": 70.58823529411765, "noul": -71.42857142857143, "score": 30.889022441055523 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 10.016228768867249 }, "gaming": { "accuracy": 0.2727272727272727, "cc": -20.483787632035405, "cc_by_type": { "choice": 26.82926829268293, "noul": -100.0, "score": 11.719368811210861 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.4666666666666667, "cc": 19.43280212374197, "cc_by_type": { "choice": 58.33333333333333, "noul": -60.0, "score": 59.96507303789258 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 19.43280212374197 }, "knowledge_graphs": { "accuracy": 0.22727272727272727, "cc": -5.167247968174982, "cc_by_type": { "choice": 10.064239828693788, "noul": -33.333333333333336, "score": 7.767349600114604 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.44, "cc": 14.161840905735474, "cc_by_type": { "choice": 76.74418604651163, "noul": -55.55555555555556, "score": 21.296892226250353 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 14.161840905735474 }, "legal_compliance": { "accuracy": 0.1, "cc": -45.33045811238148, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -6.959116272628307 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.5925925925925926, "cc": 24.176631357497374, "cc_by_type": { "choice": 53.93474088291747, "noul": -19.999999999999996, "score": 38.59515318957466 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 24.176631357497374 }, "model_routing": { "accuracy": 0.7188940092165899, "cc": 49.58116367259822, "cc_by_type": { "choice": 71.94805194805195, "noul": 45.45454545454546, "score": 31.340893615197263 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 49.58116367259822 }, "moderation_trust_safety": { "accuracy": 0.5, "cc": -5.966226657212023, "cc_by_type": { "choice": 30.9021113243762, "noul": -100.0, "score": 51.19920870398773 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 0.0 }, "recruiting": { "accuracy": 0.6111111111111112, "cc": 44.43452582725566, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 33.396427992444785 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 44.43452582725566 }, "risk_assessment": { "accuracy": 0.4444444444444444, "cc": -2.43732824862776, "cc_by_type": { "choice": 85.22167487684729, "noul": -100.0, "score": 7.4663403772694314 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.30434782608695654, "cc": -9.883258988665188, "cc_by_type": { "choice": 13.04347826086957, "noul": -66.66666666666667, "score": 23.973411439801538 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.9166666666666666, "cc": 86.81002782020666, "cc_by_type": { "choice": 89.3238434163701, "noul": 100.0, "score": 71.1062400442499 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 86.81002782020666 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 76.99516701638751, 85.01302086028254 ], "I": [ 20.725005076795032, 28.554497166994903 ], "capability": [ 50.00305458586021, 55.86429713070938 ], "composite": { "A": [ 7.82303975781986, 17.560274631336252 ], "B": [ 6.3119492196141485, 14.918494362183155 ], "C": [ 5.4326664984860145, 12.194635160650176 ] }, "seed": 39 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 83.63708212673413, "capability": 54.34426226890355, "intelligence": 25.051442411072966 }, "per_type": { "choice": { "accuracy": 0.6133333333333333, "calibration": 85.75944912812066, "cc": 50.0564496067663, "cc_by_type": { "choice": 50.0564496067663 }, "cc_tiered": 50.93617155560341, "cc_tiered_open": 53.03396058840887, "cc_tiered_sealed": 48.83838252279794, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 50.0564496067663 }, "noul": { "accuracy": 0.45066666666666666, "calibration": 79.88236995028953, "cc": -9.866666666666667, "cc_by_type": { "noul": -9.866666666666667 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.464, "calibration": 85.26942730179223, "cc": 18.86027982285219, "cc_by_type": { "score": 18.86027982285219 }, "cc_tiered": 24.218155677615503, "cc_tiered_open": 36.23573201790782, "cc_tiered_sealed": 12.200579337323186, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 18.86027982285219 } }, "raw_on_A": { "calibration": 81.23569238761198, "capability": 52.611491162690655, "intelligence": 23.987289937769326 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 11.246015435188836, "Emphasis on Accuracy 60:20:20": 8.550021337829586, "Emphasis on Cost 20:20:60": 12.597744557721855, "Emphasis on Speed 20:60:20": 14.199888514101808, "Intelligence only": 6.2886612524076995 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Phocinae / NovelNexus AI (novelnexusai@outlook.com)", "axes": { "calibration": 56.87058116402116, "cost": 95.25096508672264, "intelligence": 3.609767391262904, "speed": 77.11764018052709 }, "calibration": { "parts": { "choice": { "ece_hard": 0.06613968253968247, "ece_score": 86.7720634920635, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 86.7720634920635 }, "noul": { "abstention_rate": 0.9706666666666667, "brier": 0.24743832458666667, "ece": 0.08080160000000003, "n": 375, "score": 83.83967999999999 } }, "score": 56.87058116402116 }, "capability": 30.240174277642033, "composite_ci95": { "A": [ 0.00807108229530911, 0.22732855865413176 ], "B": [ 0.0052235173050397266, 0.15690224067440833 ], "C": [ 0.005604918260631327, 0.15786705462092482 ] }, "cost": { "basis": "ESTIMATE (*est.), size-class reference; self-hosted open weights, no public bookable tariff. A 144.3M ModernBERT encoder with no decoder lineage, so the base-size-encoder line is used unchanged - USD 0.005 per 1M input and 0.0 per 1M output, the line the PUBLISHED v1.6 field already carries on modernbert-jev-v2, tacet-sonata and watt-flash-0.1 (verified in their own registry entries; modernbert-jev-v2 was measured in v1.6.8 on the earlier v1.6-regular-20261010-a2 draw, while tacet-sonata and watt-flash-0.1 are v1.6.1 rows - none of the three was measured on THIS draw, which carries no ModernBERT row), backed by the dated 25 Sep 2026 DeepInfra snapshot sha256 84eb7926acd4e93b71d695be24898e2ca999b20011d1b66cc0348ce7c6c3546f in which thenlper/gte-base, BAAI/bge-base-en-v1.5, intfloat/e5-base-v2 and sentence-transformers/all-mpnet-base-v2 all price at USD 0.005 per 1M input and 0 output - and no new release-lane pricing line is created. Output tokens are a literal 0 reported by the author's own server (an encoder read-out; nothing is generated). x this row's own measured input tokens, which are the author's server's own count. Estimated, not charged.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.0014397966666666667 }, "display": "Phocinae-Largha-150M-v1", "endpoint_condition": "Sandy, the evaluator's own shared CPU host (AMD Ryzen 5 3600, 6 cores / 12 threads, Zen 2, AVX2), NOT a dedicated machine: other jobs ran beside it and the 1-minute load average was logged every 60 s into logs/CONDITIONS-.jsonl - for this row's window 3.75 to 9.89, mean 5.81 over 12 samples, the quietest of the three new rows. The author's own phocinae-server 0.1.5 from PyPI, UNMODIFIED, run with their own documented line `PHOC_MODEL_DIR= python -m phocinae.main` at PHOC_DEVICE=cpu, PHOC_THREADS=1 (their own documented CPU setting: 1.64 s per 5-decision case single-threaded), PHOC_EXTENSIONS=1 for their native option_scores, PHOC_PERM_AVG at its default 0 and no bearer token, inside a loopback-only network namespace with an outbound probe that failed as required and no credentials in the environment. Their package omits numpy from its declared dependencies although their own engine.py calls .numpy(); numpy 2.5.4 was installed so their code could run, and no file of theirs was edited. OUR shim (r107/src/phoc_shim_r107.py, byte-identical to the already source-reviewed r92/src/phoc_shim_r92.py, sha256 631cb60dde86b8e609382a32900db5e4ddc9fa9b2fef39bdb7b022dff558775e) is the only translation layer in front of it, because their published API carries no question text - see the mapping disclosure. Speed is a loaded-shared-CPU figure; the standard self-hosted x2 + 0.15 s adjustment is applied on top and the raw p50/p95 are kept. Re-measured for this release on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720077ef7c41ace367524896507c7f44ce71306fb3e659e8b85069d995b) under CoS deputy decision #18209, at USD 0 on Sandy’s own CPU, 1 thread, 2026-10-13 05:59:36Z–06:11:07Z, 1-minute load average 3.75–9.89 (mean 5.81, 12 samples). DISCLOSED MAPPING: choice options are sent as \"