{ "G_med": 9.839526770355711, "G_med_api_basis_P_vs_A": 9.158662835848954, "G_med_flag_gt10": false, "benchmark": "JevBench", "board": { "A": { "markers": [], "order": [ "torchcast-decision-4b-v1.1", "attest-31b-v1.0", "metask_jev_rain_12b_trained", "metask_jev_rain_31b_trained", "uniphore-4b-decider", "standard_one_3b_sh", "krino-2b", "phocinae-largha-150m-v1", "julia-1" ] }, "B": { "markers": [], "order": [ "metask_jev_rain_12b_trained", "attest-31b-v1.0", "metask_jev_rain_31b_trained", "torchcast-decision-4b-v1.1", "uniphore-4b-decider", "standard_one_3b_sh", "krino-2b", "phocinae-largha-150m-v1", "julia-1" ] }, "C": { "markers": [], "order": [ "attest-31b-v1.0", "metask_jev_rain_12b_trained", "metask_jev_rain_31b_trained", "torchcast-decision-4b-v1.1", "uniphore-4b-decider", "standard_one_3b_sh", "krino-2b", "phocinae-largha-150m-v1", "julia-1" ] }, "capability": { "order": [ "metask_jev_rain_31b_trained", "metask_jev_rain_12b_trained", "attest-31b-v1.0", "torchcast-decision-4b-v1.1", "uniphore-4b-decider", "standard_one_3b_sh", "krino-2b", "phocinae-largha-150m-v1", "julia-1" ] } }, "bootstrap": { "B": 1000, "bootstrap_seed": 16, "g_med_fixed": true }, "headline": "A", "label": "JevBench v1.6.15 ยท three self-hosted CPU systems join the seven official rows of v1.6.14; one of the three is listed but not ranked, and the unranked A2 measurement is carried", "measurement_notes": [ "Refusals, errors and context-bound failures stay in the full 1,500 denominator. Every row kept its own package, model, runtime and context settings; nothing was re-run for presentation.", "NOT DIRECTLY COMPARABLE WITH THE LIVE v1.6.1 BOARD OR WITH THE EARLIER FRESH DRAWS. This release is a different draw with its own small-field median: G_med = 9.839526770355711, frozen from the four v1.6.10 members (attest-31B-v1.0 does not move it), against 2.573387642438244 on the live field. G_med_flag_gt10 is false. The gap penalty is max(0, 1 - max(0, (gap - G_med) - 8)/100), so this field's threshold is G_med + 8 = 17.84. On the merged live board this draw is placed with offset 0 while its anchor runs are pending, exactly like the earlier fresh draws. Measured, not estimated: Applying the same penalty formula to the stored base and gap at the live field's median 2.573387642438244, attest-31B-v1.0 Intelligence 65.238 would be 65.238 (gap -0.628); the original four rows are as stated in the v1.6.10 notes: standard_one_3b_sh Intelligence 32.786 would be 31.294 and metask_jev_rain_12b_trained 67.826 would be 67.587; krino-2b and metask_jev_rain_31b_trained are unchanged.", "Cost: no item is excluded from the cost basis on this draw. Every row is priced from its own measured usage block at a clearly-labelled ESTIMATE rate fixed before scoring (r99, r101, r103 and r106 pre-registrations); none is a bill any of these systems issued. Every row is under the published v1.6 cost ceiling of USD 0.0646 per 1,000 decisions. The cost normalizer C0 is USD 0.001 per 1,000 decisions with 30 cost points per decade.", "No subject-topic radar in this release: this draw's items have not been topic-labelled yet, and the shared topic labels of earlier draws do not cover its items. Families, use cases and languages are authoring metadata and are shown.", "v1.6.13 ADDS ONE ROW, attest-31B-v1.0, to the six of v1.6.12 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so all six earlier rows keep identical point scores, axes and capability; ranks and bootstrap confidence intervals are recomputed for the seven-row roster, and the existing raw_on_A and API-basis offset diagnostics are unchanged because the new row is full-set API, not a new self-hosted equating-pool member. No API offset enters its score.", "krino-2b: All 1,500 items answered HTTP 200 with native probabilities. The 15 longest items of the draw (the same 15 that standard_one_3b_sh refuses) ran with the state truncated to 65,536 tokens, as the author's submitted serve command documents (KEV_TRUNCATE_STATES=1); every other item is at most 1,802 input tokens. Reference kernels were used (KEV_FUSED=0) because the documented pip install has no flash-linear-attention. Kev reports usage.output_tokens as the token count of the serialized answer JSON although nothing is generated (165,276 over 1,500 decisions); it is priced as reported at the frozen DeepInfra Qwen/Qwen3.5-2B reference, USD 0.02/0.10 per 1M. Measured on an evaluator-owned Lium H100 pod in an offline container (network none).", "metask_jev_rain_12b_trained: All 1,500 items answered HTTP 200. The author's service runs the authors' own int4 (W4A16) quantization of Google's Gemma 4 12B QAT checkpoint with their own rank-16 LoRA applied unmerged by vLLM; per its model card it takes a second pass on part of the items, so it reports generated output tokens (84,238 over 1,500 decisions) and its latency has a long tail (raw p95 8.1 s against a raw p50 of 0.13 s). Measured on an evaluator-owned Lium H100 80GB pod (IP geolocated to Nizhny Novgorod, Russia) in an offline container (network none). google/gemma-4-12b-it is not listed on OpenRouter, so the cost uses the nearest listed sibling, google/gemma-3-12b-it at USD 0.05/0.15 per 1M, times its own usage. It is a different system from the earlier metask_jev_rain_12b row.", "metask_jev_rain_31b_trained: All 1,500 items answered HTTP 200. The author's service runs Google's unchanged W4A16 Gemma 4 31B QAT checkpoint with the authors' own rank-16 LoRA applied unmerged by vLLM, at max_model_len 6,656 (per the measuring operator's handoff): longer inputs are shortened by the service's own documented extraction rule rather than refused, so the longest items were answered on a shortened state. The authors tested it on H200; it was measured on an evaluator-owned Lium H100 80GB pod (IP geolocated to Nizhny Novgorod, Russia) in an offline container (network none) with unchanged settings. The service reports 1 output token per decision; cost is the OpenRouter google/gemma-4-31b-it list price (USD 0.09/0.34 per 1M) times its own usage.", "standard_one_3b_sh: 15 of 1,500 items returned HTTP 422: they are the 15 longest items of the draw and exceed the model's documented 32,768-token context, so the author's server refuses them rather than truncating; scored wrong per method. Measured on an evaluator-owned Lium H100 pod in an offline container (network none). Priced at the M2 base-model floor, OpenRouter mistralai/ministral-3b-2512 at USD 0.10/0.10 per 1M, because the author's own list price (USD 0.038 per 1M input) is below that floor; the schema head generates no text, so output tokens are 0.", "torchcast-decision-4b-v1.1: evaluator-owned Lium GPU pod (H100), offline read-only container (--network none); author's torchcast_decision server unchanged at tag v1.1.0 with the bundled decision_config.json (bf16, one forward pass reading option letters, calibration temperature 0.9, yes/no commit floor 0.801) on the documented stack torch 2.9.1+cu128 / transformers 5.8.1 / flash-linear-attention 0.5.2 / causal-conv1d 1.7.0 (fast kernels active, CUDA graphs on) Paid fast-lane order 529be5d5 (public). Qwen3.5-4B fine-tune served by the author's own torchcast_decision package. Runtime: official pytorch/pytorch:2.9.1-cuda12.8-cudnn9-devel image (digest-pinned) plus an offline hash-locked overlay of the documented packages (Python 3.11 instead of the documented 3.12). decision_config.json sets a disclosed yes/no commit floor of 0.801: a yes/no probability with max(P, 1-P) below 0.801 is moved to 0.801 on its own side (answer unchanged; same mechanism as h2o-lightning-4b). Prompts over 16,384 tokens keep the head and the tail of the state (documented). Output tokens are 0 (no generation). CC BY-NC 4.0 (weights); base Qwen/Qwen3.5-4B Apache-2.0; serving package Apache-2.0. ESTIMATE, v1.5-M2 base-model reference rule: self-hosted open weights, no bookable tariff. Base model Qwen/Qwen3.5-4B (model card base_model). OpenRouter does not list Qwen3.5-4B (host snapshot source/public-docs/OPENROUTER-MODELS.json, fetched 2026-10-12T08:15:26Z, read 2026-10-12). Binding precedent INTERPRETATION-1 I-2/I-3: DeepInfra Qwen/Qwen3.5-4B, USD 0.03/M input, USD 0.15/M output, frozen 25 Sep 2026 price cut-off snapshot (listing later deprecated; frozen reference kept). Logit readout, no generation: server reports 0 output tokens. Isolation is container-level (--network none); no separate host-level netcut probe was taken. The input tar was on the rented pod host disk during the run, and the pod was destroyed after measurement (CUSTODY-RECEIPT.json).", "uniphore-4b-decider: 15 of 1,500 inputs exceed the author's documented 16,384-token wrapper limit; the author's code refuses them (HTTP 422, no truncation) and they are scored as wrong per method, so 1,485 items were answered. The native readout is one forward pass, so output tokens are 0 and the cost is the input tokens at the frozen DeepInfra Qwen/Qwen3.5-4B reference, USD 0.03/0.15 per 1M (that listing is flagged deprecated in the frozen snapshot but remains the reference). Measured on an evaluator-owned Lium H100 pod in an offline container (network none), in the vllm/vllm-openai v0.31.0 image with transformers 5.17 and reference PyTorch kernels in place of the documented flash-linear-attention / causal-conv1d kernels, so its latency was measured on the fallback path. The author discloses that the public JevBench set was inspected repeatedly during development.", "attest-31b-v1.0: Paid public fast-lane order 07d13315. Vendor-hosted attest.engineering /v1/systemone API; latency measured from Sandy (Germany), using the official per-item latency subset without API adjustment. Full S1200+P300 draw, not API A300 supplement; no API equating. 15 of 1,500 requests returned HTTP 500 after 8 retry invocations and are scored as wrong in the full denominator. Public model documentation declares frozen google/gemma-4-31B-it @842da3794eaa0b77d5f08bae87a17459d91ff475, unmodified bf16, no adapter or quantization; single forward pass reading option-label logits, no generated tokens, one fitted temperature (value not documented). Hosting implementation and deployed weights were not independently inspected. Hosted API access is not an open-weights release by the vendor; base weights are under the Gemma 4 licence. ESTIMATE, v1.5-M2 base-model reference rule: vendor-hosted API, customer list price USD 0.048/M input and USD 0.00/M output effective 2026-10-11 has been in force fewer than 30 days. Base model google/gemma-4-31b-it reference on OpenRouter USD 0.09/M input / USD 0.34/M output, snapshot fetched 2026-10-12T17:55:05Z (SHA-256 81eae2f3f94bf299c8d4499260f842c357b7a869ce82dfa1741ad6a89cf57014), same base-model estimate precedent as metask_jev_rain_31b_trained. Applied to measured response usage; failed HTTP 500 usage/billing may be unreported. No output generation; reported output tokens 0. Estimated, not a vendor bill.", "Microsoft A2 re-run: 300 sealed supplementary A2 + 300 public items, not the full S1200 set. Unranked measurement; the official Microsoft row is retained on the merged boards. Equating pool four of six: Krino, Standard One, Uniphore, MeTask 12B. MeTask 31B and Torchcast were not completed after four unsuccessful allocation attempts (price/creation/controller failures); no inferred scores for missing pool systems. Seven HTTP 422 responses are wrong in the 600-item denominator. Pool re-runs used H200 in Helsinki, versus H100 parents; wide equating confidence intervals are reported. Latency from Sandy (Germany). Cost is the Azure Foundry tariff USD 0.042/M input, USD 0/M output; invoice/warmup/refusal charges unknown. No category cells or ranks are assigned to the A2 measurement.", "v1.6.15 ADDS THREE SELF-HOSTED CPU ROWS to the seven official rows of v1.6.14 on the same draw, under the same rule as the earlier same-draw additions: the field median stays the frozen v1.6.10 value, so all seven earlier rows keep identical point scores, axes, capability and cost objects. Ranks and bootstrap confidence intervals are recomputed over the new roster - the scorer draws ONE paired resample for the whole field, so adding rows moves a carried row's confidence interval while no point estimate changes; six of the eight carried rows' intervals moved (attest-31b-v1.0 and the carried A2 measurement did not). One DIAGNOSTIC also moved, and it is named here rather than left to be found: the API-basis median diagnostic is computed over the ranked field, so it moved from 9.628699345543296 to 9.158662835848954 when three rows joined, and that moves the stored A-basis diagnostic (the row field v16.raw_on_A) of exactly one carried row, standard_one_3b_sh: intelligence 29.3339 -> 29.1921, capability 53.1539 -> 53.0830. That diagnostic is not part of any score, rank or axis on this release, and no carried row's composite, axis, capability or cost object changes. The unranked A2 measurement of v1.6.14 is carried through byte for byte and is not rescored. reflex-instinct-0.6b is LISTED BUT NOT RANKED: CoS deputy decision #18761 accepts its Intelligence, Calibration and Cost only and declares its Speed figure non-official, and the composite weights Speed. All three new rows were measured at USD 0 on the evaluator's own shared CPU host, so their Speed figures are loaded-host lower bounds; each row logs its own one-minute load trace and carries it in its note.", "Where to look for the row that is shown but not ranked, and what it does NOT mean. reflex-instinct-0.6b is deliberately absent from the ranking charts, the Composite and Capability boards, the compare view and the category radars, because those are ranked views and this row has no rank. It is named in this release's own notes and header, and it appears in the collapsed \"All data\" table further up the page, after the Noul table. DISCLOSED: in that table the row still shows its composite as the scorer computed it, and a Jev-class verdict the site derives from its cost per decision and its MEDIAN LATENCY against the frozen Jev-class caps. The composite weights the Speed axis, and that latency is the very measurement CoS deputy decision #18761 declares non-official for this row, so neither figure is an official placement. They are not an official placement and must not be quoted as one. Its accepted figures are Intelligence 3.960, Calibration 73.153, Capability 38.556 and Cost USD 0.00630127 per 1,000 decisions. A dedicated row group for listed-not-ranked rows, with Speed greyed out and the composite hidden, is the right presentation and is a site change this release does not make.", "Cost on the three new rows: each is priced from its own measured usage block at a clearly labelled ESTIMATE rate fixed before scoring, on the full 1,500-row basis, and each is far under the published v1.6 ceiling of USD 0.0646 per 1,000 decisions - phocinae-largha-150m-v1 USD 0.00143980, reflex-instinct-0.6b USD 0.00630127, julia-1 USD 0.00217445. None is a bill any of these systems issued. julia-1's declared base jhu-clsp/mmBERT-small has no market listing; adding it to the frozen pricing module's reviewed unlisted-base set was authorised by #18761, re-frozen, hashed, proved behaviour-identical over every registry entry on the evaluator's machine and independently reviewed before this row was scored.", "julia-1: 1,500 attempted, 1,483 answered. The 17 misses are the author's own strict_encoding policy refusing rather than truncating (15 items above their lossless context budget, 2 with an option above their 48-token option contract); the totals AND the per-type split were pre-registered before the run and came back exact. All answered items native; the Speed figures are the published method's percentiles over this draw's open standard and judge items. 652,334 input tokens and 0 output tokens by construction (one encoder forward pass read at marker positions, nothing generated). The author ships no server, so the HTTP surface is the evaluator's - but their own julia.typed.predict_typed already speaks JevBench's typed-question shape, so no mapping decision of the evaluator's enters this row. Their hierarchical Router, which their own card says is not a global probability distribution, is neither used nor reachable here: every choice item of this draw has 3 to 9 options, inside their native 2-20 contract, so each decision is one call with one global softmax, and the probabilities are their raw values rather than their presentation helper's. Measured on the evaluator's own shared CPU host at their own settings read back from the run itself (float32, 4 threads, torch backend); one-minute load 10.87-13.78 logged, so Speed is a loaded-shared-CPU lower bound with the standard self-hosted x2 + 0.15 s adjustment on top. Cost is a clearly labelled size-class ESTIMATE: the declared base jhu-clsp/mmBERT-small has no market listing, and the base-size-encoder line of USD 0.005 per 1M input / 0 output that modernbert-jev-v2, tacet-sonata and watt-flash-0.1 carry in the PUBLISHED v1.6 field is used unchanged - no row of this draw carries it - and it is not a bill.", "phocinae-largha-150m-v1: 1,500 attempted, 1,125 answered. The 375 score items are recorded as HTTP 422 misses by the EVALUATOR'S shim, under a rule pre-committed in writing before the run (PREREG-R92 section 4.6), because the author's API answers only a fixed nine-level 2..10 scale that does not fit this item type; the shim refuses them in about 0.7 ms without calling the author's server at all, so no claim is made that the author's code refused anything. They stay in the full 1,500 denominator as misses - the figure 1,125 was pre-registered and came back exact. All answered items native. The Speed figures on this row are the published method's percentiles over this draw's open standard and judge items, not a median over every attempt - and DISCLOSED: 37 of those 124 items are the shim's instant score refusals, which cost no model time, so the figures are faster than the model itself is. Over the 87 items the model actually answered the Speed axis would read 75.75 instead of 77.12, which changes no rank. Measured on the evaluator's own shared CPU host (AMD Zen 2, float32), NOT a dedicated machine: other jobs ran beside it and the one-minute load average was logged for the whole window (3.75 to 9.89, mean 5.81, 12 samples - the quietest of this release's three new rows). Speed is therefore a loaded-shared-CPU figure and a lower bound for this model on a dedicated core or a GPU; the standard self-hosted x2 + 0.15 s adjustment is applied on top and the raw percentiles are kept. DISCLOSED: the author's server places every short-option request in its own 512-token budget tier, so the state is truncated far below the 8,192 its MAX_POS advertises - the author's design, not the evaluator's, and visible in their own usage histogram. A SECOND mapping decision of the evaluator's, footnoted on the row: the item's instruction text (and for noul its true/false criteria) is appended to the state, because state is the only text field the author's API has. Cost is a clearly labelled size-class ESTIMATE on the base-size-encoder line the PUBLISHED v1.6 field carries - modernbert-jev-v2 (measured in v1.6.8 on the earlier a2 draw) and the v1.6.1 rows tacet-sonata and watt-flash-0.1 - and no row of this draw carries it. It is not a bill.", "reflex-instinct-0.6b: 1,500 attempted, 1,500 answered, zero failures of any class - the pre-registered figure, exact. All answered items native; the Speed figures are the published method's percentiles over this draw's open standard and judge items. NOT RANKED, and here is exactly why: CoS deputy decision #18761 accepts this row's Intelligence, Calibration and Cost only and declares its Speed figure (58.972) NON-OFFICIAL, because the one-minute load on the shared host ranged 9.42-33.12 (mean 15.28, 155 samples) during the window and the evaluator's own preparation activity overlapped it. The published composite weights Speed, so this row is shown with its axes and its Capability 38.556 - which is Intelligence and Calibration only and contains no Speed - but it carries no rank and no composite placement. Its Capability 38.556 is the highest of this release's three new rows, above phocinae-largha-150m-v1's 30.240 - said here because the Capability board does not show it: that board contains no Speed at all, so a reader could reasonably ask why this row is absent from it. The answer is that THIS RELEASE applies #18761 to the whole row rather than axis by axis - the deputy's decision excludes the Speed figure, and whether a row with a non-official Speed may still be ranked on Capability alone is the open question #18825, which has no answer yet. The conservative reading ships. DISCLOSED: the author's own encode() truncates state lines above its own max_len of 16,384 instead of refusing them, so this row answers the 15 items of 106,804-122,936 characters that four other rows of this draw recorded as context refusals (attest-31b-v1.0, julia-1, standard_one_3b_sh and uniphore-4b-decider); that is the author's design choice, not the evaluator's, and those items are the slowest in the row. Cost is a clearly labelled size-class ESTIMATE on the sub-1B line the PUBLISHED v1.6 field carries (liquid-d1-omni-600m, jiwo-0.8b, vegaml-0.8b) - no row of this draw carries it - and it is not a bill." ], "n_ranked": 9, "not_measured": [], "noul_method": { "applied": "O1S", "native_method_source_sha256": "119e59486ab3300b972fbcbadb0fe1dceceb124287607375aa61c66063c607d0" }, "options": { "A": { "intelligence_floor": 50.0, "weights": { "calibration": 25, "cost": 25, "intelligence": 25, "speed": 25 } }, "B": { "intelligence_floor": 50.0, "weights": { "calibration": 20, "cost": 20, "intelligence": 40, "speed": 20 } }, "C": { "intelligence_floor": 60.0, "weights": { "calibration": 25, "cost": 25, "intelligence": 25, "speed": 25 } } }, "protocol": "jevbench::v1.6", "provisional": false, "revision": "v1.6.15", "roster_count": 11, "run_kind": "regular-queue", "sample": { "open": 300, "published_open": 300, "sealed": 1200, "total": 1500 }, "sealed_share_of_intelligence": 0.5, "source_note": "Eleven rows on the fresh regular draw v1.6-regular-20261012 (seeded rotation-ledger draw of 1,200 sealed items + the 300-item epoch public set, drawn 2026-10-12T01:32:59Z): the seven official v1.6.14 rows, three self-hosted CPU systems added in v1.6.15, and the unranked A2 measurement carried from v1.6.14 unchanged. ONE score_v16.py invocation over the ten full-draw rows, unmodified frozen path (sha256 02132271...), JEV_NOUL_METHOD=O1S, B=1000, seed 16, --g-med-fixed 9.839526770355711, the median frozen at v1.6.10, so all seven earlier rows keep every point score. The A2 row is not rescored here. Measured and scored by jevbench-add-requests-20260919 as release owner under CoS deputy decisions #18209/#18211 (free CPU admission) and #18761 (reflex Speed non-official; the authorised pricing correction behind julia-1's price line).", "source_sha256": "28b3ba7d9ef83f907f679116c66085c150d9fa2534f03f97cb795c33e6ae0b37", "status": "published", "systems": [ { "addendum": null, "api_flag": true, "api_subset": false, "author": "attest", "axes": { "calibration": 73.84793106824938, "cost": 50.86035455384382, "intelligence": 65.23849472031306, "speed": 97.10993413996468 }, "calibration": { "parts": { "choice": { "ece_hard": 0.08913950574898584, "ece_score": 82.17209885020283, "mean_tvd": null, "n": 740, "n_ece_hard": 305, "n_tvd": 0, "score": 82.17209885020283 }, "noul": { "abstention_rate": 0.02680965147453085, "brier": 0.12374104570386348, "ece": 0.1233374127983535, "n": 373, "score": 75.3325174403293 }, "score": { "interval90_coverage": 0.7016129032258065, "interval90_width": 0.0378584229390681, "n": 372, "nrps": 0.10002561109774619, "score": 64.03917691421601, "top_ece": 0.3095954253089667 } }, "score": 73.84793106824938 }, "capability": 69.54321289428123, "composite_ci95": { "A": [ 65.66813002353437, 69.70169455928922 ], "B": [ 64.19111453033545, 69.67178531731808 ], "C": [ 63.43104562382649, 69.70169455928922 ] }, "cost": { "basis": "ESTIMATE, v1.5-M2 base-model reference rule: vendor-hosted API, customer list price USD 0.048/M input and USD 0.00/M output effective 2026-10-11 has been in force fewer than 30 days. Base model google/gemma-4-31b-it reference on OpenRouter USD 0.09/M input / USD 0.34/M output, snapshot fetched 2026-10-12T17:55:05Z (SHA-256 81eae2f3f94bf299c8d4499260f842c357b7a869ce82dfa1741ad6a89cf57014), same base-model estimate precedent as metask_jev_rain_31b_trained. Applied to measured response usage; failed HTTP 500 usage/billing may be unreported. No output generation; reported output tokens 0. Estimated, not a vendor bill.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.043449839999999997 }, "display": "attest-31B-v1.0", "endpoint_condition": "Paid public fast-lane order 07d13315. Vendor-hosted attest.engineering /v1/systemone API; latency measured from Sandy (Germany), using the official per-item latency subset without API adjustment. Full S1200+P300 draw, not API A300 supplement; no API equating. 15 of 1,500 requests returned HTTP 500 after 8 retry invocations and are scored as wrong in the full denominator. Public model documentation declares frozen google/gemma-4-31B-it @842da3794eaa0b77d5f08bae87a17459d91ff475, unmodified bf16, no adapter or quantization; single forward pass reading option-label logits, no generated tokens, one fitted temperature (value not documented). Hosting implementation and deployed weights were not independently inspected. Hosted API access is not an open-weights release by the vendor; base weights are under the Gemma 4 licence. ESTIMATE, v1.5-M2 base-model reference rule: vendor-hosted API, customer list price USD 0.048/M input and USD 0.00/M output effective 2026-10-11 has been in force fewer than 30 days. Base model google/gemma-4-31b-it reference on OpenRouter USD 0.09/M input / USD 0.34/M output, snapshot fetched 2026-10-12T17:55:05Z (SHA-256 81eae2f3f94bf299c8d4499260f842c357b7a869ce82dfa1741ad6a89cf57014), same base-model estimate precedent as metask_jev_rain_31b_trained. Applied to measured response usage; failed HTTP 500 usage/billing may be unreported. No output generation; reported output tokens 0. Estimated, not a vendor bill.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 64.92424666524875, "I_sealed": 65.55274277537737, "base": 65.23849472031306, "excess": -10.468022880484334, "gap": -0.6284961101286228, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 73.17162102453656, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 88.32432432432431, "hard": 67.9144385026738, "judge": 72.31377719182598, "standard": 77.39640016743407 } }, "open|noul": { "cc": 68.67632850241546, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 73.91304347826086, "judge": 60.00000000000001, "standard": 55.55555555555556 } }, "open|score": { "cc": 52.92479046879421, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 63.941722778327325, "hard": 60.749677220164536, "judge": 51.85551193839042, "standard": 33.370468606892665 } }, "sealed|choice": { "cc": 74.62818230752828, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 80.94027954256671, "hard": 71.89776801202729, "judge": 74.43682664054849, "standard": 77.2199957814807 } }, "sealed|noul": { "cc": 68.21259029184918, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 86.20689655172413, "hard": 58.18181818181818, "judge": 72.30769230769232, "standard": 73.13432835820895 } }, "sealed|score": { "cc": 53.817455726754645, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 78.07383467048999, "hard": 52.90470314143051, "judge": 47.46936317941805, "standard": 53.03691024654009 } } } }, "jevbench_score": 67.9936749770638, "key": "attest-31b-v1.0", "last_measured_on": "2026-10-12", "licence": "Hosted API; declared base weights under the Gemma 4 licence (no vendor weights release)", "listing": "ranked", "measured_in": "v1.6.13", "measurement_date_status": "measured", "model_pin": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475", "noul_decisive": { "acc_among_decisive": 0.8787878787878788, "decisive": 363, "decisive_rate": 0.9731903485254692, "n": 375, "supported": true, "valid": 373 }, "planned_weight_denominator": 1.0, "rank": 2, "ranked": true, "ranks": { "A": 2, "B": 2, "C": 1, "capability": 3 }, "repo": "https://attest.engineering/model", "scores": { "A": 67.9936749770638, "B": 67.42417758535593, "C": 67.9936749770638 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.10787388649987406, "p50_s_raw": 0.10787388649987406, "p95_s_adjusted": 0.18033925039962923, "p95_s_raw": 0.18033925039962923 }, "status": { "answered_ok": 1485, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.8636363636363636, "cc": 91.66666666649981, "cc_by_type": { "choice": 75.0, "noul": 100.0, "score": 99.99999999949942 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 91.66666666649981 }, "calibration_gen": { "accuracy": 0.7227722772277227, "cc": 65.68205659085977, "cc_by_type": { "choice": 61.82396606574761, "noul": 76.0, "score": 59.22220370683169 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 65.68205659085977 }, "composite_gen": { "accuracy": 0.8217821782178217, "cc": 71.7068789691002, "cc_by_type": { "choice": 93.4959349593496, "noul": 62.962962962962955, "score": 58.66173898498804 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 71.7068789691002 }, "fairness": { "accuracy": 0.9375, "cc": 90.47619047605701, "cc_by_type": { "choice": 71.42857142857143, "noul": 100.0, "score": 99.99999999959958 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 90.47619047605701 }, "jaggedness": { "accuracy": 0.5722222222222222, "cc": 41.84478952801769, "cc_by_type": { "choice": 45.13955726660251, "noul": 56.75675675675676, "score": 23.63805456069382 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 41.84478952801769 }, "llm_router": { "accuracy": 0.8977635782747604, "cc": 83.50801061677168, "cc_by_type": { "choice": 88.90552082416131, "noul": 93.81443298969072, "score": 67.804078036463 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 83.50801061677168 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.7407407407407407, "cc": 60.78261745433469, "cc_by_type": { "choice": 50.51546391752578, "noul": 75.0, "score": 56.83238844547831 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 60.78261745433469 }, "multilingual": { "accuracy": 0.7563636363636363, "cc": 63.88819552264045, "cc_by_type": { "choice": 71.63351809585916, "noul": 61.111111111111114, "score": 58.91995736095108 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 63.88819552264045 }, "negation_pair_gen": { "accuracy": 0.7441860465116279, "cc": 41.72809466728673, "cc_by_type": { "choice": 93.33333333333333, "noul": 6.666666666666665, "score": 25.184284001860192 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 41.72809466728673 }, "option_order": { "accuracy": 0.8958333333333334, "cc": 86.68442077230361, "cc_by_type": { "choice": 86.68442077230361 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 86.68442077230361 }, "perturb_pair": { "accuracy": 0.875, "cc": 85.01995309361071, "cc_by_type": { "choice": 81.98198198198199, "noul": 100.0, "score": 73.07787729885015 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 85.01995309361071 }, "usecase": { "accuracy": 0.7772727272727272, "cc": 69.59874631177811, "cc_by_type": { "choice": 76.7156862745098, "noul": 75.0, "score": 57.080552660824544 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 69.59874631177811 } }, "lang": { "ar": { "accuracy": 0.5625, "cc": 25.657000371915586, "cc_by_type": { "choice": 42.857142857142854, "noul": 50.0, "score": -15.886141741396088 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 25.657000371915586 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 1.0, "cc": 99.83499124305172, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 99.50497372915514 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 99.83499124305172 }, "de": { "accuracy": 0.875, "cc": 88.7927420089141, "cc_by_type": { "choice": 76.10619469026548, "noul": 100.0, "score": 90.27203133647683 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 88.7927420089141 }, "el": { "accuracy": 0.7333333333333333, "cc": 66.60459342006904, "cc_by_type": { "choice": 66.4804469273743, "noul": 33.33333333333333, "score": 99.9999999994995 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 66.60459342006904 }, "en": { "accuracy": 0.7793223284100782, "cc": 66.85046932152197, "cc_by_type": { "choice": 75.39693193883403, "noul": 70.50359712230217, "score": 54.650878903429714 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 66.85046932152197 }, "es": { "accuracy": 0.8064516129032258, "cc": 66.69113187317208, "cc_by_type": { "choice": 80.42414355628058, "noul": 100.0, "score": 19.649252063235643 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 66.69113187317208 }, "fi": { "accuracy": 0.75, "cc": 66.84881602901042, "cc_by_type": { "choice": 67.21311475409836, "noul": 33.33333333333333, "score": 99.99999999959955 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 66.84881602901042 }, "fr": { "accuracy": 0.7142857142857143, "cc": 75.35603715160619, "cc_by_type": { "choice": 57.64705882352941, "noul": 100.0, "score": 68.42105263128916 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 75.35603715160619 }, "hi": { "accuracy": 0.9047619047619048, "cc": 81.9134078211331, "cc_by_type": { "choice": 83.24022346368716, "noul": 100.0, "score": 62.499999999712166 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 81.9134078211331 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8, "cc": 53.22313405090798, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": -26.300979764657196 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 53.22313405090798 }, "ja": { "accuracy": 0.68, "cc": 45.07973339782123, "cc_by_type": { "choice": 71.40587768069896, "noul": 60.00000000000001, "score": 3.833322512764714 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 45.07973339782123 }, "ko": { "accuracy": 0.7, "cc": 46.296506956290735, "cc_by_type": { "choice": 55.55555555555555, "noul": 0.0, "score": 83.33396531331667 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 46.296506956290735 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 77.77777777761094, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 99.9999999994995 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 77.77777777761094 }, "nl": { "accuracy": 0.6, "cc": 43.862463409737074, "cc_by_type": { "choice": 48.05194805194805, "noul": 33.33333333333333, "score": 50.202108843929864 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 43.862463409737074 }, "no": { "accuracy": 0.5, "cc": 10.483870967741936, "cc_by_type": { "choice": -29.032258064516128, "noul": 50.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 10.483870967741936 }, "pl": { "accuracy": 0.7435897435897436, "cc": 58.11281326204074, "cc_by_type": { "choice": 84.76512907321202, "noul": 39.99999999999999, "score": 49.5733107129102 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 58.11281326204074 }, "pt": { "accuracy": 0.75, "cc": 73.9630456140903, "cc_by_type": { "choice": 57.57218621096051, "noul": 100.0, "score": 64.3169506313104 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 73.9630456140903 }, "sv": { "accuracy": 0.6363636363636364, "cc": 52.97872340417189, "cc_by_type": { "choice": 48.93617021276595, "noul": 60.00000000000001, "score": 49.99999999974971 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 52.97872340417189 }, "tr": { "accuracy": 0.9230769230769231, "cc": 88.24218637816098, "cc_by_type": { "choice": 78.26086956521739, "noul": 100.0, "score": 86.46568956926559 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 88.24218637816098 }, "uk": { "accuracy": 0.7142857142857143, "cc": 72.53664033508191, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 49.86798552137475 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 72.53664033508191 }, "zh": { "accuracy": 0.9, "cc": 66.66666666666664, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": -4.440892098500626e-14 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 66.66666666666664 } }, "type": { "choice": { "accuracy": 0.804, "cc": 74.68378652480914, "cc_by_type": { "choice": 74.68378652480914 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 74.68378652480914 }, "noul": { "accuracy": 0.8506666666666667, "cc": 70.13333333333334, "cc_by_type": { "noul": 70.13333333333334 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 70.13333333333334 }, "score": { "accuracy": 0.6506666666666666, "cc": 55.14020270807265, "cc_by_type": { "score": 55.14020270807265 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 55.14020270807265 } }, "usecase": { "advertising": { "accuracy": 0.6363636363636364, "cc": 41.05013190841991, "cc_by_type": { "choice": 84.0, "noul": 0.0, "score": 39.15039572525971 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 41.05013190841991 }, "code_linting": { "accuracy": 0.7916666666666666, "cc": 69.18221735790132, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 7.546652073703974 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 69.18221735790132 }, "customer_support": { "accuracy": 0.7142857142857143, "cc": 33.73983739844072, "cc_by_type": { "choice": 51.21951219512195, "noul": 100.0, "score": -49.999999999799805 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 33.73983739844072 }, "demand_forecasting": { "accuracy": 0.35294117647058826, "cc": 40.20929030056567, "cc_by_type": { "choice": -5.263157894736844, "noul": 100.0, "score": 25.891028796433858 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 40.20929030056567 }, "ecommerce_marketplaces": { "accuracy": 0.7647058823529411, "cc": 59.627914855635204, "cc_by_type": { "choice": 71.0144927536232, "noul": 33.33333333333333, "score": 74.53591847994909 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 59.627914855635204 }, "feature_extraction": { "accuracy": 0.7894736842105263, "cc": 81.67133102077554, "cc_by_type": { "choice": 49.685534591194966, "noul": 100.0, "score": 95.32845847113167 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 81.67133102077554 }, "financial_crime": { "accuracy": 0.6333333333333333, "cc": 51.05695837350043, "cc_by_type": { "choice": 70.58823529411765, "noul": 42.85714285714286, "score": 39.72549696924077 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 51.05695837350043 }, "gaming": { "accuracy": 0.6363636363636364, "cc": 31.06744966418366, "cc_by_type": { "choice": 58.18815331010453, "noul": -50.0, "score": 85.01419568244644 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 31.06744966418366 }, "insurance_claims": { "accuracy": 0.6, "cc": 45.83333333332221, "cc_by_type": { "choice": 37.5, "noul": 100.0, "score": -3.3373304120232206e-11 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 45.83333333332221 }, "knowledge_graphs": { "accuracy": 0.8181818181818182, "cc": 79.31347738618848, "cc_by_type": { "choice": 74.30406852248395, "noul": 100.0, "score": 63.636363636081526 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 79.31347738618848 }, "lead_generation": { "accuracy": 0.8, "cc": 72.4290843339358, "cc_by_type": { "choice": 88.3720930232558, "noul": 77.77777777777777, "score": 51.13738220077384 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 72.4290843339358 }, "legal_compliance": { "accuracy": 0.8, "cc": 74.35692248903656, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 87.58689649936777 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 74.35692248903656 }, "llm_guardrails": { "accuracy": 0.8888888888888888, "cc": 78.81343307279933, "cc_by_type": { "choice": 100.00000000000001, "noul": 60.00000000000001, "score": 76.44029921839794 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 78.81343307279933 }, "model_routing": { "accuracy": 0.8894009216589862, "cc": 82.42648273169006, "cc_by_type": { "choice": 90.64935064935067, "noul": 100.0, "score": 56.63009754571953 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 82.42648273169006 }, "moderation_trust_safety": { "accuracy": 0.9444444444444444, "cc": 92.05979877042455, "cc_by_type": { "choice": 100.00000000000001, "noul": 100.0, "score": 76.17939631127363 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 92.05979877042455 }, "recruiting": { "accuracy": 0.7222222222222222, "cc": 57.779434740179134, "cc_by_type": { "choice": 66.57381615598885, "noul": 33.33333333333333, "score": 73.43115473121522 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 57.779434740179134 }, "risk_assessment": { "accuracy": 0.6666666666666666, "cc": 47.99092050610208, "cc_by_type": { "choice": 70.44334975369459, "noul": 50.0, "score": 23.52941176461164 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 47.99092050610208 }, "scientific_discovery": { "accuracy": 0.7391304347826086, "cc": 59.19668873382481, "cc_by_type": { "choice": 71.01449275362319, "noul": 33.33333333333333, "score": 73.24224011451793 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 59.19668873382481 }, "search_retrieval": { "accuracy": 1.0, "cc": 99.99999999986125, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 99.99999999958375 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 99.99999999986125 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 69.87797972313629, 77.28889167273778 ], "I": [ 58.92998553241625, 69.8560029567472 ], "capability": [ 65.07732498188777, 73.10995603043845 ], "composite": { "A": [ 65.66813002353437, 69.70169455928922 ], "B": [ 64.19111453033545, 69.67178531731808 ], "C": [ 63.43104562382649, 69.70169455928922 ] }, "seed": 16 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": true, "g_med_used": 9.839526770355711, "lane": "api", "metric_basis": "S u P (full set, API lane, v1.6.1 amendment; not equated)", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 73.84793106824938, "capability": 69.54321289428123, "intelligence": 65.23849472031306 }, "per_type": { "choice": { "accuracy": 0.804, "calibration": 82.17209885020283, "cc": 74.68378652480914, "cc_by_type": { "choice": 74.68378652480914 }, "cc_tiered": 73.89990166603242, "cc_tiered_open": 73.17162102453656, "cc_tiered_sealed": 74.62818230752828, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 74.68378652480914 }, "noul": { "accuracy": 0.8506666666666667, "calibration": 75.3325174403293, "cc": 70.13333333333334, "cc_by_type": { "noul": 70.13333333333334 }, "cc_tiered": 68.44445939713232, "cc_tiered_open": 68.67632850241546, "cc_tiered_sealed": 68.21259029184918, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 70.13333333333334 }, "score": { "accuracy": 0.6506666666666666, "calibration": 64.03917691421601, "cc": 55.14020270807265, "cc_by_type": { "score": 55.14020270807265 }, "cc_tiered": 53.37112309777443, "cc_tiered_open": 52.92479046879421, "cc_tiered_sealed": 53.817455726754645, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 55.14020270807265 } }, "raw_on_A": null, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 750 }, "noul": { "invalid_rate": 0.005333333333333301, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 66.24320971525083, "Emphasis on Accuracy 60:20:20": 65.83763287132996, "Emphasis on Cost 20:20:60": 59.093957268612236, "Emphasis on Speed 20:60:20": 75.89225472047454, "Intelligence only": 65.23849472031306 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Supersonic Labs / @supersonicai (Hugging Face SupersonicLabs)", "axes": { "calibration": 17.26803648238681, "cost": 89.8795371983372, "intelligence": 2.4412732607743255, "speed": 79.42119248319958 }, "calibration": { "parts": { "choice": { "ece_hard": 0.49162634593303195, "ece_score": 1.67473081339361, "mean_tvd": null, "n": 739, "n_ece_hard": 304, "n_tvd": 0, "score": 1.67473081339361 }, "noul": { "abstention_rate": 0.27419354838709675, "brier": 0.45131804975332956, "ece": 0.4322726807691505, "n": 372, "score": 13.545463846169902 }, "score": { "interval90_coverage": 0.5887096774193549, "interval90_width": 0.3958333333333335, "n": 372, "nrps": 0.26832170424806157, "score": 36.583914787596925, "top_ece": 0.5187092957892186 } }, "score": 17.26803648238681 }, "capability": 9.854654871580568, "composite_ci95": { "A": [ 0.0009450466124014096, 0.14957124870860547 ], "B": [ 0.000611407847685655, 0.10775944401732225 ], "C": [ 0.0006562823697232012, 0.10386892271430934 ] }, "cost": { "basis": "ESTIMATE (*est.), size-class reference; self-hosted open weights, no public bookable tariff for these weights. The frozen pricing_v15 path is asserted with ONE authorised, scoped correction: CoS deputy decision #18761 added the declared base jhu-clsp/mmBERT-small to pricing_v15.UNLISTED_BASE_MODELS, exactly as openbmb/MiniCPM5-2B and answerdotai/ModernBERT-base already are, so reference() returns no market reference for it rather than raising and the ancestry can be carried in base_model instead of being dropped to null. The addition changes no other row: it was proved behaviour-identical over every registry entry on the box before it was installed (r108/receipts/PRICING-EQUIV-R108.json) and independently reviewed (r108/receipts/REVIEW-PRICING-R108.json). Florian's standing 24 Sep rule then gives a clearly labelled size-class estimate: the base-size-encoder line the PUBLISHED v1.6 field already carries - USD 0.005 per 1M input and 0.0 per 1M output, the line modernbert-jev-v2, tacet-sonata and watt-flash-0.1 carry there (no row of THIS draw carries it, and lev-350m carries 0.01/0.0, not this line) (four base-size encoders in the dated 25 Sep 2026 DeepInfra snapshot sha256 84eb7926... all price at exactly that) - so no new release-lane pricing line is created. x this row's own measured tokens: usage.input_tokens comes from the author's OWN audit function engine.encoding_info() and usage.output_tokens is 0 by construction (one encoder forward pass read at marker positions, nothing generated). Estimated, not charged.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.0021744466666666668 }, "display": "Julia-1 (Supersonic Labs)", "endpoint_condition": "Sandy, the evaluator's own shared CPU host (AMD Ryzen 5 3600, 6 cores / 12 threads, Zen 2, AVX2), NOT a dedicated machine: other jobs ran beside it and the 1-minute load average was logged every 60 s into logs/CONDITIONS-julia-1.jsonl (11 samples, 10.87-13.78, mean 12.5). The author ships NO server, so the HTTP surface is OURS: r107/src/julia_shim_r107.py, a pure transport that makes no mapping decision, because the author's own julia.typed.predict_typed already speaks JevBench's typed-question shape exactly (choice keyed by the item's own criteria keys, score keyed '0'..'n-1' with score = sum(i*p_i), noul = p(true)). Engine built with the author's own inference-policy.json (max_length 8192, head_length 512, strict_encoding true), torch backend, device cpu, float32 (their julia_config weight_dtype), thread count left at their own default of 4 - all four read back from the run's own /health rather than asserted. Run inside a loopback-only network namespace whose outbound probe failed as required, offline (HF_HUB_OFFLINE=1), credential-free, nice 5, in a systemd scope with MemoryMax=8G and no swap. Speed is therefore a loaded-shared-CPU figure and a lower bound for this model on a dedicated core or a GPU; the standard self-hosted x2 + 0.15 s adjustment is applied on top and the raw p50/p95 are kept. Measured for this release on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720...) under CoS deputy decision #18209, which admitted this row only after an independent source review and a synthetic-only smoke, both of which were met in that order before the first sealed item. DISCLOSED: the author's own strict_encoding policy REFUSES rather than truncates, so 17 of the 1,500 items (15 over their lossless context budget, 2 with an option above their 48-token option contract) are honest item-level misses recorded as HTTP 422 - predicted exactly, class split included, before the run.", "endpoint_kind": "cpu", "full_coverage": true, "intelligence": { "I_open": 2.994076934245856, "I_sealed": 1.888469587302795, "base": 2.4412732607743255, "excess": -8.733919423412651, "gap": 1.105607346943061, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 8.982230802737568, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 2.7027027027027115, "hard": 13.180245360176164, "judge": 5.075807514831907, "standard": 9.585600669736293 } }, "open|noul": { "cc": 0.0, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": -57.14285714285714, "hard": -4.347826086956519, "judge": 0.0, "standard": -22.22222222222222 } }, "open|score": { "cc": 0.0, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": -4.595553956916043, "hard": -21.831788639496995, "judge": -76.77226748673378, "standard": -19.64084658486791 } }, "sealed|choice": { "cc": 5.6654087619083855, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 12.325285895806868, "hard": 0.6013646351335588, "judge": 11.85112634671892, "standard": 3.1849820712929935 } }, "sealed|noul": { "cc": 0.0, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": -41.379310344827594, "hard": -41.81818181818182, "judge": -50.76923076923077, "standard": -19.402985074626866 } }, "sealed|score": { "cc": 0.0, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": -36.27109595021023, "hard": -40.51450244498809, "judge": -18.32435058181421, "standard": -13.222659125532354 } } } }, "jevbench_score": 0.019411107562972558, "key": "julia-1", "last_measured_on": "2026-10-13", "licence": "Apache-2.0 (Hugging Face cardData.license and README; the pinned tree contains no LICENSE file)", "listing": "ranked", "measured_in": "v1.6.15", "measurement_date_status": "measured", "model_pin": "SupersonicLabs/Julia-1@a85b127321d580d65176c89ced8273f305745d85", "noul_decisive": { "acc_among_decisive": 0.45555555555555555, "decisive": 270, "decisive_rate": 0.7258064516129032, "n": 375, "supported": true, "valid": 372 }, "planned_weight_denominator": 1.0, "rank": 9, "ranked": true, "ranks": { "A": 9, "B": 9, "C": 9, "capability": 9 }, "repo": "https://huggingface.co/SupersonicLabs/Julia-1", "scores": { "A": 0.019411107562972558, "B": 0.01323121125368159, "C": 0.013479935807619834 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.7337018699967303, "p50_s_raw": 0.2918509349983651, "p95_s_adjusted": 1.5572599903855007, "p95_s_raw": 0.7036299951927504 }, "status": { "answered_ok": 1483, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.2727272727272727, "cc": -27.879433144127862, "cc_by_type": { "choice": 0.0, "noul": 0.0, "score": -83.63829943238359 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 0.0 }, "calibration_gen": { "accuracy": 0.3564356435643564, "cc": -3.761010359494321, "cc_by_type": { "choice": 4.559915164369044, "noul": 4.0000000000000036, "score": -19.842946242852012 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 0.0 }, "composite_gen": { "accuracy": 0.25742574257425743, "cc": -31.644341476391443, "cc_by_type": { "choice": -7.317073170731707, "noul": -18.518518518518523, "score": -69.0974327399241 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 0.0 }, "fairness": { "accuracy": 0.3541666666666667, "cc": -5.944000384605097, "cc_by_type": { "choice": 4.761904761904761, "noul": -60.0, "score": 37.40609408427995 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 0.0 }, "jaggedness": { "accuracy": 0.37222222222222223, "cc": -8.884037970016676, "cc_by_type": { "choice": 24.927815206929747, "noul": -45.945945945945944, "score": -5.6339831710338295 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 0.0 }, "llm_router": { "accuracy": 0.23003194888178913, "cc": -20.437719287542667, "cc_by_type": { "choice": -0.6427753808223877, "noul": -38.144329896907216, "score": -22.526052584898395 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 0.0 }, "long_state": { "accuracy": 0.0, "cc": -143.01075268817203, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0, "score": -300.0 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.18518518518518517, "cc": -32.03827054080279, "cc_by_type": { "choice": 1.0309278350515454, "noul": -75.0, "score": -22.145739457459925 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.29454545454545455, "cc": -25.78754754021827, "cc_by_type": { "choice": 17.835017932833402, "noul": -55.55555555555556, "score": -39.64210499793266 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 0.0 }, "negation_pair_gen": { "accuracy": 0.3372093023255814, "cc": -37.503184879754805, "cc_by_type": { "choice": 6.666666666666665, "noul": -13.33333333333333, "score": -105.84288797259775 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.0 }, "option_order": { "accuracy": 0.20833333333333334, "cc": -1.1984021304926722, "cc_by_type": { "choice": -1.1984021304926722 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 0.0 }, "perturb_pair": { "accuracy": 0.265625, "cc": -25.52938809242961, "cc_by_type": { "choice": -5.105105105105101, "noul": 19.999999999999996, "score": -91.48305917218373 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 0.0 }, "usecase": { "accuracy": 0.2409090909090909, "cc": -16.515408619322642, "cc_by_type": { "choice": 3.1862745098039587, "noul": -25.0, "score": -27.732500367771884 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 0.0 } }, "lang": { "ar": { "accuracy": 0.375, "cc": -2.776462098911864, "cc_by_type": { "choice": 42.857142857142854, "noul": 0.0, "score": -51.186529153878446 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.14285714285714285, "cc": 36.84210526315789, "cc_by_type": { "choice": -26.315789473684216, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 36.84210526315789 }, "da": { "accuracy": 0.3076923076923077, "cc": -37.718345339022456, "cc_by_type": { "choice": 37.78801843317972, "noul": -100.0, "score": -50.9430544502471 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 0.0 }, "de": { "accuracy": 0.4375, "cc": 22.354227161079496, "cc_by_type": { "choice": 20.353982300884955, "noul": 25.0, "score": 21.70869918235354 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 22.354227161079496 }, "el": { "accuracy": 0.2, "cc": 0.6000833683740154, "cc_by_type": { "choice": -17.31843575418994, "noul": -33.333333333333336, "score": 52.452019192645324 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 0.6000833683740154 }, "en": { "accuracy": 0.2797567332754127, "cc": -16.97684900279254, "cc_by_type": { "choice": 3.7844302607973472, "noul": -26.618705035971225, "score": -28.096272233203745 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 0.0 }, "es": { "accuracy": 0.41935483870967744, "cc": -11.423096652088782, "cc_by_type": { "choice": 31.484502446982063, "noul": -25.0, "score": -40.75379240324841 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 0.0 }, "fi": { "accuracy": 0.125, "cc": -39.640298601530766, "cc_by_type": { "choice": 1.6393442622950836, "noul": -100.0, "score": -20.56024006688737 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 0.0 }, "fr": { "accuracy": 0.21428571428571427, "cc": -41.1336147354147, "cc_by_type": { "choice": 15.294117647058824, "noul": -100.0, "score": -38.69496185330294 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 0.0 }, "hi": { "accuracy": 0.14285714285714285, "cc": -67.07248359384337, "cc_by_type": { "choice": -0.5586592178770929, "noul": -100.0, "score": -100.65879156365303 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 0.0 }, "id": { "accuracy": 0.0, "cc": -66.66666666666667, "cc_by_type": { "choice": -33.333333333333336, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.3333333333333333, "cc": -25.47727987845241, "cc_by_type": { "choice": 15.822291504286824, "noul": -33.333333333333336, "score": -58.92079780631072 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 0.0 }, "ja": { "accuracy": 0.24, "cc": -18.87257764229911, "cc_by_type": { "choice": 28.514694201747414, "noul": -80.0, "score": -5.132427128644745 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 0.0 }, "ko": { "accuracy": 0.0, "cc": -77.83191986347525, "cc_by_type": { "choice": -33.333333333333336, "noul": -100.0, "score": -100.16242625709242 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.35714285714285715, "cc": -10.156143207655505, "cc_by_type": { "choice": 16.666666666666668, "noul": -33.333333333333336, "score": -13.801762956299847 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 0.0 }, "nl": { "accuracy": 0.2, "cc": -51.11558643410791, "cc_by_type": { "choice": -29.87012987012987, "noul": -33.333333333333336, "score": -90.14329609886053 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 }, "no": { "accuracy": 0.16666666666666666, "cc": -32.25806451612903, "cc_by_type": { "choice": 35.483870967741936, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.23076923076923078, "cc": -31.24251604325812, "cc_by_type": { "choice": 16.208209902666106, "noul": -80.0, "score": -29.93575803244046 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 0.0 }, "pt": { "accuracy": 0.39285714285714285, "cc": 2.5150561249822148, "cc_by_type": { "choice": 25.75132586918091, "noul": 19.999999999999996, "score": -38.20615749423426 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 2.5150561249822148 }, "sv": { "accuracy": 0.09090909090909091, "cc": -37.55166369327866, "cc_by_type": { "choice": -2.127659574468081, "noul": -100.0, "score": -10.52733150536791 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 0.0 }, "tr": { "accuracy": 0.38461538461538464, "cc": -27.04550827576324, "cc_by_type": { "choice": 34.78260869565218, "noul": -100.0, "score": -15.919133522941898 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 0.0 }, "uk": { "accuracy": 0.14285714285714285, "cc": -41.81990641185333, "cc_by_type": { "choice": 3.225806451612906, "noul": -100.0, "score": -28.68552568717291 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 0.0 }, "zh": { "accuracy": 0.3, "cc": -68.47760196419559, "cc_by_type": { "choice": 23.72881355932204, "noul": -100.0, "score": -129.1616194519088 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 0.0 } }, "type": { "choice": { "accuracy": 0.272, "cc": 5.968349949291043, "cc_by_type": { "choice": 5.968349949291043 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 5.968349949291043 }, "noul": { "accuracy": 0.328, "cc": -34.4, "cc_by_type": { "noul": -34.4 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.24533333333333332, "cc": -29.344915811166672, "cc_by_type": { "score": -29.344915811166672 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "usecase": { "advertising": { "accuracy": 0.22727272727272727, "cc": -38.4024160564452, "cc_by_type": { "choice": -12.0, "noul": -75.0, "score": -28.207248169335596 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.3333333333333333, "cc": -2.238256630096153, "cc_by_type": { "choice": 44.099378881987576, "noul": -11.111111111111116, "score": -39.70303766116492 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 0.0 }, "customer_support": { "accuracy": 0.42857142857142855, "cc": -3.7905034661254127, "cc_by_type": { "choice": 18.699186991869915, "noul": 50.0, "score": -80.07069739024615 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 0.0 }, "demand_forecasting": { "accuracy": 0.29411764705882354, "cc": -50.406556307064626, "cc_by_type": { "choice": 18.128654970760234, "noul": -100.0, "score": -69.34832389195411 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.35294117647058826, "cc": -8.733832465640946, "cc_by_type": { "choice": 27.53623188405797, "noul": -33.333333333333336, "score": -20.40439594764747 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 0.0 }, "feature_extraction": { "accuracy": 0.42105263157894735, "cc": -52.37905037945314, "cc_by_type": { "choice": 11.949685534591195, "noul": 0.0, "score": -169.08683667295062 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.23333333333333334, "cc": -29.56253397544977, "cc_by_type": { "choice": 21.568627450980394, "noul": -71.42857142857143, "score": -38.827657948758265 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 0.0 }, "gaming": { "accuracy": 0.09090909090909091, "cc": -54.303624656078085, "cc_by_type": { "choice": -4.529616724738673, "noul": -100.0, "score": -58.38125724349559 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.06666666666666667, "cc": -27.932241245491838, "cc_by_type": { "choice": -4.166666666666665, "noul": -100.0, "score": 20.369942930191154 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 0.0 }, "knowledge_graphs": { "accuracy": 0.22727272727272727, "cc": -4.197964663778406, "cc_by_type": { "choice": -2.7837259100642404, "noul": 0.0, "score": -9.810168081270977 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.24, "cc": -7.467987900190109, "cc_by_type": { "choice": 6.976744186046504, "noul": -33.333333333333336, "score": 3.9526254467165067 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 0.0 }, "legal_compliance": { "accuracy": 0.3, "cc": -26.741444679738834, "cc_by_type": { "choice": -29.032258064516128, "noul": -33.333333333333336, "score": -17.858742641367044 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 0.0 }, "llm_guardrails": { "accuracy": 0.2962962962962963, "cc": -17.699209031887154, "cc_by_type": { "choice": 19.385796545105567, "noul": -60.0, "score": -12.483423640767022 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 0.0 }, "model_routing": { "accuracy": 0.22580645161290322, "cc": -17.59559356925139, "cc_by_type": { "choice": -5.194805194805169, "noul": -33.333333333333336, "score": -14.258642179615656 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 0.0 }, "moderation_trust_safety": { "accuracy": 0.16666666666666666, "cc": -51.88151472980022, "cc_by_type": { "choice": -3.646833013435703, "noul": -100.0, "score": -51.99771117596499 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 0.0 }, "recruiting": { "accuracy": 0.4444444444444444, "cc": 0.4935502236767717, "cc_by_type": { "choice": -0.2785515320334252, "noul": 33.33333333333333, "score": -31.57413113026959 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 0.4935502236767717 }, "risk_assessment": { "accuracy": 0.1111111111111111, "cc": -49.23270679758713, "cc_by_type": { "choice": -18.226600985221676, "noul": -100.0, "score": -29.471519407539716 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.08695652173913043, "cc": -38.04052057494357, "cc_by_type": { "choice": -15.942028985507239, "noul": -66.66666666666667, "score": -31.5128660726568 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 0.0 }, "search_retrieval": { "accuracy": 0.20833333333333334, "cc": -9.065556816294313, "cc_by_type": { "choice": 3.914590747330961, "noul": 0.0, "score": -31.111261196213903 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 0.0 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 13.750064644759421, 22.050930135087324 ], "I": [ 0.8594111982295949, 5.031965364102468 ], "capability": [ 7.882175735762922, 12.89515224132689 ], "composite": { "A": [ 0.0009450466124014096, 0.14957124870860547 ], "B": [ 0.000611407847685655, 0.10775944401732225 ], "C": [ 0.0006562823697232012, 0.10386892271430934 ] }, "seed": 17 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 17.26803648238681, "capability": 9.854654871580568, "intelligence": 2.4412732607743255 }, "per_type": { "choice": { "accuracy": 0.272, "calibration": 1.67473081339361, "cc": 5.968349949291043, "cc_by_type": { "choice": 5.968349949291043 }, "cc_tiered": 7.3238197823229765, "cc_tiered_open": 8.982230802737568, "cc_tiered_sealed": 5.6654087619083855, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 5.968349949291043 }, "noul": { "accuracy": 0.328, "calibration": 13.545463846169902, "cc": -34.4, "cc_by_type": { "noul": -34.4 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 0.0 }, "score": { "accuracy": 0.24533333333333332, "calibration": 36.583914787596925, "cc": -29.344915811166672, "cc_by_type": { "score": -29.344915811166672 }, "cc_tiered": 0.0, "cc_tiered_open": 0.0, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "raw_on_A": { "calibration": 19.70003977436594, "capability": 11.004198815490696, "intelligence": 2.3083578566154506 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.014666666666666717, "n": 750 }, "noul": { "invalid_rate": 0.008000000000000007, "n": 375 }, "score": { "invalid_rate": 0.008000000000000007, "n": 375 } }, "views": { "Balanced 33:33:33": 0.01650387186472358, "Emphasis on Accuracy 60:20:20": 0.009516032366765197, "Emphasis on Cost 20:20:60": 0.026162982329893978, "Emphasis on Speed 20:60:20": 0.025995787379506426, "Intelligence only": 0.005819814930094012 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "guru0381", "axes": { "calibration": 62.82741496733333, "cost": 54.45313889836773, "intelligence": 21.99889248592056, "speed": 91.85485240061512 }, "calibration": { "parts": { "choice": { "ece_hard": 0.06249555555555558, "ece_score": 87.50088888888888, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 87.50088888888888 }, "noul": { "abstention_rate": 0.16533333333333333, "brier": 0.2664339086133331, "ece": 0.2359746666666666, "n": 375, "score": 52.805066666666676 }, "score": { "interval90_coverage": 0.6186666666666667, "interval90_width": 0.2953333333333333, "n": 375, "nrps": 0.21633341307111098, "score": 48.17628934644445, "top_ece": 0.4100704 } }, "score": 62.82741496733333 }, "capability": 42.413153726626945, "composite_ci95": { "A": [ 3.9300167728982482, 13.802386631159255 ], "B": [ 3.1171169998268824, 11.843911105357316 ], "C": [ 2.7291783145126725, 9.584990716082817 ] }, "cost": { "basis": "v1.5-M2 base-model reference: frozen DeepInfra snapshot jevbench-v15-measure-20260925 receipts/DEEPINFRA-MODELS-20260925.json (SHA-256 84eb7926acd4e93b71d695be24898e2ca999b20011d1b66cc0348ce7c6c3546f); listing Qwen/Qwen3.5-2B, USD 0.02/M input and USD 0.10/M output. The released checkpoint names Qwen/Qwen3.5-2B-Base; the frozen DeepInfra listing omits the -Base suffix. The requester cited the Qwen3.5-4B USD 0.03/M input reference, but the matching 2B family listing is used here (precedent classone-qwen3.5-2b). Self-hosted GPU deployment has no provider tariff, so price_kind is estimate. Kev generates no free-form text. However, kev/api.py output_tokens() tokenizes JSON-serialized answers, so reported usage.output_tokens is nonzero for normal answers; it is not zero by construction.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.03297833333333333 }, "display": "Krino-2B", "endpoint_condition": "evaluator-owned Lium GPU pod (H100), offline read-only container (--network none); author's `python3 -m krino.serve` (Kev @fe64b127 serve path, documented per-type temperature map choice 1.95 / noul 0.30 / score 0.50 from krino.json), KEV_TRUNCATE_STATES=1, reference kernels (KEV_FUSED=0: the documented pip install has no flash-linear-attention) Paid fast-lane order 8c7dbe11 (public). Kev architecture (LoRA r16 + pointer head) on Qwen/Qwen3.5-2B-Base @b1485b2f. Runtime: reviewed pytorch 2.11 base + Kev uv.lock closure (torch 2.8.0) installed offline from SHA-256-pinned wheels (PR #335). Kev reports usage.output_tokens as the token count of the serialized answer JSON (nothing is generated); priced as reported at the reference output rate.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 26.37590231932532, "I_sealed": 17.6218826525158, "base": 21.99889248592056, "excess": -1.0855071035461918, "gap": 8.75401966680952, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 36.29349492821419, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 49.40540540540541, "hard": 37.71626297577854, "judge": 32.76203032300594, "standard": 32.18920050230222 } }, "open|noul": { "cc": 21.265010351966872, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 57.14285714285714, "hard": 13.043478260869556, "judge": -9.999999999999998, "standard": 66.66666666666667 } }, "open|score": { "cc": 21.5692016777949, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 51.23363636363638, "hard": 9.25462962962964, "judge": 16.76983050847457, "standard": 38.56518518518517 } }, "sealed|choice": { "cc": 36.28277020740255, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 46.632782719186785, "hard": 30.785243437030175, "judge": 35.651322233104814, "standard": 43.049989453701755 } }, "sealed|noul": { "cc": 16.58287775014486, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 24.13793103448276, "hard": 9.090909090909083, "judge": 26.15384615384615, "standard": 13.432835820895516 } }, "sealed|score": { "cc": 0.0, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": -5.103771929824585, "hard": -28.450816901408405, "judge": -18.68588541666667, "standard": 12.30559090909088 } } } }, "jevbench_score": 8.54428010154306, "key": "krino-2b", "last_measured_on": "2026-10-12", "licence": "Apache-2.0 (adapter, head and code; Kev Apache-2.0; base Qwen3.5-2B-Base Apache-2.0)", "listing": "ranked", "measured_in": "v1.6.10", "measurement_date_status": "measured", "model_pin": "Guru0381/krino-2b@ff185abf129206174c67ebbe6058cab4cc9ccaed", "noul_decisive": { "acc_among_decisive": 0.7124600638977636, "decisive": 313, "decisive_rate": 0.8346666666666667, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 7, "ranked": true, "ranks": { "A": 7, "B": 7, "C": 7, "capability": 7 }, "repo": "https://huggingface.co/Guru0381/krino-2b", "scores": { "A": 8.54428010154306, "B": 7.11267113145192, "C": 5.933527848293792 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.21046640197746455, "p50_s_raw": 0.03023320098873228, "p95_s_adjusted": 0.30997879329370326, "p95_s_raw": 0.07998939664685163 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.45454545454545453, "cc": -3.9711111111111044, "cc_by_type": { "choice": 16.666666666666668, "noul": 50.0, "score": -78.57999999999998 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 0.0 }, "calibration_gen": { "accuracy": 0.504950495049505, "cc": 31.44464464033227, "cc_by_type": { "choice": 49.09862142099682, "noul": 52.0, "score": -6.764687500000011 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 31.44464464033227 }, "composite_gen": { "accuracy": 0.36633663366336633, "cc": -31.327821902578023, "cc_by_type": { "choice": 41.463414634146346, "noul": -11.111111111111116, "score": -124.3357692307693 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 0.0 }, "fairness": { "accuracy": 0.6041666666666666, "cc": 48.12166666666667, "cc_by_type": { "choice": 100.0, "noul": 6.666666666666665, "score": 37.69833333333336 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 48.12166666666667 }, "jaggedness": { "accuracy": 0.35, "cc": 1.3896814773217478, "cc_by_type": { "choice": 17.70933589990377, "noul": -18.918918918918916, "score": 5.378627450980389 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 1.3896814773217478 }, "llm_router": { "accuracy": 0.670926517571885, "cc": 54.443170038415836, "cc_by_type": { "choice": 50.86730650700009, "noul": 64.94845360824742, "score": 47.51374999999999 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 54.443170038415836 }, "long_state": { "accuracy": 0.26666666666666666, "cc": -24.669193548387096, "cc_by_type": { "choice": 9.677419354838706, "noul": -100.0, "score": 16.315 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.2222222222222222, "cc": -23.08969072164948, "cc_by_type": { "choice": 1.0309278350515454, "noul": -25.0, "score": -45.29999999999998 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 0.0 }, "multilingual": { "accuracy": 0.5018181818181818, "cc": 15.536204017083938, "cc_by_type": { "choice": 39.3544179980437, "noul": 13.888888888888884, "score": -6.634694835680777 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 15.536204017083938 }, "negation_pair_gen": { "accuracy": 0.38372093023255816, "cc": -51.32922222222222, "cc_by_type": { "choice": 26.666666666666668, "noul": -13.33333333333333, "score": -167.321 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 0.0 }, "option_order": { "accuracy": 0.375, "cc": 20.106524633821575, "cc_by_type": { "choice": 20.106524633821575 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 20.106524633821575 }, "perturb_pair": { "accuracy": 0.625, "cc": 45.63698198198199, "cc_by_type": { "choice": 45.94594594594596, "noul": 60.00000000000001, "score": 30.964999999999996 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 45.63698198198199 }, "usecase": { "accuracy": 0.41363636363636364, "cc": 5.069023006535965, "cc_by_type": { "choice": 31.372549019607874, "noul": -12.5, "score": -3.6654799999999765 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 5.069023006535965 } }, "lang": { "ar": { "accuracy": 0.1875, "cc": -33.39369963369963, "cc_by_type": { "choice": -14.285714285714286, "noul": -50.0, "score": -35.895384615384614 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 0.0 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 0.46153846153846156, "cc": -9.327886840757811, "cc_by_type": { "choice": 79.26267281105991, "noul": -33.333333333333336, "score": -73.91300000000001 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 0.0 }, "de": { "accuracy": 0.625, "cc": 45.03688341669872, "cc_by_type": { "choice": 52.21238938053097, "noul": 50.0, "score": 32.89826086956522 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 45.03688341669872 }, "el": { "accuracy": 0.26666666666666666, "cc": 12.071002482929854, "cc_by_type": { "choice": -0.5586592178770929, "noul": -33.333333333333336, "score": 70.10499999999999 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 12.071002482929854 }, "en": { "accuracy": 0.4778453518679409, "cc": 15.730797082087726, "cc_by_type": { "choice": 35.85628684053157, "noul": 17.266187050359715, "score": -5.93008264462811 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 15.730797082087726 }, "es": { "accuracy": 0.5483870967741935, "cc": 11.50404119886126, "cc_by_type": { "choice": 51.06035889070147, "noul": 75.0, "score": -91.54823529411769 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 11.50404119886126 }, "fi": { "accuracy": 0.375, "cc": 4.108670309653917, "cc_by_type": { "choice": 1.6393442622950836, "noul": 33.33333333333333, "score": -22.64666666666666 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 4.108670309653917 }, "fr": { "accuracy": 0.42857142857142855, "cc": 11.742827657378745, "cc_by_type": { "choice": 36.47058823529412, "noul": 0.0, "score": -1.2421052631578888 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 11.742827657378745 }, "hi": { "accuracy": 0.38095238095238093, "cc": -23.99129422718808, "cc_by_type": { "choice": 16.201117318435756, "noul": 0.0, "score": -88.175 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 0.0 }, "id": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0, "noul": 100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 100.0 }, "it": { "accuracy": 0.8, "cc": 18.527619047619037, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": -144.4171428571429 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 18.527619047619037 }, "ja": { "accuracy": 0.52, "cc": 21.515655458476743, "cc_by_type": { "choice": 14.2176330420969, "noul": 39.99999999999999, "score": 10.329333333333345 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 21.515655458476743 }, "ko": { "accuracy": 0.3, "cc": -12.233253968253969, "cc_by_type": { "choice": -33.333333333333336, "noul": 0.0, "score": -3.36642857142857 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 0.0 }, "mixed": { "accuracy": 0.7857142857142857, "cc": 79.685, "cc_by_type": { "choice": 66.66666666666667, "noul": 100.0, "score": 72.38833333333332 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 79.685 }, "nl": { "accuracy": 0.5, "cc": 26.801093795093788, "cc_by_type": { "choice": 48.05194805194805, "noul": 33.33333333333333, "score": -0.9820000000000162 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 26.801093795093788 }, "no": { "accuracy": 0.0, "cc": -64.51612903225806, "cc_by_type": { "choice": -29.032258064516128, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 0.0 }, "pl": { "accuracy": 0.6923076923076923, "cc": 42.704345349985395, "cc_by_type": { "choice": 84.76512907321202, "noul": 39.99999999999999, "score": 3.3479069767441416 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 42.704345349985395 }, "pt": { "accuracy": 0.5, "cc": 21.2760213786505, "cc_by_type": { "choice": 4.537418974661164, "noul": 60.00000000000001, "score": -0.7093548387096638 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 21.2760213786505 }, "sv": { "accuracy": 0.2727272727272727, "cc": 4.250780141843975, "cc_by_type": { "choice": -2.127659574468081, "noul": -19.999999999999996, "score": 34.88 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 4.250780141843975 }, "tr": { "accuracy": 0.5384615384615384, "cc": 3.729202898550722, "cc_by_type": { "choice": 34.78260869565218, "noul": 50.0, "score": -73.59500000000001 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 3.729202898550722 }, "uk": { "accuracy": 0.7142857142857143, "cc": 77.39897849462365, "cc_by_type": { "choice": 67.74193548387096, "noul": 100.0, "score": 64.455 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 77.39897849462365 }, "zh": { "accuracy": 0.5, "cc": 3.433069679849351, "cc_by_type": { "choice": 49.152542372881356, "noul": -33.333333333333336, "score": -5.519999999999969 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 3.433069679849351 } }, "type": { "choice": { "accuracy": 0.512, "cc": 36.96779502095333, "cc_by_type": { "choice": 36.96779502095333 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 36.96779502095333 }, "noul": { "accuracy": 0.5946666666666667, "cc": 18.933333333333337, "cc_by_type": { "noul": 18.933333333333337 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 18.933333333333337 }, "score": { "accuracy": 0.328, "cc": -6.174584837544961, "cc_by_type": { "score": -6.174584837544961 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "usecase": { "advertising": { "accuracy": 0.3181818181818182, "cc": -42.54725490196079, "cc_by_type": { "choice": 36.0, "noul": -50.0, "score": -113.64176470588237 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 0.0 }, "code_linting": { "accuracy": 0.3333333333333333, "cc": -12.739704039921422, "cc_by_type": { "choice": 6.832298136645961, "noul": -33.333333333333336, "score": -11.718076923076893 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 0.0 }, "customer_support": { "accuracy": 0.7857142857142857, "cc": 55.12989159891598, "cc_by_type": { "choice": 67.47967479674797, "noul": 100.0, "score": -2.090000000000014 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 55.12989159891598 }, "demand_forecasting": { "accuracy": 0.47058823529411764, "cc": 34.89788499025341, "cc_by_type": { "choice": 18.128654970760234, "noul": 100.0, "score": -13.43500000000002 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 34.89788499025341 }, "ecommerce_marketplaces": { "accuracy": 0.4117647058823529, "cc": 0.4748007246376762, "cc_by_type": { "choice": 42.02898550724638, "noul": -33.333333333333336, "score": -7.271250000000018 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 0.4748007246376762 }, "feature_extraction": { "accuracy": 0.15789473684210525, "cc": -116.71251572327047, "cc_by_type": { "choice": -13.207547169811319, "noul": -100.0, "score": -236.9300000000001 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.5, "cc": 11.735613014436538, "cc_by_type": { "choice": 50.98039215686275, "noul": -42.85714285714286, "score": 27.08358974358973 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 11.735613014436538 }, "gaming": { "accuracy": 0.18181818181818182, "cc": -12.115642074433163, "cc_by_type": { "choice": -14.98257839721254, "noul": -50.0, "score": 28.63565217391304 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 0.0 }, "insurance_claims": { "accuracy": 0.4666666666666667, "cc": 26.11814814814815, "cc_by_type": { "choice": 16.666666666666668, "noul": 60.00000000000001, "score": 1.687777777777777 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 26.11814814814815 }, "knowledge_graphs": { "accuracy": 0.36363636363636365, "cc": -3.2785867237687394, "cc_by_type": { "choice": 10.064239828693788, "noul": 0.0, "score": -19.900000000000006 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 0.0 }, "lead_generation": { "accuracy": 0.6, "cc": 31.765082217524082, "cc_by_type": { "choice": 76.74418604651163, "noul": 33.33333333333333, "score": -14.782272727272705 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 31.765082217524082 }, "legal_compliance": { "accuracy": 0.5, "cc": 9.02948476702508, "cc_by_type": { "choice": 35.483870967741936, "noul": 33.33333333333333, "score": -41.728750000000026 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 9.02948476702508 }, "llm_guardrails": { "accuracy": 0.2962962962962963, "cc": -19.13207400298573, "cc_by_type": { "choice": 30.9021113243762, "noul": -60.0, "score": -28.29833333333338 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 0.0 }, "model_routing": { "accuracy": 0.6728110599078341, "cc": 52.803651357323226, "cc_by_type": { "choice": 59.0909090909091, "noul": 60.606060606060595, "score": 38.713984374999995 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 52.803651357323226 }, "moderation_trust_safety": { "accuracy": 0.5, "cc": 16.1079067404313, "cc_by_type": { "choice": 42.41842610364683, "noul": 0.0, "score": 5.905294117647076 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 16.1079067404313 }, "recruiting": { "accuracy": 0.3333333333333333, "cc": -14.35318239173391, "cc_by_type": { "choice": 33.147632311977716, "noul": -33.333333333333336, "score": -42.87384615384611 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 0.0 }, "risk_assessment": { "accuracy": 0.2222222222222222, "cc": -30.597434560030916, "cc_by_type": { "choice": 11.330049261083742, "noul": -50.0, "score": -53.12235294117649 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 0.0 }, "scientific_discovery": { "accuracy": 0.391304347826087, "cc": 11.78285507246376, "cc_by_type": { "choice": 27.536231884057973, "noul": 0.0, "score": 7.81233333333331 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 11.78285507246376 }, "search_retrieval": { "accuracy": 0.5416666666666666, "cc": 29.112966223387648, "cc_by_type": { "choice": 67.97153024911032, "noul": 0.0, "score": 19.367368421052632 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 29.112966223387648 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 56.57165621991852, 64.94080235151694 ], "I": [ 16.253816795278865, 26.657245614240708 ], "capability": [ 37.54870558262, 45.06351151789616 ], "composite": { "A": [ 3.9300167728982482, 13.802386631159255 ], "B": [ 3.1171169998268824, 11.843911105357316 ], "C": [ 2.7291783145126725, 9.584990716082817 ] }, "seed": 18 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 62.82741496733333, "capability": 42.413153726626945, "intelligence": 21.99889248592056 }, "per_type": { "choice": { "accuracy": 0.512, "calibration": 87.50088888888888, "cc": 36.96779502095333, "cc_by_type": { "choice": 36.96779502095333 }, "cc_tiered": 36.28813256780837, "cc_tiered_open": 36.29349492821419, "cc_tiered_sealed": 36.28277020740255, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 36.96779502095333 }, "noul": { "accuracy": 0.5946666666666667, "calibration": 52.805066666666676, "cc": 18.933333333333337, "cc_by_type": { "noul": 18.933333333333337 }, "cc_tiered": 18.923944051055866, "cc_tiered_open": 21.265010351966872, "cc_tiered_sealed": 16.58287775014486, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 18.933333333333337 }, "score": { "accuracy": 0.328, "calibration": 48.17628934644445, "cc": -6.174584837544961, "cc_by_type": { "score": -6.174584837544961 }, "cc_tiered": 10.78460083889745, "cc_tiered_open": 21.5692016777949, "cc_tiered_sealed": 0.0, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 0.0 } }, "raw_on_A": { "calibration": 63.43795542912849, "capability": 42.556073742724294, "intelligence": 21.674192056320095 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 7.773486221374046, "Emphasis on Accuracy 60:20:20": 5.844057834080417, "Emphasis on Cost 20:20:60": 8.6856620739047, "Emphasis on Speed 20:60:20": 10.03200434666341, "Intelligence only": 4.258556788204026 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "MeTask (Hangzhou MeTask Technology Co., Ltd.)", "axes": { "calibration": 87.91805023499681, "cost": 51.488678556745086, "intelligence": 67.8255189476327, "speed": 73.37579625429389 }, "calibration": { "parts": { "choice": { "ece_hard": 0.04579106842955546, "ece_score": 90.8417863140889, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 90.8417863140889 }, "noul": { "abstention_rate": 0.0, "brier": 0.12338284879474153, "ece": 0.05903423423423422, "n": 375, "score": 88.19315315315316 }, "score": { "interval90_coverage": 0.9226666666666666, "interval90_width": 0.28755555555555545, "n": 375, "nrps": 0.07999688319975698, "score": 84.71921123774837, "top_ece": 0.11280944602263793 } }, "score": 87.91805023499681 }, "capability": 77.87178459131476, "composite_ci95": { "A": [ 65.89296743559949, 68.53799315979285 ], "B": [ 65.2339463603177, 69.10024716281225 ], "C": [ 65.89296743559949, 68.53799315979285 ] }, "cost": { "basis": "v1.5-M2 base-model reference: base google/gemma-4-12B-it (author int4 AWQ of google/gemma-4-12B-it-qat-q4_0-unquantized + own LoRA). google/gemma-4-12b-it is not listed on OpenRouter; nearest listed sibling of the same family and size class (fixed Gemma-4-12B precedent) OpenRouter google/gemma-3-12b-it USD 0.05/M input, USD 0.15/M output, read 2026-10-12T00:13:38Z from https://openrouter.ai/api/v1/models x the service's own measured usage tokens; estimated, not charged.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.041404166666666666 }, "display": "metask-jev-rain-12B-trained", "endpoint_condition": "evaluator-owned Lium GPU pod (H100 80GB), offline read-only container (--network none); author's own service (serve.sh -> launch.py -> server.py on vLLM 0.31.0, pinned digest image), unchanged defaults Paid fast-lane order 66c7b6d2 (public). Own LoRA adapter (rank 16) on the author's int4 Gemma 4 12B base, applied unmerged by vLLM. Replaces nothing: the earlier metask_jev_rain_12b row (third-party AWQ wrapper, order f1b81eec) is a different system.", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 73.28803588458365, "I_sealed": 62.36300201068175, "base": 67.8255189476327, "excess": 1.0855071035461918, "gap": 10.925033873901903, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 80.46364135488923, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 92.21621621621622, "hard": 79.2387543252595, "judge": 80.22412656558998, "standard": 77.39640016743407 } }, "open|noul": { "cc": 79.59903381642512, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 82.6086956521739, "judge": 70.0, "standard": 77.77777777777777 } }, "open|score": { "cc": 59.80143248243659, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 76.59298352366106, "hard": 57.8351755760549, "judge": 64.35617443717419, "standard": 48.50605784248132 } }, "sealed|choice": { "cc": 69.0735944674903, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 84.75222363405337, "hard": 65.65282757025557, "judge": 62.97747306562194, "standard": 77.2199957814807 } }, "sealed|noul": { "cc": 66.55391237686658, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 82.75862068965519, "hard": 52.727272727272734, "judge": 69.23076923076923, "standard": 82.08955223880596 } }, "sealed|score": { "cc": 51.46149918768837, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 62.57017720623247, "hard": 51.50741442798628, "judge": 47.129277578702755, "standard": 52.31366211129891 } } } }, "jevbench_score": 67.60392832716802, "key": "metask_jev_rain_12b_trained", "last_measured_on": "2026-10-12", "licence": "Weights: Apache-2.0 per model card (base Gemma 4 under the Gemma 4 licence); service code MIT (github.com/chenyuwang166-oss/metask-jev-rain-12B-trained@e36c7a54)", "listing": "ranked", "measured_in": "v1.6.10", "measurement_date_status": "measured", "model_pin": "MeTask-Rain/metask-jev-rain-12B-trained@cf3706fdd57e23025a8c97fbde86bd5b4a547010", "noul_decisive": { "acc_among_decisive": 0.856, "decisive": 375, "decisive_rate": 1.0, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 3, "ranked": true, "ranks": { "A": 3, "B": 1, "C": 2, "capability": 2 }, "repo": "https://huggingface.co/MeTask-Rain/metask-jev-rain-12B-trained", "scores": { "A": 67.60392832716802, "B": 67.64813054324519, "C": 67.60392832716802 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.3790490360464901, "p50_s_raw": 0.11452451802324504, "p95_s_adjusted": 12.126207013113888, "p95_s_raw": 5.988103506556944 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 1.0, "cc": 87.84887353749237, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 63.546620612477135 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 87.84887353749237 }, "calibration_gen": { "accuracy": 0.6435643564356436, "cc": 56.940570475966105, "cc_by_type": { "choice": 45.91728525980912, "noul": 76.0, "score": 48.9044261680892 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 56.940570475966105 }, "composite_gen": { "accuracy": 0.7821782178217822, "cc": 64.2194805783471, "cc_by_type": { "choice": 64.22764227642277, "noul": 55.55555555555556, "score": 72.875243903063 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 64.2194805783471 }, "fairness": { "accuracy": 0.9375, "cc": 89.4731058334587, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 68.41931750037614 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 89.4731058334587 }, "jaggedness": { "accuracy": 0.6555555555555556, "cc": 51.15173057621822, "cc_by_type": { "choice": 53.801732435033685, "noul": 56.75675675675676, "score": 42.8967025368642 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 51.15173057621822 }, "llm_router": { "accuracy": 0.8434504792332268, "cc": 72.24878717449957, "cc_by_type": { "choice": 80.18843004314519, "noul": 87.62886597938144, "score": 48.92906550097207 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 72.24878717449957 }, "long_state": { "accuracy": 0.26666666666666666, "cc": -31.382028062735532, "cc_by_type": { "choice": 9.677419354838706, "noul": -100.0, "score": -3.8235035430453035 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 0.0 }, "ml_usecase": { "accuracy": 0.5555555555555556, "cc": 31.665606480992682, "cc_by_type": { "choice": 38.14432989690721, "noul": 50.0, "score": 6.85248954607085 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 31.665606480992682 }, "multilingual": { "accuracy": 0.8036363636363636, "cc": 70.94558775850952, "cc_by_type": { "choice": 75.54613628953375, "noul": 77.77777777777777, "score": 59.51284920821705 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 70.94558775850952 }, "negation_pair_gen": { "accuracy": 0.7790697674418605, "cc": 55.42410495913857, "cc_by_type": { "choice": 83.33333333333333, "noul": 39.99999999999999, "score": 42.9389815440824 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 55.42410495913857 }, "option_order": { "accuracy": 0.9583333333333334, "cc": 94.67376830892145, "cc_by_type": { "choice": 94.67376830892145 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 94.67376830892145 }, "perturb_pair": { "accuracy": 0.859375, "cc": 80.61760708445547, "cc_by_type": { "choice": 78.97897897897899, "noul": 100.0, "score": 62.873842274387414 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 80.61760708445547 }, "usecase": { "accuracy": 0.740909090909091, "cc": 61.56342140022497, "cc_by_type": { "choice": 69.36274509803921, "noul": 58.33333333333333, "score": 56.994185769302376 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 61.56342140022497 } }, "lang": { "ar": { "accuracy": 0.5, "cc": 24.545761058228774, "cc_by_type": { "choice": 42.857142857142854, "noul": 50.0, "score": -19.21985968245654 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 24.545761058228774 }, "cs": { "accuracy": 0.7142857142857143, "cc": 78.94736842105263, "cc_by_type": { "choice": 57.89473684210526, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 78.94736842105263 }, "da": { "accuracy": 0.7692307692307693, "cc": 79.27519011200793, "cc_by_type": { "choice": 79.26267281105991, "noul": 100.0, "score": 58.56289752496387 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 79.27519011200793 }, "de": { "accuracy": 0.90625, "cc": 85.48969822591152, "cc_by_type": { "choice": 84.07079646017698, "noul": 100.0, "score": 72.39829821755762 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 85.48969822591152 }, "el": { "accuracy": 0.8, "cc": 76.94633377862073, "cc_by_type": { "choice": 66.4804469273743, "noul": 66.66666666666667, "score": 97.6918877418212 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 76.94633377862073 }, "en": { "accuracy": 0.7741094700260643, "cc": 65.19896688108702, "cc_by_type": { "choice": 71.88220793009603, "noul": 68.34532374100719, "score": 55.36936897215781 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 65.19896688108702 }, "es": { "accuracy": 0.8387096774193549, "cc": 73.5487504069601, "cc_by_type": { "choice": 90.2120717781403, "noul": 100.0, "score": 30.434179442740017 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 73.5487504069601 }, "fi": { "accuracy": 0.625, "cc": 65.70550033996419, "cc_by_type": { "choice": 1.6393442622950836, "noul": 100.0, "score": 95.47715675759747 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 65.70550033996419 }, "fr": { "accuracy": 0.8571428571428571, "cc": 83.74288210729817, "cc_by_type": { "choice": 78.82352941176471, "noul": 100.0, "score": 72.40511691012979 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 83.74288210729817 }, "hi": { "accuracy": 0.8095238095238095, "cc": 59.09382926533001, "cc_by_type": { "choice": 83.24022346368716, "noul": 100.0, "score": -5.958735667697135 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 59.09382926533001 }, "id": { "accuracy": 0.75, "cc": 77.77777777777777, "cc_by_type": { "choice": 55.55555555555555, "noul": 100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 77.77777777777777 }, "it": { "accuracy": 0.7333333333333333, "cc": 48.56239452957701, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": -40.28319832865008 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 48.56239452957701 }, "ja": { "accuracy": 0.64, "cc": 45.094346455841844, "cc_by_type": { "choice": 42.81175536139793, "noul": 39.99999999999999, "score": 52.471284006127604 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 45.094346455841844 }, "ko": { "accuracy": 0.5, "cc": 34.600352011262636, "cc_by_type": { "choice": 55.55555555555555, "noul": 0.0, "score": 48.24550047823236 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 34.600352011262636 }, "mixed": { "accuracy": 1.0, "cc": 98.74153284520177, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 96.22459853560528 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 98.74153284520177 }, "nl": { "accuracy": 0.8, "cc": 48.51424476202488, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 12.209400952741323 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 48.51424476202488 }, "no": { "accuracy": 0.8333333333333334, "cc": 67.74193548387098, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 67.74193548387098 }, "pl": { "accuracy": 0.8205128205128205, "cc": 71.6454228057721, "cc_by_type": { "choice": 84.76512907321202, "noul": 80.0, "score": 50.17113934410426 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 71.6454228057721 }, "pt": { "accuracy": 0.8571428571428571, "cc": 81.35337503204035, "cc_by_type": { "choice": 78.78609310548026, "noul": 100.0, "score": 65.27403199064081 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 81.35337503204035 }, "sv": { "accuracy": 0.45454545454545453, "cc": 37.10841544635961, "cc_by_type": { "choice": -27.659574468085104, "noul": 60.00000000000001, "score": 78.98482080716391 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 37.10841544635961 }, "tr": { "accuracy": 0.8461538461538461, "cc": 54.20537342574059, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": -37.38387972277823 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 54.20537342574059 }, "uk": { "accuracy": 1.0, "cc": 96.22347206389003, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 88.67041619167007 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 96.22347206389003 }, "zh": { "accuracy": 0.8, "cc": 37.32160001214934, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": -21.368533296885307 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 37.32160001214934 } }, "type": { "choice": { "accuracy": 0.7853333333333333, "cc": 72.27271857479094, "cc_by_type": { "choice": 72.27271857479094 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 72.27271857479094 }, "noul": { "accuracy": 0.856, "cc": 71.2, "cc_by_type": { "noul": 71.2 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 71.2 }, "score": { "accuracy": 0.6773333333333333, "cc": 54.022025771730455, "cc_by_type": { "score": 54.022025771730455 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.022025771730455 } }, "usecase": { "advertising": { "accuracy": 0.6363636363636364, "cc": 39.183979710642106, "cc_by_type": { "choice": 68.0, "noul": 25.0, "score": 24.551939131926314 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 39.183979710642106 }, "code_linting": { "accuracy": 0.7083333333333334, "cc": 51.97732888297762, "cc_by_type": { "choice": 81.36645962732919, "noul": 77.77777777777777, "score": -3.2122507561741065 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 51.97732888297762 }, "customer_support": { "accuracy": 0.7857142857142857, "cc": 42.34789039430175, "cc_by_type": { "choice": 83.73983739837398, "noul": 50.0, "score": -6.696166215468713 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 42.34789039430175 }, "demand_forecasting": { "accuracy": 0.6470588235294118, "cc": 43.93438993611236, "cc_by_type": { "choice": 53.21637426900585, "noul": 33.33333333333333, "score": 45.253462205997906 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 43.93438993611236 }, "ecommerce_marketplaces": { "accuracy": 0.6470588235294118, "cc": 30.104630020806553, "cc_by_type": { "choice": 56.52173913043478, "noul": -33.333333333333336, "score": 67.12548426531822 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 30.104630020806553 }, "feature_extraction": { "accuracy": 0.5789473684210527, "cc": -40.6229056009301, "cc_by_type": { "choice": 49.685534591194966, "noul": 33.33333333333333, "score": -204.8875847273186 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 0.0 }, "financial_crime": { "accuracy": 0.7, "cc": 60.81399584070914, "cc_by_type": { "choice": 80.3921568627451, "noul": 42.85714285714286, "score": 59.192687802239476 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 60.81399584070914 }, "gaming": { "accuracy": 0.7727272727272727, "cc": 51.12928904620589, "cc_by_type": { "choice": 89.54703832752612, "noul": 0.0, "score": 63.840828811091555 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 51.12928904620589 }, "insurance_claims": { "accuracy": 0.6666666666666666, "cc": 65.41586316119422, "cc_by_type": { "choice": 16.666666666666668, "noul": 100.0, "score": 79.580922816916 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 65.41586316119422 }, "knowledge_graphs": { "accuracy": 0.6818181818181818, "cc": 53.44925983148553, "cc_by_type": { "choice": 35.76017130620985, "noul": 66.66666666666667, "score": 57.92094152158005 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 53.44925983148553 }, "lead_generation": { "accuracy": 0.84, "cc": 72.23152413124878, "cc_by_type": { "choice": 88.3720930232558, "noul": 100.0, "score": 28.322479370490505 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 72.23152413124878 }, "legal_compliance": { "accuracy": 0.5, "cc": 14.752066065373555, "cc_by_type": { "choice": 35.483870967741936, "noul": 33.33333333333333, "score": -24.561006104954597 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 14.752066065373555 }, "llm_guardrails": { "accuracy": 0.8518518518518519, "cc": 72.19021405060475, "cc_by_type": { "choice": 88.48368522072937, "noul": 60.00000000000001, "score": 68.08695693108486 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 72.19021405060475 }, "model_routing": { "accuracy": 0.8755760368663594, "cc": 74.29999007860982, "cc_by_type": { "choice": 90.64935064935067, "noul": 96.96969696969697, "score": 35.28092261678184 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 74.29999007860982 }, "moderation_trust_safety": { "accuracy": 0.7222222222222222, "cc": 73.75106865956981, "cc_by_type": { "choice": 76.96737044145875, "noul": 100.0, "score": 44.28583553725066 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 73.75106865956981 }, "recruiting": { "accuracy": 0.7777777777777778, "cc": 76.15530393930493, "cc_by_type": { "choice": 49.860724233983284, "noul": 100.0, "score": 78.6051875839315 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 76.15530393930493 }, "risk_assessment": { "accuracy": 0.7222222222222222, "cc": 68.0715834514785, "cc_by_type": { "choice": 55.665024630541865, "noul": 100.0, "score": 48.54972572389362 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 68.0715834514785 }, "scientific_discovery": { "accuracy": 0.4782608695652174, "cc": 30.843083819184248, "cc_by_type": { "choice": -1.4492753623188352, "noul": 33.33333333333333, "score": 60.645193486538254 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 30.843083819184248 }, "search_retrieval": { "accuracy": 0.9583333333333334, "cc": 94.96186264566033, "cc_by_type": { "choice": 89.3238434163701, "noul": 100.0, "score": 95.56174452061086 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 94.96186264566033 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 83.03489436852567, 89.51334618625602 ], "I": [ 62.79944603305875, 71.69687505938181 ], "capability": [ 73.86103956693225, 79.89646066619623 ], "composite": { "A": [ 65.89296743559949, 68.53799315979285 ], "B": [ 65.2339463603177, 69.10024716281225 ], "C": [ 65.89296743559949, 68.53799315979285 ] }, "seed": 19 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 87.91805023499681, "capability": 77.87178459131476, "intelligence": 67.8255189476327 }, "per_type": { "choice": { "accuracy": 0.7853333333333333, "calibration": 90.8417863140889, "cc": 72.27271857479094, "cc_by_type": { "choice": 72.27271857479094 }, "cc_tiered": 74.76861791118976, "cc_tiered_open": 80.46364135488923, "cc_tiered_sealed": 69.0735944674903, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 72.27271857479094 }, "noul": { "accuracy": 0.856, "calibration": 88.19315315315316, "cc": 71.2, "cc_by_type": { "noul": 71.2 }, "cc_tiered": 73.07647309664586, "cc_tiered_open": 79.59903381642512, "cc_tiered_sealed": 66.55391237686658, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 71.2 }, "score": { "accuracy": 0.6773333333333333, "calibration": 84.71921123774837, "cc": 54.022025771730455, "cc_by_type": { "score": 54.022025771730455 }, "cc_tiered": 55.63146583506248, "cc_tiered_open": 59.80143248243659, "cc_tiered_sealed": 51.46149918768837, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.022025771730455 } }, "raw_on_A": { "calibration": 86.30134459903176, "capability": 77.02938857798543, "intelligence": 67.75743255693911 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 62.76947628204991, "Emphasis on Accuracy 60:20:20": 64.69865676810258, "Emphasis on Cost 20:20:60": 57.71178234790809, "Emphasis on Speed 20:60:20": 66.62147498913741, "Intelligence only": 67.8255189476327 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "MeTask (Hangzhou MeTask Technology Co., Ltd.)", "axes": { "calibration": 87.09471194592824, "cost": 49.238192537895735, "intelligence": 70.71905450578407, "speed": 83.17884674032656 }, "calibration": { "parts": { "choice": { "ece_hard": 0.057130297017528, "ece_score": 88.5739405964944, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 88.5739405964944 }, "noul": { "abstention_rate": 0.0, "brier": 0.1123272461650836, "ece": 0.06861261261261234, "n": 375, "score": 86.27747747747753 }, "score": { "interval90_coverage": 0.9226666666666666, "interval90_width": 0.2508888888888888, "n": 375, "nrps": 0.07405460121549541, "score": 86.43271776381279, "top_ece": 0.09864552175412443 } }, "score": 87.09471194592824 }, "capability": 78.90688322585615, "composite_ci95": { "A": [ 65.25737914445524, 67.99421159790742 ], "B": [ 65.08829812186133, 68.88224028986754 ], "C": [ 65.25737914445524, 67.99421159790742 ] }, "cost": { "basis": "v1.5-M2 base-model reference: exact base google/gemma-4-31B-it (served as Google's unchanged google/gemma-4-31B-it-qat-w4a16-ct@52f3f65b + own LoRA). OpenRouter google/gemma-4-31b-it list price USD 0.09/M input, USD 0.34/M output (listed since Apr 2026; same reference as the published Jeff, deck31b and Quyet-1.0-Large rows), read 2026-10-12T00:13:38Z from https://openrouter.ai/api/v1/models x the service's own measured usage tokens; estimated, not charged.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.04921078 }, "display": "metask-jev-rain-31B-trained", "endpoint_condition": "evaluator-owned Lium GPU pod (H100 80GB), offline read-only container (--network none); author's own service (service/serve.sh -> launch.py -> server.py on vLLM 0.31.0, pinned digest image), unchanged defaults Paid fast-lane order 147c5e18 (public). Own LoRA adapter (rank 16) on Google's official W4A16 Gemma 4 31B QAT checkpoint, applied unmerged by vLLM; max_model_len 6656, longer inputs are shortened by the service's documented extraction rule (author-tested only on H200; measured on H100 80GB with unchanged settings).", "endpoint_kind": "gpu", "full_coverage": true, "intelligence": { "I_open": 72.5520469583762, "I_sealed": 68.88606205319194, "base": 70.71905450578407, "excess": -6.173541865171451, "gap": 3.66598490518426, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 80.81042171537538, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 100.00000000000001, "hard": 81.12614029569046, "judge": 76.26895187870798, "standard": 77.39640016743407 } }, "open|noul": { "cc": 73.59903381642512, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 82.6086956521739, "judge": 50.0, "standard": 77.77777777777777 } }, "open|score": { "cc": 63.24668534332811, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 70.41765078820325, "hard": 69.57893658072337, "judge": 60.72068725487829, "standard": 50.785697278774734 } }, "sealed|choice": { "cc": 80.73173886349481, "n": { "easy": 70, "hard": 247, "judge": 146, "standard": 137 }, "tiers": { "easy": 94.28208386277001, "hard": 78.142708453799, "judge": 78.84427032321254, "standard": 81.96582999367222 } }, "sealed|noul": { "cc": 74.61818469744357, "n": { "easy": 58, "hard": 110, "judge": 65, "standard": 67 }, "tiers": { "easy": 86.20689655172413, "hard": 67.27272727272727, "judge": 81.53846153846153, "standard": 73.13432835820895 } }, "sealed|score": { "cc": 51.30826259863744, "n": { "easy": 35, "hard": 123, "judge": 67, "standard": 75 }, "tiers": { "easy": 70.15351391347656, "hard": 51.34670276898963, "judge": 46.83933293506929, "standard": 48.51215109586572 } } } }, "jevbench_score": 66.93314450921662, "key": "metask_jev_rain_31b_trained", "last_measured_on": "2026-10-12", "licence": "Weights: Apache-2.0 per model card (base Gemma 4 under the Gemma 4 licence); service code (service/) MIT", "listing": "ranked", "measured_in": "v1.6.10", "measurement_date_status": "measured", "model_pin": "MeTask-Rain/metask-jev-rain-31B-trained@da56acc0bb607cc214c184ad9f1e9231a0d63dee", "noul_decisive": { "acc_among_decisive": 0.8773333333333333, "decisive": 375, "decisive_rate": 1.0, "n": 375, "supported": true, "valid": 375 }, "planned_weight_denominator": 1.0, "rank": 4, "ranked": true, "ranks": { "A": 4, "B": 3, "C": 3, "capability": 1 }, "repo": "https://huggingface.co/MeTask-Rain/metask-jev-rain-31B-trained", "scores": { "A": 66.93314450921662, "B": 67.25625355613948, "C": 66.93314450921662 }, "speed": { "adjustment": "x2 + 0.15 s (assumption, not measured)", "n": 124, "p50_s_adjusted": 0.5687712740385905, "p50_s_raw": 0.20938563701929525, "p95_s_adjusted": 0.8456247233552858, "p95_s_raw": 0.3478123616776429 }, "status": { "answered_ok": 1500, "missing": 0, "rows": 1500, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": false, "api_subset_tag": null, "breakdowns": { "family": { "calibration": { "accuracy": 0.8181818181818182, "cc": 63.16886779201999, "cc_by_type": { "choice": 75.0, "noul": 100.0, "score": 14.506603376059957 }, "low_n": true, "n": 22, "n_by_type": { "choice": 16, "noul": 4, "score": 2 }, "score": 63.16886779201999 }, "calibration_gen": { "accuracy": 0.7722772277227723, "cc": 64.53799979415957, "cc_by_type": { "choice": 77.7306468716861, "noul": 60.00000000000001, "score": 55.88335251079264 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 25, "score": 35 }, "score": 64.53799979415957 }, "composite_gen": { "accuracy": 0.8811881188118812, "cc": 83.81438823065723, "cc_by_type": { "choice": 96.7479674796748, "noul": 92.59259259259258, "score": 62.102604619704316 }, "low_n": false, "n": 101, "n_by_type": { "choice": 41, "noul": 27, "score": 33 }, "score": 83.81438823065723 }, "fairness": { "accuracy": 0.9375, "cc": 86.04066368360208, "cc_by_type": { "choice": 100.0, "noul": 60.00000000000001, "score": 98.12199105080622 }, "low_n": false, "n": 48, "n_by_type": { "choice": 15, "noul": 15, "score": 18 }, "score": 86.04066368360208 }, "jaggedness": { "accuracy": 0.6666666666666666, "cc": 49.85525524192105, "cc_by_type": { "choice": 62.463907603464875, "noul": 51.35135135135136, "score": 35.750506770946934 }, "low_n": false, "n": 180, "n_by_type": { "choice": 94, "noul": 37, "score": 49 }, "score": 49.85525524192105 }, "llm_router": { "accuracy": 0.9297124600638977, "cc": 83.7793054809374, "cc_by_type": { "choice": 93.66029761380645, "noul": 95.8762886597938, "score": 61.801330169211944 }, "low_n": false, "n": 313, "n_by_type": { "choice": 152, "noul": 97, "score": 64 }, "score": 83.7793054809374 }, "long_state": { "accuracy": 0.3333333333333333, "cc": 9.832941925502283, "cc_by_type": { "choice": 9.677419354838706, "noul": 0.0, "score": 19.821406421668144 }, "low_n": true, "n": 15, "n_by_type": { "choice": 10, "noul": 2, "score": 3 }, "score": 9.832941925502283 }, "ml_usecase": { "accuracy": 0.7407407407407407, "cc": 54.59598157228413, "cc_by_type": { "choice": 62.886597938144334, "noul": 75.0, "score": 25.901346778708078 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 8, "score": 8 }, "score": 54.59598157228413 }, "multilingual": { "accuracy": 0.7963636363636364, "cc": 68.20002344704231, "cc_by_type": { "choice": 75.54613628953375, "noul": 69.44444444444444, "score": 59.60948960714872 }, "low_n": false, "n": 275, "n_by_type": { "choice": 134, "noul": 72, "score": 69 }, "score": 68.20002344704231 }, "negation_pair_gen": { "accuracy": 0.872093023255814, "cc": 68.5323347023584, "cc_by_type": { "choice": 93.33333333333333, "noul": 60.00000000000001, "score": 52.263670773741836 }, "low_n": false, "n": 86, "n_by_type": { "choice": 40, "noul": 30, "score": 16 }, "score": 68.5323347023584 }, "option_order": { "accuracy": 0.9791666666666666, "cc": 97.3368841544607, "cc_by_type": { "choice": 97.3368841544607 }, "low_n": false, "n": 48, "n_by_type": { "choice": 48 }, "score": 97.3368841544607 }, "perturb_pair": { "accuracy": 0.875, "cc": 81.60522267243225, "cc_by_type": { "choice": 84.98498498498498, "noul": 100.0, "score": 59.83068303231177 }, "low_n": false, "n": 64, "n_by_type": { "choice": 44, "noul": 10, "score": 10 }, "score": 81.60522267243225 }, "usecase": { "accuracy": 0.7363636363636363, "cc": 64.54982362341306, "cc_by_type": { "choice": 76.7156862745098, "noul": 70.83333333333333, "score": 46.100451262396035 }, "low_n": false, "n": 220, "n_by_type": { "choice": 104, "noul": 48, "score": 68 }, "score": 64.54982362341306 } }, "lang": { "ar": { "accuracy": 0.6875, "cc": 36.07646009918423, "cc_by_type": { "choice": 61.904761904761905, "noul": 50.0, "score": -3.675381607209216 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 4, "score": 5 }, "score": 36.07646009918423 }, "cs": { "accuracy": 0.5714285714285714, "cc": 68.42105263157895, "cc_by_type": { "choice": 36.84210526315789, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 68.42105263157895 }, "da": { "accuracy": 0.8461538461538461, "cc": 73.84108691654622, "cc_by_type": { "choice": 79.26267281105991, "noul": 100.0, "score": 42.260587938578766 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 3, "score": 4 }, "score": 73.84108691654622 }, "de": { "accuracy": 0.90625, "cc": 90.42477813573423, "cc_by_type": { "choice": 84.07079646017698, "noul": 100.0, "score": 87.20353794702571 }, "low_n": false, "n": 32, "n_by_type": { "choice": 16, "noul": 8, "score": 8 }, "score": 90.42477813573423 }, "el": { "accuracy": 0.8, "cc": 77.579631749179, "cc_by_type": { "choice": 66.4804469273743, "noul": 66.66666666666667, "score": 99.59178165349604 }, "low_n": true, "n": 15, "n_by_type": { "choice": 8, "noul": 6, "score": 1 }, "score": 77.579631749179 }, "en": { "accuracy": 0.8227628149435273, "cc": 71.3569930395452, "cc_by_type": { "choice": 83.0853907079484, "noul": 75.53956834532374, "score": 55.44602006536346 }, "low_n": false, "n": 1151, "n_by_type": { "choice": 588, "noul": 278, "score": 285 }, "score": 71.3569930395452 }, "es": { "accuracy": 0.8064516129032258, "cc": 69.16556628426598, "cc_by_type": { "choice": 80.42414355628058, "noul": 100.0, "score": 27.07255529651733 }, "low_n": false, "n": 31, "n_by_type": { "choice": 13, "noul": 8, "score": 10 }, "score": 69.16556628426598 }, "fi": { "accuracy": 0.875, "cc": 88.9244647500369, "cc_by_type": { "choice": 67.21311475409836, "noul": 100.0, "score": 99.56027949601237 }, "low_n": true, "n": 8, "n_by_type": { "choice": 4, "noul": 3, "score": 1 }, "score": 88.9244647500369 }, "fr": { "accuracy": 0.9285714285714286, "cc": 93.75128961226879, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 81.25386883680636 }, "low_n": true, "n": 14, "n_by_type": { "choice": 6, "noul": 2, "score": 6 }, "score": 93.75128961226879 }, "hi": { "accuracy": 0.8571428571428571, "cc": 73.31989019858385, "cc_by_type": { "choice": 99.99999999999999, "noul": 100.0, "score": 19.959670595751543 }, "low_n": true, "n": 21, "n_by_type": { "choice": 8, "noul": 8, "score": 5 }, "score": 73.31989019858385 }, "id": { "accuracy": 0.75, "cc": 0.0, "cc_by_type": { "choice": 100.0, "noul": -100.0 }, "low_n": true, "n": 4, "n_by_type": { "choice": 3, "noul": 1 }, "score": 0.0 }, "it": { "accuracy": 0.8, "cc": 56.92598694485033, "cc_by_type": { "choice": 85.97038191738113, "noul": 100.0, "score": -15.192421082830165 }, "low_n": true, "n": 15, "n_by_type": { "choice": 9, "noul": 3, "score": 3 }, "score": 56.92598694485033 }, "ja": { "accuracy": 0.72, "cc": 46.34982921310412, "cc_by_type": { "choice": 57.108816521048446, "noul": 80.0, "score": 1.9406711182639036 }, "low_n": true, "n": 25, "n_by_type": { "choice": 9, "noul": 10, "score": 6 }, "score": 46.34982921310412 }, "ko": { "accuracy": 0.9, "cc": 60.502271657899314, "cc_by_type": { "choice": 100.0, "noul": 0.0, "score": 81.50681497369794 }, "low_n": true, "n": 10, "n_by_type": { "choice": 3, "noul": 2, "score": 5 }, "score": 60.502271657899314 }, "mixed": { "accuracy": 0.9285714285714286, "cc": 77.73180568607792, "cc_by_type": { "choice": 100.0, "noul": 33.33333333333333, "score": 99.86208372490044 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 77.73180568607792 }, "nl": { "accuracy": 0.5, "cc": 8.642558761906725, "cc_by_type": { "choice": 48.05194805194805, "noul": -33.333333333333336, "score": 11.209061567105461 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 8.642558761906725 }, "no": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0, "noul": 100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 2, "noul": 4 }, "score": 100.0 }, "pl": { "accuracy": 0.8205128205128205, "cc": 63.392816051625765, "cc_by_type": { "choice": 92.382564536606, "noul": 39.99999999999999, "score": 57.79588361827128 }, "low_n": false, "n": 39, "n_by_type": { "choice": 17, "noul": 10, "score": 12 }, "score": 63.392816051625765 }, "pt": { "accuracy": 0.7142857142857143, "cc": 67.80504865797501, "cc_by_type": { "choice": 46.96523276370065, "noul": 100.0, "score": 56.44991321022436 }, "low_n": true, "n": 28, "n_by_type": { "choice": 12, "noul": 5, "score": 11 }, "score": 67.80504865797501 }, "sv": { "accuracy": 0.6363636363636364, "cc": 46.19054683365891, "cc_by_type": { "choice": 48.93617021276595, "noul": 60.00000000000001, "score": 29.63547028821076 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "noul": 5, "score": 1 }, "score": 46.19054683365891 }, "tr": { "accuracy": 0.7692307692307693, "cc": 68.75012962289753, "cc_by_type": { "choice": 78.26086956521739, "noul": 100.0, "score": 27.989519303475184 }, "low_n": true, "n": 13, "n_by_type": { "choice": 6, "noul": 4, "score": 3 }, "score": 68.75012962289753 }, "uk": { "accuracy": 0.5714285714285714, "cc": 61.68878465176133, "cc_by_type": { "choice": 35.483870967741936, "noul": 100.0, "score": 49.582482987542065 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 1, "score": 2 }, "score": 61.68878465176133 }, "zh": { "accuracy": 0.9, "cc": 69.21114832796756, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 7.63344498390266 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 3, "score": 2 }, "score": 69.21114832796756 } }, "type": { "choice": { "accuracy": 0.8573333333333333, "cc": 81.57255209628963, "cc_by_type": { "choice": 81.57255209628963 }, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 81.57255209628963 }, "noul": { "accuracy": 0.8773333333333333, "cc": 75.46666666666665, "cc_by_type": { "noul": 75.46666666666665 }, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 75.46666666666665 }, "score": { "accuracy": 0.6746666666666666, "cc": 54.7684541832536, "cc_by_type": { "score": 54.7684541832536 }, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.7684541832536 } }, "usecase": { "advertising": { "accuracy": 0.7272727272727273, "cc": 48.45861881975295, "cc_by_type": { "choice": 84.0, "noul": 25.0, "score": 36.375856459258834 }, "low_n": true, "n": 22, "n_by_type": { "choice": 8, "noul": 8, "score": 6 }, "score": 48.45861881975295 }, "code_linting": { "accuracy": 0.7916666666666666, "cc": 76.75345303484892, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 30.26035910454675 }, "low_n": true, "n": 24, "n_by_type": { "choice": 7, "noul": 9, "score": 8 }, "score": 76.75345303484892 }, "customer_support": { "accuracy": 0.8571428571428571, "cc": 69.6944643957582, "cc_by_type": { "choice": 83.73983739837398, "noul": 100.0, "score": 25.343555788900616 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 4, "score": 2 }, "score": 69.6944643957582 }, "demand_forecasting": { "accuracy": 0.23529411764705882, "cc": -24.108890369968993, "cc_by_type": { "choice": -5.263157894736844, "noul": -33.333333333333336, "score": -33.7301798818368 }, "low_n": true, "n": 17, "n_by_type": { "choice": 11, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.7647058823529411, "cc": 57.255300551831404, "cc_by_type": { "choice": 71.0144927536232, "noul": 33.33333333333333, "score": 67.41807556853769 }, "low_n": true, "n": 17, "n_by_type": { "choice": 9, "noul": 3, "score": 5 }, "score": 57.255300551831404 }, "feature_extraction": { "accuracy": 0.7894736842105263, "cc": 35.41674609079076, "cc_by_type": { "choice": 74.84276729559748, "noul": 66.66666666666667, "score": -35.25919568989187 }, "low_n": true, "n": 19, "n_by_type": { "choice": 10, "noul": 6, "score": 3 }, "score": 35.41674609079076 }, "financial_crime": { "accuracy": 0.6333333333333333, "cc": 56.61757692065063, "cc_by_type": { "choice": 70.58823529411765, "noul": 71.42857142857142, "score": 27.835924039262792 }, "low_n": false, "n": 30, "n_by_type": { "choice": 13, "noul": 7, "score": 10 }, "score": 56.61757692065063 }, "gaming": { "accuracy": 0.6818181818181818, "cc": 43.298609533332666, "cc_by_type": { "choice": 68.6411149825784, "noul": 0.0, "score": 61.2547136174196 }, "low_n": true, "n": 22, "n_by_type": { "choice": 12, "noul": 4, "score": 6 }, "score": 43.298609533332666 }, "insurance_claims": { "accuracy": 0.4666666666666667, "cc": 40.88581737462815, "cc_by_type": { "choice": 37.5, "noul": 60.00000000000001, "score": 25.15745212388444 }, "low_n": true, "n": 15, "n_by_type": { "choice": 6, "noul": 5, "score": 4 }, "score": 40.88581737462815 }, "knowledge_graphs": { "accuracy": 0.7727272727272727, "cc": 70.06051887452556, "cc_by_type": { "choice": 74.30406852248395, "noul": 100.0, "score": 35.877488101092766 }, "low_n": true, "n": 22, "n_by_type": { "choice": 10, "noul": 6, "score": 6 }, "score": 70.06051887452556 }, "lead_generation": { "accuracy": 0.8, "cc": 64.06806981529785, "cc_by_type": { "choice": 100.0, "noul": 77.77777777777777, "score": 14.426431668115768 }, "low_n": true, "n": 25, "n_by_type": { "choice": 11, "noul": 9, "score": 5 }, "score": 64.06806981529785 }, "legal_compliance": { "accuracy": 0.6, "cc": 49.78201560348273, "cc_by_type": { "choice": 3.2258064516129026, "noul": 100.0, "score": 46.1202403588353 }, "low_n": true, "n": 10, "n_by_type": { "choice": 4, "noul": 3, "score": 3 }, "score": 49.78201560348273 }, "llm_guardrails": { "accuracy": 0.8148148148148148, "cc": 72.7024042480826, "cc_by_type": { "choice": 100.00000000000001, "noul": 60.00000000000001, "score": 58.107212744247796 }, "low_n": true, "n": 27, "n_by_type": { "choice": 11, "noul": 5, "score": 11 }, "score": 72.7024042480826 }, "model_routing": { "accuracy": 0.9032258064516129, "cc": 80.21287919965458, "cc_by_type": { "choice": 91.81818181818183, "noul": 100.0, "score": 48.82045578078191 }, "low_n": false, "n": 217, "n_by_type": { "choice": 101, "noul": 66, "score": 50 }, "score": 80.21287919965458 }, "moderation_trust_safety": { "accuracy": 0.7222222222222222, "cc": 48.98671618683705, "cc_by_type": { "choice": 76.96737044145875, "noul": 0.0, "score": 69.9927781190524 }, "low_n": true, "n": 18, "n_by_type": { "choice": 11, "noul": 2, "score": 5 }, "score": 48.98671618683705 }, "recruiting": { "accuracy": 0.7777777777777778, "cc": 78.10700850764788, "cc_by_type": { "choice": 66.57381615598885, "noul": 100.0, "score": 67.74720936695476 }, "low_n": true, "n": 18, "n_by_type": { "choice": 8, "noul": 3, "score": 7 }, "score": 78.10700850764788 }, "risk_assessment": { "accuracy": 0.7222222222222222, "cc": 56.67924976273154, "cc_by_type": { "choice": 70.44334975369459, "noul": 100.0, "score": -0.40560046549995565 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 56.67924976273154 }, "scientific_discovery": { "accuracy": 0.6521739130434783, "cc": 45.24701491441865, "cc_by_type": { "choice": 56.52173913043478, "noul": 33.33333333333333, "score": 45.88597227948784 }, "low_n": true, "n": 23, "n_by_type": { "choice": 9, "noul": 6, "score": 8 }, "score": 45.24701491441865 }, "search_retrieval": { "accuracy": 1.0, "cc": 99.61559815097014, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 98.84679445291042 }, "low_n": true, "n": 24, "n_by_type": { "choice": 12, "noul": 4, "score": 8 }, "score": 99.61559815097014 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 82.52322020565772, 88.69708403808961 ], "I": [ 66.13238871680062, 74.9857409493447 ], "capability": [ 74.70840037420157, 81.40019831990315 ], "composite": { "A": [ 65.25737914445524, 67.99421159790742 ], "B": [ 65.08829812186133, 68.88224028986754 ], "C": [ 65.25737914445524, 67.99421159790742 ] }, "seed": 20 }, "ci95_n": 1200, "complete": true, "equated": null, "equating_flag": null, "equating_offset": null, "full_set_api": false, "g_med_used": 9.839526770355711, "lane": "selfhosted", "metric_basis": "S u P", "n_items": 1500, "n_public": 300, "n_sealed_side": 1200, "on_S": { "calibration": 87.09471194592824, "capability": 78.90688322585615, "intelligence": 70.71905450578407 }, "per_type": { "choice": { "accuracy": 0.8573333333333333, "calibration": 88.5739405964944, "cc": 81.57255209628963, "cc_by_type": { "choice": 81.57255209628963 }, "cc_tiered": 80.7710802894351, "cc_tiered_open": 80.81042171537538, "cc_tiered_sealed": 80.73173886349481, "low_n": false, "n": 750, "n_by_type": { "choice": 750 }, "score": 81.57255209628963 }, "noul": { "accuracy": 0.8773333333333333, "calibration": 86.27747747747753, "cc": 75.46666666666665, "cc_by_type": { "noul": 75.46666666666665 }, "cc_tiered": 74.10860925693434, "cc_tiered_open": 73.59903381642512, "cc_tiered_sealed": 74.61818469744357, "low_n": false, "n": 375, "n_by_type": { "noul": 375 }, "score": 75.46666666666665 }, "score": { "accuracy": 0.6746666666666666, "calibration": 86.43271776381279, "cc": 54.7684541832536, "cc_by_type": { "score": 54.7684541832536 }, "cc_tiered": 57.277473970982776, "cc_tiered_open": 63.24668534332811, "cc_tiered_sealed": 51.30826259863744, "low_n": false, "n": 375, "n_by_type": { "score": 375 }, "score": 54.7684541832536 } }, "raw_on_A": { "calibration": 83.28617618857919, "capability": 76.3197068643228, "intelligence": 69.35323754006639 }, "set": "S+P" }, "validity": { "choice": { "invalid_rate": 0.0, "n": 750 }, "noul": { "invalid_rate": 0.0, "n": 375 }, "score": { "invalid_rate": 0.0, "n": 375 } }, "views": { "Balanced 33:33:33": 62.60259745903769, "Emphasis on Accuracy 60:20:20": 64.86418326296348, "Emphasis on Cost 20:20:60": 55.67505050802875, "Emphasis on Speed 20:60:20": 68.76094684861118, "Intelligence only": 68.58049910691462 } }, { "addendum": null, "api_flag": true, "api_subset": true, "author": "Microsoft", "axes": { "calibration": 85.15566582345811, "cost": 62.19876038790379, "intelligence": 59.226098571054024, "speed": 88.69804989142763 }, "calibration": { "parts": { "choice": { "ece_hard": 0.1108223199674765, "ece_score": 77.8355360065047, "mean_tvd": null, "n": 296, "n_ece_hard": 128, "n_tvd": 0, "score": 77.8355360065047 }, "noul": { "abstention_rate": 0.22972972972972971, "brier": 0.08789565767934152, "ece": 0.06454285728875721, "n": 148, "score": 87.09142854224856 }, "score": { "interval90_coverage": 0.9194630872483222, "interval90_width": 0.1784116331096197, "n": 149, "nrps": 0.06306645857440242, "score": 87.1325461475575, "top_ece": 0.09714130923722374 } }, "score": 84.01983689877025 }, "canonical_system_key": "microsoft-decision-1", "capability": 72.19088219725606, "composite_ci95": { "A": [ 60.33591615493133, 74.53742586189195 ], "B": [ 56.31152174249009, 73.1804420130518 ], "C": [ 41.89994177425786, 74.53742586189195 ] }, "cost": { "basis": "LEAD DECISION r3: USD 0.042/M input and USD 0/M output, basis of the live Microsoft-Decision-1 (Azure Foundry) row on https://benchmarkheaven.com/jev-models/api. Reported usage across all 600 decisions, including seven HTTP422 refusals retained in denominator; frozen parent all-item cost basis. Invoice, warmup and refusal charges unknown.", "common_basis": null, "kind": "tariff", "usage_estimated_rows": 0, "usd_per_1000": 0.01819874 }, "display": "Microsoft-Decision-1 (Azure Foundry) โ€” A2 re-run", "endpoint_condition": "Vendor-hosted Azure Foundry API; latency measured from Sandy (Germany). Seven of 600 HTTP 422 responses scored wrong. Four self-hosted systems re-run on H200, versus H100 parents; wide equating interval. MeTask 31B and Torchcast not completed after four failed pod attempts; see measurement notes.", "endpoint_kind": "api", "full_coverage": true, "intelligence": { "I_open": 67.00640482707561, "I_sealed": 46.29756060035449, "base": 56.65198271371505, "excess": 3.8882152491360813, "gap": 20.708844226721126, "penalty": 1.0, "per_type_split": { "open|choice": { "cc": 82.78833309663467, "n": { "easy": 33, "hard": 68, "judge": 32, "standard": 17 }, "tiers": { "easy": 96.10810810810811, "hard": 73.57659641396666, "judge": 84.17930125247197, "standard": 92.46546672247803 } }, "open|noul": { "cc": 54.94202898550725, "n": { "easy": 14, "hard": 23, "judge": 20, "standard": 18 }, "tiers": { "easy": 100.0, "hard": 56.52173913043479, "judge": 30.000000000000004, "standard": 66.66666666666667 } }, "open|score": { "cc": 63.288852399084945, "n": { "easy": 9, "hard": 29, "judge": 21, "standard": 16 }, "tiers": { "easy": 52.860607370724196, "hard": 67.50339329867514, "judge": 74.36852911363773, "standard": 43.45437804225577 } }, "sealed|choice": { "cc": 63.30695607434754, "n": { "easy": 15, "hard": 64, "judge": 35, "standard": 36 }, "tiers": { "easy": 35.48387096774193, "hard": 52.901023890784984, "judge": 76.86375321336762, "standard": 77.69516728624535 } }, "sealed|noul": { "cc": 23.62820512820513, "n": { "easy": 13, "hard": 32, "judge": 15, "standard": 15 }, "tiers": { "easy": 84.61538461538463, "hard": 6.25, "judge": -6.666666666666665, "standard": 73.33333333333334 } }, "sealed|score": { "cc": 51.95752059851082, "n": { "easy": 9, "hard": 33, "judge": 20, "standard": 13 }, "tiers": { "easy": 76.44302715293294, "hard": 49.562049785657535, "judge": 44.71869741001873, "standard": 55.363943729744456 } } } }, "jevbench_score": 71.4549280926844, "key": "microsoft-decision-1-a2-20261012", "last_measured_on": "2026-10-12", "listing": "preliminary", "measured_in": "v1.6.14", "measurement_date_status": "measured", "not_ranked_because": "A2 supplement measurement with an incomplete equating pool (four of six); excluded from every ranking.", "planned_weight_denominator": 1.0, "preliminary": { "equating_pool_size": 4, "equating_pool_target": 6, "label": "PRELIMINARY โ€” API supplement A2 (300 sealed + 300 public), not the S1200 full set; equating pool 4 of 6", "measured_on": "2026-10-12", "n_items": 600, "n_ok": 593, "scope": "API supplement A2" }, "provider": "microsoft", "rank": null, "ranked": false, "ranks": { "A": null, "B": null, "C": null, "capability": null }, "repo": "https://commandline.microsoft.com/microsoft-decision-1-model-foundry/", "scores": { "A": 71.4549280926844, "B": 68.62118813181577, "C": 69.62351351359915 }, "speed": { "adjustment": "none (API)", "n": 124, "p50_s_adjusted": 0.3374342759998399, "p50_s_raw": 0.3374342759998399, "p95_s_adjusted": 0.3999501053016501, "p95_s_raw": 0.3999501053016501 }, "status": { "answered_ok": 593, "missing": 0, "rows": 600, "status": "complete" }, "support": { "choice": "native", "noul": "native", "score": "native" }, "v16": { "api_subset": true, "api_subset_tag": "A2", "breakdowns": { "family": { "calibration": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0, "noul": 100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 2 }, "score": 100.0 }, "calibration_gen": { "accuracy": 0.7692307692307693, "cc": 63.29111148533698, "cc_by_type": { "choice": 60.8695652173913, "noul": 75.0, "score": 54.003769238619604 }, "low_n": true, "n": 26, "n_by_type": { "choice": 10, "noul": 8, "score": 8 }, "score": 63.29111148533698 }, "composite_gen": { "accuracy": 0.7741935483870968, "cc": 55.04557875346427, "cc_by_type": { "choice": 100.0, "noul": 19.999999999999996, "score": 45.136736260392794 }, "low_n": false, "n": 31, "n_by_type": { "choice": 10, "noul": 10, "score": 11 }, "score": 55.04557875346427 }, "fairness": { "accuracy": 1.0, "cc": 99.45663234640188, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 98.36989703920564 }, "low_n": true, "n": 18, "n_by_type": { "choice": 6, "noul": 6, "score": 6 }, "score": 99.45663234640188 }, "jaggedness": { "accuracy": 0.6, "cc": 36.18096361245757, "cc_by_type": { "choice": 72.78911564625851, "noul": -28.57142857142857, "score": 64.32520376254277 }, "low_n": false, "n": 45, "n_by_type": { "choice": 20, "noul": 14, "score": 11 }, "score": 36.18096361245757 }, "llm_router": { "accuracy": 0.8155339805825242, "cc": 77.29773373392734, "cc_by_type": { "choice": 66.38655462184875, "noul": 100.0, "score": 65.50664657993329 }, "low_n": false, "n": 103, "n_by_type": { "choice": 50, "noul": 30, "score": 23 }, "score": 77.29773373392734 }, "long_state": { "accuracy": 0.0, "cc": -110.3825136612022, "cc_by_type": { "choice": -31.14754098360656, "noul": -100.0, "score": -200.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 2, "score": 1 }, "score": 0.0 }, "multilingual": { "accuracy": 0.7405660377358491, "cc": 56.84729346177679, "cc_by_type": { "choice": 67.37084291989123, "noul": 39.99999999999999, "score": 63.17103746543915 }, "low_n": false, "n": 212, "n_by_type": { "choice": 107, "noul": 40, "score": 65 }, "score": 56.84729346177679 }, "negation_pair_gen": { "accuracy": 0.8, "cc": 60.0, "cc_by_type": { "choice": 100.0, "noul": 19.999999999999996 }, "low_n": true, "n": 20, "n_by_type": { "choice": 10, "noul": 10 }, "score": 60.0 }, "option_order": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0 }, "low_n": true, "n": 21, "n_by_type": { "choice": 21 }, "score": 100.0 }, "perturb_pair": { "accuracy": 1.0, "cc": 97.86434741302423, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 93.59304223907267 }, "low_n": true, "n": 26, "n_by_type": { "choice": 14, "noul": 8, "score": 4 }, "score": 97.86434741302423 }, "usecase": { "accuracy": 0.7058823529411765, "cc": 37.19296114334664, "cc_by_type": { "choice": 67.7891654465593, "noul": 10.000000000000009, "score": 33.78971798348063 }, "low_n": false, "n": 85, "n_by_type": { "choice": 44, "noul": 20, "score": 21 }, "score": 37.19296114334664 } }, "lang": { "ar": { "accuracy": 0.8888888888888888, "cc": 92.43674893442142, "cc_by_type": { "choice": 77.77777777777779, "noul": 100.0, "score": 99.53246902548645 }, "low_n": true, "n": 9, "n_by_type": { "choice": 6, "noul": 2, "score": 1 }, "score": 92.43674893442142 }, "cs": { "accuracy": 0.625, "cc": 66.1654135338346, "cc_by_type": { "choice": 32.330827067669176, "noul": 100.0 }, "low_n": true, "n": 8, "n_by_type": { "choice": 6, "noul": 2 }, "score": 66.1654135338346 }, "da": { "accuracy": 0.6666666666666666, "cc": 29.010593471104, "cc_by_type": { "choice": 80.83067092651757, "noul": 0.0, "score": 6.2011094867944205 }, "low_n": true, "n": 12, "n_by_type": { "choice": 7, "noul": 4, "score": 1 }, "score": 29.010593471104 }, "de": { "accuracy": 0.9230769230769231, "cc": 90.62759668123668, "cc_by_type": { "choice": 74.64788732394366, "noul": 100.0, "score": 97.23490271976637 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 4, "score": 4 }, "score": 90.62759668123668 }, "el": { "accuracy": 0.3333333333333333, "cc": -21.42857142857143, "cc_by_type": { "choice": 57.14285714285714, "noul": -100.0 }, "low_n": true, "n": 6, "n_by_type": { "choice": 3, "noul": 3 }, "score": 0.0 }, "en": { "accuracy": 0.7753424657534247, "cc": 58.59078985584421, "cc_by_type": { "choice": 76.21638426861493, "noul": 42.574257425742566, "score": 56.981727873175124 }, "low_n": false, "n": 365, "n_by_type": { "choice": 185, "noul": 101, "score": 79 }, "score": 58.59078985584421 }, "es": { "accuracy": 0.9166666666666666, "cc": 70.0797903785218, "cc_by_type": { "choice": 100.00000000000001, "noul": 100.0, "score": 10.239371135565412 }, "low_n": true, "n": 12, "n_by_type": { "choice": 7, "noul": 4, "score": 1 }, "score": 70.0797903785218 }, "fi": { "accuracy": 1.0, "cc": 94.53482291929028, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 83.60446875787086 }, "low_n": true, "n": 10, "n_by_type": { "choice": 2, "noul": 2, "score": 6 }, "score": 94.53482291929028 }, "fr": { "accuracy": 0.75, "cc": 83.59037174371464, "cc_by_type": { "choice": 51.99999999999999, "noul": 100.0, "score": 98.77111523114391 }, "low_n": true, "n": 8, "n_by_type": { "choice": 6, "noul": 1, "score": 1 }, "score": 83.59037174371464 }, "hi": { "accuracy": 1.0, "cc": 92.81199164662956, "cc_by_type": { "choice": 100.00000000000001, "noul": 100.0, "score": 78.43597493988868 }, "low_n": true, "n": 6, "n_by_type": { "choice": 4, "noul": 1, "score": 1 }, "score": 92.81199164662956 }, "id": { "accuracy": 0.7272727272727273, "cc": 70.7560754894387, "cc_by_type": { "choice": 73.09417040358746, "score": 68.41798057528993 }, "low_n": true, "n": 11, "n_by_type": { "choice": 5, "score": 6 }, "score": 70.7560754894387 }, "it": { "accuracy": 0.7272727272727273, "cc": 80.04349357771646, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 40.13048073314939 }, "low_n": true, "n": 11, "n_by_type": { "choice": 2, "noul": 2, "score": 7 }, "score": 80.04349357771646 }, "ja": { "accuracy": 0.6363636363636364, "cc": 47.61381032028706, "cc_by_type": { "choice": 1.6393442622950836, "noul": 50.0, "score": 91.20208669856609 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 4, "score": 3 }, "score": 47.61381032028706 }, "ko": { "accuracy": 0.8181818181818182, "cc": 77.78079668122632, "cc_by_type": { "choice": 100.0, "score": 55.561593362452655 }, "low_n": true, "n": 11, "n_by_type": { "choice": 3, "score": 8 }, "score": 77.78079668122632 }, "mixed": { "accuracy": 1.0, "cc": 100.0, "cc_by_type": { "choice": 100.0, "noul": 100.0 }, "low_n": true, "n": 13, "n_by_type": { "choice": 11, "noul": 2 }, "score": 100.0 }, "nl": { "accuracy": 0.5714285714285714, "cc": 1.7832893135850607, "cc_by_type": { "choice": 100.0, "noul": -100.0, "score": 5.349867940755182 }, "low_n": true, "n": 7, "n_by_type": { "choice": 4, "noul": 2, "score": 1 }, "score": 1.7832893135850607 }, "no": { "accuracy": 0.6363636363636364, "cc": 39.253029666918, "cc_by_type": { "choice": 14.893617021276594, "noul": 33.33333333333333, "score": 69.53213864614409 }, "low_n": true, "n": 11, "n_by_type": { "choice": 3, "noul": 3, "score": 5 }, "score": 39.253029666918 }, "pl": { "accuracy": 0.7, "cc": 52.957577540994386, "cc_by_type": { "choice": 67.97153024911032, "noul": 50.0, "score": 40.901202373872835 }, "low_n": true, "n": 20, "n_by_type": { "choice": 8, "noul": 4, "score": 8 }, "score": 52.957577540994386 }, "pt": { "accuracy": 0.6923076923076923, "cc": 74.25511482979587, "cc_by_type": { "choice": 26.530612244897956, "noul": 100.0, "score": 96.23473224448966 }, "low_n": true, "n": 13, "n_by_type": { "choice": 7, "noul": 4, "score": 2 }, "score": 74.25511482979587 }, "sv": { "accuracy": 0.6363636363636364, "cc": 3.3459357774096503, "cc_by_type": { "choice": 33.70165745856353, "noul": -100.0, "score": 76.33614987366542 }, "low_n": true, "n": 11, "n_by_type": { "choice": 4, "noul": 1, "score": 6 }, "score": 3.3459357774096503 }, "tr": { "accuracy": 0.7142857142857143, "cc": 77.61194029850746, "cc_by_type": { "choice": 55.22388059701492, "noul": 100.0 }, "low_n": true, "n": 7, "n_by_type": { "choice": 6, "noul": 1 }, "score": 77.61194029850746 }, "uk": { "accuracy": 0.5384615384615384, "cc": -2.908915160021811, "cc_by_type": { "choice": 55.55555555555555, "noul": -100.0, "score": 35.71769896437901 }, "low_n": true, "n": 13, "n_by_type": { "choice": 3, "noul": 1, "score": 9 }, "score": 0.0 }, "zh": { "accuracy": 0.8333333333333334, "cc": 89.63050869746054, "cc_by_type": { "choice": 70.58823529411765, "noul": 100.0, "score": 98.30329079826399 }, "low_n": true, "n": 12, "n_by_type": { "choice": 9, "noul": 2, "score": 1 }, "score": 89.63050869746054 } }, "type": { "choice": { "accuracy": 0.7933333333333333, "cc": 73.01414581066375, "cc_by_type": { "choice": 73.01414581066375 }, "low_n": false, "n": 300, "n_by_type": { "choice": 300 }, "score": 73.01414581066375 }, "noul": { "accuracy": 0.7266666666666667, "cc": 45.333333333333336, "cc_by_type": { "noul": 45.333333333333336 }, "low_n": false, "n": 150, "n_by_type": { "noul": 150 }, "score": 45.333333333333336 }, "score": { "accuracy": 0.7466666666666667, "cc": 59.17272659737858, "cc_by_type": { "score": 59.17272659737858 }, "low_n": false, "n": 150, "n_by_type": { "score": 150 }, "score": 59.17272659737858 } }, "usecase": { "advertising": { "accuracy": 0.6153846153846154, "cc": 22.655509813036304, "cc_by_type": { "choice": 73.7991266375546, "noul": -33.333333333333336, "score": 27.50073613488765 }, "low_n": true, "n": 13, "n_by_type": { "choice": 5, "noul": 3, "score": 5 }, "score": 22.655509813036304 }, "code_linting": { "accuracy": 0.5555555555555556, "cc": 30.103576043627953, "cc_by_type": { "choice": 56.83453237410072, "noul": 0.0, "score": 33.47619575678314 }, "low_n": true, "n": 18, "n_by_type": { "choice": 9, "noul": 4, "score": 5 }, "score": 30.103576043627953 }, "customer_support": { "accuracy": 0.875, "cc": 62.22727451527351, "cc_by_type": { "choice": 100.0, "noul": 0.0, "score": 86.68182354582052 }, "low_n": true, "n": 8, "n_by_type": { "choice": 5, "noul": 2, "score": 1 }, "score": 62.22727451527351 }, "demand_forecasting": { "accuracy": 0.2857142857142857, "cc": -25.73258663586017, "cc_by_type": { "choice": 16.89750692520776, "noul": -33.333333333333336, "score": -60.76193349945493 }, "low_n": true, "n": 14, "n_by_type": { "choice": 8, "noul": 3, "score": 3 }, "score": 0.0 }, "ecommerce_marketplaces": { "accuracy": 0.6875, "cc": 37.128494536040996, "cc_by_type": { "choice": 51.61290322580645, "noul": -33.333333333333336, "score": 93.10591371564988 }, "low_n": true, "n": 16, "n_by_type": { "choice": 8, "noul": 3, "score": 5 }, "score": 37.128494536040996 }, "feature_extraction": { "accuracy": 0.875, "cc": 79.10347860874892, "cc_by_type": { "choice": 85.94847775175644, "noul": 100.0, "score": 51.361958074490275 }, "low_n": true, "n": 16, "n_by_type": { "choice": 9, "noul": 1, "score": 6 }, "score": 79.10347860874892 }, "financial_crime": { "accuracy": 0.8666666666666667, "cc": 86.22232602398712, "cc_by_type": { "choice": 80.70739549839227, "noul": 100.0, "score": 77.95958257356911 }, "low_n": true, "n": 15, "n_by_type": { "choice": 7, "noul": 2, "score": 6 }, "score": 86.22232602398712 }, "gaming": { "accuracy": 0.5333333333333333, "cc": 12.129649732040141, "cc_by_type": { "choice": 45.454545454545446, "noul": -33.333333333333336, "score": 24.267737074908315 }, "low_n": true, "n": 15, "n_by_type": { "choice": 5, "noul": 3, "score": 7 }, "score": 12.129649732040141 }, "insurance_claims": { "accuracy": 0.5454545454545454, "cc": 11.15157429187248, "cc_by_type": { "choice": 43.573667711598745, "noul": 0.0, "score": -10.118944835981303 }, "low_n": true, "n": 11, "n_by_type": { "choice": 7, "noul": 2, "score": 2 }, "score": 11.15157429187248 }, "knowledge_graphs": { "accuracy": 0.7272727272727273, "cc": 65.92165664860643, "cc_by_type": { "choice": 56.83453237410071, "noul": 50.0, "score": 90.93043757171858 }, "low_n": true, "n": 11, "n_by_type": { "choice": 6, "noul": 4, "score": 1 }, "score": 65.92165664860643 }, "lead_generation": { "accuracy": 0.5833333333333334, "cc": 32.27917770928797, "cc_by_type": { "choice": 52.56916996047431, "noul": 0.0, "score": 44.2683631673896 }, "low_n": true, "n": 12, "n_by_type": { "choice": 6, "noul": 4, "score": 2 }, "score": 32.27917770928797 }, "legal_compliance": { "accuracy": 1.0, "cc": 94.12770453740406, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 82.38311361221218 }, "low_n": true, "n": 10, "n_by_type": { "choice": 5, "noul": 1, "score": 4 }, "score": 94.12770453740406 }, "llm_guardrails": { "accuracy": 0.8888888888888888, "cc": 87.81390014088669, "cc_by_type": { "choice": 78.14207650273225, "noul": 100.0, "score": 85.29962391992782 }, "low_n": true, "n": 18, "n_by_type": { "choice": 12, "noul": 1, "score": 5 }, "score": 87.81390014088669 }, "model_routing": { "accuracy": 0.8588235294117647, "cc": 81.49062517209573, "cc_by_type": { "choice": 79.13561847988078, "noul": 100.0, "score": 65.33625703640644 }, "low_n": false, "n": 85, "n_by_type": { "choice": 40, "noul": 24, "score": 21 }, "score": 81.49062517209573 }, "moderation_trust_safety": { "accuracy": 0.5833333333333334, "cc": 1.4715857588347916, "cc_by_type": { "choice": 61.29032258064516, "noul": -100.0, "score": 43.124434695859215 }, "low_n": true, "n": 12, "n_by_type": { "choice": 7, "noul": 1, "score": 4 }, "score": 1.4715857588347916 }, "other": { "accuracy": 0.4, "cc": 15.494864573175938, "cc_by_type": { "choice": 9.774436090225558, "score": 21.215293056126317 }, "low_n": true, "n": 5, "n_by_type": { "choice": 3, "score": 2 }, "score": 15.494864573175938 }, "recruiting": { "accuracy": 0.6666666666666666, "cc": 30.434646255259782, "cc_by_type": { "choice": 49.999999999999986, "noul": 0.0, "score": 41.30393876577936 }, "low_n": true, "n": 9, "n_by_type": { "choice": 3, "noul": 2, "score": 4 }, "score": 30.434646255259782 }, "risk_assessment": { "accuracy": 0.75, "cc": 10.1937605340201, "cc_by_type": { "choice": 52.94117647058823, "noul": -100.0, "score": 77.64010513147205 }, "low_n": true, "n": 12, "n_by_type": { "choice": 6, "noul": 1, "score": 5 }, "score": 10.1937605340201 }, "scientific_discovery": { "accuracy": 0.5714285714285714, "cc": 27.722363031181995, "cc_by_type": { "choice": 63.07692307692309, "noul": 0.0, "score": 20.09016601662289 }, "low_n": true, "n": 14, "n_by_type": { "choice": 7, "noul": 2, "score": 5 }, "score": 27.722363031181995 }, "search_retrieval": { "accuracy": 1.0, "cc": 98.185292235786, "cc_by_type": { "choice": 100.0, "noul": 100.0, "score": 94.55587670735798 }, "low_n": true, "n": 16, "n_by_type": { "choice": 7, "noul": 5, "score": 4 }, "score": 98.185292235786 } } }, "breakdowns_equated": false, "ci95": { "B": 1000, "C": [ 77.61094153455542, 91.6237504581693 ], "I": [ 47.90571478644313, 68.3083574651847 ], "capability": [ 64.56492363928368, 77.67236999065625 ], "composite": { "A": [ 60.33591615493133, 74.53742586189195 ], "B": [ 56.31152174249009, 73.1804420130518 ], "C": [ 41.89994177425786, 74.53742586189195 ] }, "seed": 22 }, "ci95_basis": "bootstrap over the API system's items (n = A sealed items) AND over the equating offset (pool systems resampled with replacement, items resampled; B_offset = 200)", "ci95_n": 300, "complete": true, "equated": { "calibration": 85.15566582345811, "capability": 72.19088219725606, "intelligence": 59.226098571054024 }, "equating_flag": null, "equating_offset": { "C": 1.1358289246878535, "I": 2.574115857338972 }, "full_set_api": false, "g_med_used": 16.820628977585045, "lane": "api", "metric_basis": "A2 u P (sealed side = supplementary API subset A2, n=300)", "n_items": 600, "n_public": 300, "n_sealed_side": 300, "offset_ci95": { "C": [ -2.4463102425519985, 9.397802125135494 ], "I": [ -4.427206040125616, 10.076945418608098 ] }, "on_S": null, "per_type": { "choice": { "accuracy": 0.7933333333333333, "calibration": 77.8355360065047, "cc": 73.01414581066375, "cc_by_type": { "choice": 73.01414581066375 }, "cc_tiered": 73.0476445854911, "cc_tiered_open": 82.78833309663467, "cc_tiered_sealed": 63.30695607434754, "low_n": false, "n": 300, "n_by_type": { "choice": 300 }, "score": 73.01414581066375 }, "noul": { "accuracy": 0.7266666666666667, "calibration": 87.09142854224856, "cc": 45.333333333333336, "cc_by_type": { "noul": 45.333333333333336 }, "cc_tiered": 39.285117056856194, "cc_tiered_open": 54.94202898550725, "cc_tiered_sealed": 23.62820512820513, "low_n": false, "n": 150, "n_by_type": { "noul": 150 }, "score": 45.333333333333336 }, "score": { "accuracy": 0.7466666666666667, "calibration": 87.1325461475575, "cc": 59.17272659737858, "cc_by_type": { "score": 59.17272659737858 }, "cc_tiered": 57.62318649879788, "cc_tiered_open": 63.288852399084945, "cc_tiered_sealed": 51.95752059851082, "low_n": false, "n": 150, "n_by_type": { "score": 150 }, "score": 59.17272659737858 } }, "raw_on_A": { "calibration": 84.01983689877025, "capability": 70.33590980624265, "intelligence": 56.65198271371505 }, "set": "A2+P" }, "validity": { "choice": { "invalid_rate": 0.013333333333333308, "n": 300 }, "noul": { "invalid_rate": 0.013333333333333308, "n": 150 }, "score": { "invalid_rate": 0.00666666666666671, "n": 150 } }, "views": { "Balanced 33:33:33": 67.81784570375109, "Emphasis on Accuracy 60:20:20": 64.0984217053166, "Emphasis on Cost 20:20:60": 65.4526287582965, "Emphasis on Speed 20:60:20": 74.86761060072513, "Intelligence only": 59.22609857105403 } }, { "addendum": null, "api_flag": false, "api_subset": false, "author": "Phocinae / NovelNexus AI (novelnexusai@outlook.com)", "axes": { "calibration": 56.87058116402116, "cost": 95.25096508672264, "intelligence": 3.609767391262904, "speed": 77.11764018052709 }, "calibration": { "parts": { "choice": { "ece_hard": 0.06613968253968247, "ece_score": 86.7720634920635, "mean_tvd": null, "n": 750, "n_ece_hard": 315, "n_tvd": 0, "score": 86.7720634920635 }, "noul": { "abstention_rate": 0.9706666666666667, "brier": 0.24743832458666667, "ece": 0.08080160000000003, "n": 375, "score": 83.83967999999999 } }, "score": 56.87058116402116 }, "capability": 30.240174277642033, "composite_ci95": { "A": [ 0.00862203578483849, 0.21830578721673696 ], "B": [ 0.00558321631423672, 0.15046670100725723 ], "C": [ 0.0059875248505822844, 0.15160124112273404 ] }, "cost": { "basis": "ESTIMATE (*est.), size-class reference; self-hosted open weights, no public bookable tariff. A 144.3M ModernBERT encoder with no decoder lineage, so the base-size-encoder line is used unchanged - USD 0.005 per 1M input and 0.0 per 1M output, the line the PUBLISHED v1.6 field already carries on modernbert-jev-v2, tacet-sonata and watt-flash-0.1 (verified in their own registry entries; modernbert-jev-v2 was measured in v1.6.8 on the earlier v1.6-regular-20261010-a2 draw, while tacet-sonata and watt-flash-0.1 are v1.6.1 rows - none of the three was measured on THIS draw, which carries no ModernBERT row), backed by the dated 25 Sep 2026 DeepInfra snapshot sha256 84eb7926acd4e93b71d695be24898e2ca999b20011d1b66cc0348ce7c6c3546f in which thenlper/gte-base, BAAI/bge-base-en-v1.5, intfloat/e5-base-v2 and sentence-transformers/all-mpnet-base-v2 all price at USD 0.005 per 1M input and 0 output - and no new release-lane pricing line is created. Output tokens are a literal 0 reported by the author's own server (an encoder read-out; nothing is generated). x this row's own measured input tokens, which are the author's server's own count. Estimated, not charged.", "common_basis": null, "kind": "estimate", "usage_estimated_rows": 0, "usd_per_1000": 0.0014397966666666667 }, "display": "Phocinae-Largha-150M-v1", "endpoint_condition": "Sandy, the evaluator's own shared CPU host (AMD Ryzen 5 3600, 6 cores / 12 threads, Zen 2, AVX2), NOT a dedicated machine: other jobs ran beside it and the 1-minute load average was logged every 60 s into logs/CONDITIONS-.jsonl - for this row's window 3.75 to 9.89, mean 5.81 over 12 samples, the quietest of the three new rows. The author's own phocinae-server 0.1.5 from PyPI, UNMODIFIED, run with their own documented line `PHOC_MODEL_DIR= python -m phocinae.main` at PHOC_DEVICE=cpu, PHOC_THREADS=1 (their own documented CPU setting: 1.64 s per 5-decision case single-threaded), PHOC_EXTENSIONS=1 for their native option_scores, PHOC_PERM_AVG at its default 0 and no bearer token, inside a loopback-only network namespace with an outbound probe that failed as required and no credentials in the environment. Their package omits numpy from its declared dependencies although their own engine.py calls .numpy(); numpy 2.5.4 was installed so their code could run, and no file of theirs was edited. OUR shim (r107/src/phoc_shim_r107.py, byte-identical to the already source-reviewed r92/src/phoc_shim_r92.py, sha256 631cb60dde86b8e609382a32900db5e4ddc9fa9b2fef39bdb7b022dff558775e) is the only translation layer in front of it, because their published API carries no question text - see the mapping disclosure. Speed is a loaded-shared-CPU figure; the standard self-hosted x2 + 0.15 s adjustment is applied on top and the raw p50/p95 are kept. Re-measured for this release on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720077ef7c41ace367524896507c7f44ce71306fb3e659e8b85069d995b) under CoS deputy decision #18209, at USD 0 on Sandyโ€™s own CPU, 1 thread, 2026-10-13 05:59:36Zโ€“06:11:07Z, 1-minute load average 3.75โ€“9.89 (mean 5.81, 12 samples). DISCLOSED MAPPING: choice options are sent as \"