APUS-OpenJev-v1-9B (merged bf16 checkpoint-3000 on Qwen3.5-9B, their own native runtime, full depth, self-hosted)v1.7 · 10 s rule
Data: JevBench v1.8.18, published 2026-11-20 · method v1.7 (10 s rule), re-scored from the recorded per-decision times
Capability 56.9 · rank #161 of 251 on the open-weights board
Last measured: 2026-10-07 · measurement release: v1.6.1
apache-2.0
intelligence
47.2
calibration
66.6
speed
82.1
cost
68.4
Composite (secondary)
56.6 · Composite rank #113
Cost and measurement conditions
$0.011 per 1,000 decisions (estimated)
GPU-time estimate from v1.8.0: one monthly-rented RTX PRO 6000 96 GB, EUR 999/month / 730 hours, ECB USD 1.1133/EUR dated 28 Oct 2026, USD 1.5235/hour; average load 70%. Architecture: small qwen, 9B active parameters; throughput 27344.4444444 tokens/s (architecture-class calibration, 29 Oct 2026, efficient-engine prefill; capped at 100,000 tokens/s). The row's own scored cost basis: 755,283 input + 0 output tokens over 1,477 decisions = 511.362897766 tokens/decision. GPU-time cost USD 0.0113058189868536/1,000; old list-price cost USD 0.0511362897765741/1,000; applied min(old list-price cost, GPU-time cost) = USD 0.0113058189868536/1,000. The GPU-time estimate is lower. Estimated, not a bill. Intelligence, Calibration, measured latency and the common cost-item membership are unchanged. Prior prices remain in the archived v1.7 releases.
Measured p50 latency: 0.282 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.71 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.282 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.71 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...). Weights apus-ailab/APUS-OpenJev-v1-9B@82c9c56cfa9de8d36704ed91948d4726ef111635, Apache-2.0, 17.55 GiB bf16 merged weights, checkpoint-3000, declared base Qwen/Qwen3.5-9B, full depth 32 layers. STACK: the authors' own requirements.txt, unmodified - torch==2.8.0 and transformers==5.16.1. The transformers pin is THEIRS, not a workaround: OpenJet.__init__ raises 'Layer execution is audited for transformers==5.16.1' on any other version. Served behind a TRANSPORT SHIM ONLY (r56/serve/apus_systemone_r56.py): loaded with THEIR OpenJet.from_pretrained(device=cuda:0, dtype=bfloat16) and every request handed to THEIR decide(), returning THEIR format_response() output. No contributor file edited. effort='high', the full-depth path, their decide() default and the configuration their own headline Frozen80 number is quoted on; effort='low' is not measured and no routing between the two is invented. One H100 PCIe 80 GB (Lium). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. MAPPING (run 56's, not the authors'; pre-registered in PREREG-R56 s2.2 before any item was sent and repeated here because it is part of the number): their runtime accepts primitive in {choice, noul, score_level} with 2-16 candidates labelled A-P. choice -> their 'choice' primitive, candidates = the item's own labels as id and its own criteria[label] as description, in labels order; their format_response returns probabilities keyed by candidate id, i.e. by the JevBench label already, so nothing is rewritten. All 750 choice items carry 3-9 options, inside their 2-16 window. noul -> their 'noul' primitive; their contract REQUIRES the canonical yes/no criteria there, so the item's own true/false description strings are NOT delivered - the proposition itself travels verbatim in instructions, which is what their noul is built to judge. score -> their 'choice' primitive over the ordinal levels, because their own RUNTIME.md rules score_level out as an ordinal Score API ('score_level is the judgement of a single proposition, it cannot be treated as a complete ordinal Score API'); the 4-5 ordered levels become candidates with id = the level index and description = the item's own level text, which carries the order. Recording all 375 score items as misses instead would understate a system whose published interface answers them. CALIBRATION IS MEASURED AS THE AUTHORS SHIP IT AND THEY DISCLOSED IT IN ADVANCE: their response carries calibrated: False and both their card and RUNTIME.md state the probabilities are an uncalibrated relative softmax over the offered candidates. OUTPUT TOKENS ARE 0 BY CONSTRUCTION, NOT BY ESTIMATE: decide() runs one forward pass with logits_to_keep=1 and reads candidate logits; nothing is generated anywhere in that path. Input tokens are their own prompt_tokens field. THE 23 LONG ITEMS: their compile() raises ValueError('input exceeds runtime limit; no truncation permitted') above 8,192 tokens and their RUNTIME.md states it in advance. Recorded invalid. This is the first row in the series whose refusal on those items is guaranteed in the author's own code with an explicit no-truncation promise. add-requests run 56, GitHub #106, benchmarkheaven.com/submit ref 2033065A
Measured latency over 1,500 decisions: p50 0.282 s · p95 0.361 s
0 decisions over 10 s.
A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.
Breakdowns (not part of the score)
Chance-corrected competence (0 = chance, 100 = perfect) by category on S+P+L1+L2+L3. Raw values, not equated; the same cells feed the compare view.
Capability by subject topic
| Rules, policy & law | 33.9 | n=1616 |
|---|---|---|
| Coding & software | 56.1 | n=382 |
| Math & numbers | 0.0 | n=370 |
| Finance & commerce | 49.4 | n=223 |
| Support & operations | 32.2 | n=160 |
| Everyday language | 66.9 | n=145 |
| Safety & security | 57.2 | n=109 |
Use cases (TypeSafe categories)
| Model routing | 70.8 | n=524 |
|---|---|---|
| Legal & compliance | 49.0 | n=515 |
| Customer support | 18.2 | n=240 |
| E-commerce | 20.2 | n=119 |
| Insurance claims | 35.4 | n=119 |
| Risk assessment | 46.8 | n=118 |
| Financial crime | 7.3 | n=101 |
| Feature extraction | 9.0 | n=99 |
| Lead generation | 63.5 | n=95 |
| Recruiting | 15.5 | n=94 |
| Knowledge graphs | 31.4 | n=94 |
| LLM guardrails | 32.3 | n=93 |
| Moderation | 39.0 | n=93 |
| Code linting | 0.0 | n=90 |
| Search & retrieval | 74.2 | n=90 |
| Science | 0.5 | n=90 |
| Advertising | 15.0 | n=88 |
| Gaming | 12.0 | n=86 |
| Demand forecasting | 0.0 | n=83 |
Cells need at least 30 answered items; † fewer than 30 (indicative only; drawn as hollow, lighter points on the compare radars).
JevBench v1.6.43 values (method v1.6, before the 10 s rule)
Capability 56.91 · Composite 49.18 · rank #65 on the all-systems board · #41 open weights
intelligence 47.21 · calibration 66.61 · speed 82.18 · cost 48.74
v1.6 speed basis (124 items): p50 0.278 s · p95 0.354 s
These figures are kept for comparison. The current figures above are re-scored under the 10 s rule from the recorded per-decision times and replace them.