Respan Span-02

Data: JevBench v1.6.40, published 2026-10-22

Capability 69.3 · Not ranked in the current release

cost 2.1× Jev (v1.5 reference)

Last measured: 2026-10-14 · measurement release: v1.6.18

Apache-2.0 per model card; weights are a Gemma 4 derivative under the Gemma 4 terms; NOTICE names its vLLM / SGLang / FlashInfer ports (Apache-2.0)

Base model: undisclosed

intelligence

60.9

calibration

77.8

speed

85.3

cost

45.1

Composite (secondary)

51.5 · Composite rank #39

Cost and measurement conditions

$0.068 per 1,000 decisions (estimated)

ESTIMATE (base-model reference; self-hosted open weights, no public tariff for these weights): the frozen market reference google/gemma-4-26b-a4b-it USD 0.09/M input, USD 0.3/M output for the declared base Gemma 4 26B-A4B (catalog id google/gemma-4-26b-a4b-it, supplied by the evaluator because the frozen path matches base names case-sensitively; the submission's model card spells it google/gemma-4-26B-A4B-it) x this row's own measured tokens on the common 1,485-item basis (776,560 input and 102,328 output tokens as reported by the author's server; the 15 refused items count as zero). Estimated, not charged; not a quote from the author or any host. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.067737 per 1,000 decisions on the common basis (USD 0.067059 over all 1,500 items) is 1.049x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 precedent) and falls outside the official-caps board views by construction. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them). They are the 15 items the Jev reference refused with max_tokens_exceeded, which the other rows that did not answer them refused or failed on; their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence. This row: USD 0.067737 per 1,000 decisions (1.049x the official cost cap USD 0.064595), against USD 0.067059 over all 1,500 items (v1.6.29).

Measured p50 latency: 0.027 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)

Ranking and cap use: 0.20 s (est. incl. network (2x + 150 ms))

Self-hosted model: measured p50 0.027 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.20 s. Model authors can ask us to time a hosted endpoint instead (on request).

x2 + 0.15 s (assumption, not measured)

evaluator-owned rented Lium GPU pod, 1x NVIDIA RTX 5090 32 GB (Romania), NOT the 1x H100 80 GB on which most of this draw's GPU rows were timed: the author's engine runs only on Blackwell (sm_120a), so this row's Speed axis rests on different hardware. The author's own Rust/CUDA dllm-server, built unchanged from their Dockerfile at the pinned revision (weights model.safetensors sha256 5105d39d..., tokenizer cc8d3a0c..., public MTP draft google/gemma-4-26B-A4B-it-assistant pre-staged), started by their own launch.sh with their documented Decision Index settings (container environment HOST=127.0.0.1, PORT=8000; weights mounted read-only at /model, the pre-staged draft at /data) in a container with --network none. Isolation timeline: the container's cut was proven by docker inspect and a failing outbound probe at 05:45:26Z (pod clock) before the sealed input file was copied to the pod host at 05:45:48Z (evaluator clock); the measuring container was then re-created with the same --network none setting and re-proven at 05:45:50Z before the first sealed item was sent at 05:45:55Z. The pod host itself kept its network for the evaluator's ssh session, as for every self-hosted row; no request left the container; one disclosed invocation change: launch.sh is stored without the executable bit in the repository (mode 100644), so the unchanged file was run through the interpreter its own shebang names (/bin/bash) instead of being exec'd directly. Driven by the unchanged run_v15 runner and TypeSafe adapter against /v1/systemone, sequentially, after a 4-item synthetic smoke gate. The standard self-hosted x2 + 0.15 s own-server adjustment (an assumption, not measured) applies on top of the raw p50/p95. 15 of 1,500 requests returned HTTP 400 because prompt + 256-token cap exceeds the server's 16,384-token context: exactly the 15 task ids pre-registered from an offline count with the author's own tokenizer (the same 15 that Attest-31B, Standard One 3B SH, Uniphore-4B-Decider and PurpleMIST-Mini failed on); they are scored as wrong in the full denominator. Measured 14 Oct 2026 on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720077ef7c41ace367524896507c7f44ce71306fb3e659e8b85069d995b); pod removed afterwards. Regular-queue request GitHub issue #215; CoS deputy #19087/#19088 (spend), custodian #19055 (draw use).

Published source