vLLM-SR d3
Data: JevBench v1.6.40, published 2026-10-22
Capability 83.0 · Not ranked in the current release
cost 6.8× Jev (v1.5 reference)
Last measured: 2026-10-12 · measurement release: v1.6.16
Apache-2.0 (weights, model card and NOTICE; base Qwen/Qwen3.8-27B Apache-2.0). Training data not disclosed
intelligence
73.7
calibration
92.2
speed
87.4
cost
29.8
Composite (secondary)
20.5 · Composite rank #93
Cost and measurement conditions
$0.218 per 1,000 decisions (estimated)
ESTIMATE (base-model reference; self-hosted open weights, no public tariff for these weights): the frozen Qwen/Qwen3.8-27B market reference of the published v1.6 field, USD 0.42 per 1M input and USD 3.00 per 1M output - the same pair the published pplx-decider-v1-1-27b and torchcast-decision-27b rows carry - x this row's own measured tokens on the common 1,485-item basis (1,951,205 input tokens; the author's server reports 0 output tokens because nothing is generated). USD 0.218114 per 1,000 decisions on the common basis (USD 0.5463374 over all 1,500 items), about 3.4x the published v1.6 ceiling of USD 0.0646 per 1,000 decisions, so this row is outside the default cost-capped Capability ranking and the squared cost gate keeps its composite small. Estimated, not charged; not a quote from the author or any host. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them). They are the 15 items the Jev reference refused with max_tokens_exceeded, which the other rows that did not answer them refused or failed on; their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence. This row: USD 0.218114 per 1,000 decisions (3.377x the official cost cap USD 0.064595), against USD 0.546337 over all 1,500 items (v1.6.29).
Measured p50 latency: 0.109 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.37 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.109 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.37 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
evaluator-owned rented Verda GPU VM, 1x NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (confidential-computing mode, attested), NOT the 1x H100 80 GB on which this draw's other GPU rows and the published Perplexity Decider and Torchcast-Decision-27B rows were timed - no single H100 was available at measurement time - so this row's Speed axis rests on different hardware from those rows (some other published v1.6 rows, for example xor-26b-a4b and rene-1-31b-fp8, were also timed on an RTX PRO 6000). The author's own unmodified d3_server.py (plain CUDA / PyTorch path; the author's AMD ROCm fast path does not apply on CUDA) in an offline container with --network none, Hub offline mode and all credentials unset, weights verified against the author's MODEL_MANIFEST sha256 list; driven by the unchanged pinned pod_driver_v16 with one official warm-up and a sequential scoring loop. The standard self-hosted x2 + 0.15 s own-server adjustment (an assumption, not measured) is applied on top of the raw p50/p95. Measured 12 Oct 2026 on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720077ef7c41ace367524896507c7f44ce71306fb3e659e8b85069d995b); VM destroyed afterwards.