Nimble-v4-max (open weights, self-hosted)v1.7 · 10 s rule
Data: JevBench v1.7.0, published 2026-10-24 · method v1.7 (10 s rule), measured under the rule
Capability 81.2 · Not ranked in the current release
cost 6.6× Jev (v1.5 reference)
Last measured: 2026-10-21 · measurement release: v1.7.0
CC BY-NC 4.0 (weights, non-commercial); base Qwen/Qwen3.8-27B Apache-2.0
intelligence
71.0
calibration
91.4
speed
84.8
cost
30.1
Composite (secondary)
20.7 · Composite rank #105
Cost and measurement conditions
$0.214 per 1,000 decisions (estimated)
ESTIMATE (base-model reference; self-hosted open weights, no public tariff for these weights): the frozen Qwen/Qwen3.8-27B market reference of the published v1.6 field through the frozen pricing_v15 module (reference() -> qwen/qwen3.8-27b, USD 0.42 per 1M input and USD 3.00 per 1M output; the same pair the published vllm-sr-d3, pplx-decider-v1-1-27b and torchcast-decision-27b rows carry) x this row's own measured tokens on the common 1,485-item basis (755,508 input tokens, counted by the author's own engine and equal per item to the offline count made before the timed run; the author's engine reports 0 output tokens because nothing is generated). USD 0.213679 per 1,000 decisions on the common basis (USD 0.2115422 over all 1,500 items), 3.308x the official cost cap USD 0.064595, so this row is outside the default cost-capped Capability ranking and the squared cost gate keeps its composite small. Estimated, not charged; not a quote from the author or any host (the rented GPU cost the evaluator USD 1.7726 for the whole run, which is not the row's price). COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them). They are the 15 items the Jev reference refused with max_tokens_exceeded, which this row refuses too (HTTP 422, over its 32,768-token serving limit); their tokens count for no row, and a refused item still counts as a wrong answer for Intelligence. This row: USD 0.213679 per 1,000 decisions (3.308x the official cost cap USD 0.064595), against USD 0.211542 over all 1,500 items (v1.6.29).
Measured p50 latency: 0.154 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.46 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.154 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.46 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Evaluator-owned rented Verda GPU VM (instance type 1A100.22V, location FIN-03, Finland): 1x NVIDIA A100-SXM4-80GB 80 GB, driver 580.178.04, torch 2.8.0+cu128. A different card from the 1x H100 80 GB on which most of this draw's other GPU rows were timed, so this row's Speed axis rests on different hardware (A100 is slower than H100). The author's own production engine nimble.evaluation.decision_index_engine:NimbleEngine (bespokelabs/nimble-v4-max-decision-index @ adea14289afa29dca3ff38b1fcb76e58f6add230, code/nimble/, unmodified) run in process behind a first-party HTTP shim (evaluator code; it only relabels a score answer's type, adds output_tokens 0 and turns the engine's Unsupported into HTTP 422), driven by the frozen run_v15 harness and the unchanged TypeSafe adapter. Engine options are the author's own environment.json settings, not tuned: shared_prefix on, max_tails 48, max_tokens 32,768. Weights bespokelabs/Nimble-v4-max @ 655aa7051c2ab39660193ebb70c8926d3d2cb7aa (11 safetensors shards hash-verified on the pod), base and tokenizer Qwen/Qwen3.8-27B @ 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. Isolation: fresh container with --network none (the cut proven from inside before the input was read), read-only root filesystem, all capabilities dropped, no credentials, Hub offline mode; 1,500 items in one pass, no resume, one sealed use. 15 items were refused with HTTP 422 because their prompts (76,358-80,516 input tokens) exceed the 32,768-token serving limit; nothing was truncated, and the refused set equals the set predicted by the offline count before the run. They count as wrong in every axis (full 1,500 denominator). The standard self-hosted x2 + 0.15 s own-server adjustment (an assumption, not measured) is applied on top of the raw p50/p95. Measured 21 Oct 2026 on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720077ef7c41ace367524896507c7f44ce71306fb3e659e8b85069d995b); VM deleted afterwards (deletion verified).
Measured latency over 1,500 decisions: p50 0.154 s · p95 0.288 s
1 decision over 10 s (1 of them answered correctly, counted as wrong).
A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.
Model and serving disclosure
bespokelabs/nimble-v4-max-decision-index @ adea14289afa29dca3ff38b1fcb76e58f6add230 (code/nimble/, unmodified); evaluator HTTP shim. MANDATORY DISCLOSURE: measured on a draw the author's hosted endpoint received on 14 Oct; weights committed 18 Oct. Second and last use by this author on this draw. This row is not an independent corroboration of the hosted row nimble-v4-max: the two rows come from the same author, on different hardware through a different prompt path, and the hosted row stays a separate row. We cannot verify that these weights are the bytes served by the hosted endpoint on 14 Oct, or that the exposed items played no role in producing them. Licence CC BY-NC 4.0 (weights, non-commercial). The 15 long items are refused (HTTP 422 from the engine's serving limit) and count as wrong in every axis (full 1,500 denominator). Output tokens are 0 on every answered item (the engine has no generate loop); kept as sent.