vLLM-SR d3-miniv1.7 · 10 s rule
Data: JevBench v1.7.0, published 2026-10-24 · method v1.7 (10 s rule), measured under the rule
Capability 68.8 · rank #33 of 157 on the open-weights board
Last measured: 2026-10-24 · measurement release: v1.7.0
Apache-2.0 at the measured pin (model card, LICENSE and NOTICE); base Qwen/Qwen3.5-4B Apache-2.0; public, not gated. Training data not disclosed
intelligence
50.8
calibration
86.8
speed
93.1
cost
64.2
Composite (secondary)
69.5 · Composite rank #15
Cost and measurement conditions
$0.016 per 1,000 decisions (estimated)
ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 aa2afcd5a58d0c4cc0e3d6867d590c70b6ac58e971410c27faf9628f97bcf8b7, re-frozen 22 Oct 2026): base string Qwen/Qwen3.5-4B priced as Qwen/Qwen3.5-4B USD 0.03/M input, USD 0.15/M output, as required by the frozen pricing policy. The module selects the deprecated-after-cutoff DeepInfra reference retained under the frozen policy (price snapshots of 25 Sep 2026), USD 0.03/M input and USD 0.15/M output. Estimated, not charged; not a quote from the author or any host. Tokens as reported by the author's server: input usage includes the rendered question and options; output_tokens = 0 because nothing is generated. Reported usage is 1,951,205 input tokens and 0 output tokens, equal to the evaluator's offline count, offset 0 and 0 item-level mismatches. The 15 standard long items contribute 1,180,015 input tokens. Excluding them as for every row on the common 1,485-item cost basis (v1.6.30) leaves 771,190 input tokens and 0 output tokens. All 1,500 items still count for Intelligence. Measured arithmetic: USD 0.015579596 per 1,000 decisions on the common basis = 0.241190172x the unchanged USD 0.06459465517241379 cap; separately USD 0.039024100/1,000 over all 1,500 items = 0.604138220x. GPU rental is separate from this estimated model tariff. COST ON THE COMMON BASIS: USD 0.015580 per 1,000 decisions on 1,485 items, 0.241 times the official cost cap USD 0.064595; estimated, not charged.
Measured p50 latency: 0.028 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.21 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.028 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.21 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Evaluator-owned rented Verda GPU VM, one NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB, FIN-03 (Finland, EU), PAYG non-spot, instance type 1RTXPRO6000.30V, USD 2.5991917808219176/h recorded by the runner (300 GB disk). Same GPU class as the published d3 and d3-turbo rows; hardware/mode conditions differ across the wider field. The author's own unmodified stdlib server on loopback in a --network none --read-only container; serial requests, read-only code/model/input, empty credential variables and HF offline flags. In-container network cut proven before sealed input; all 22 snapshot files re-hashed, offline flag values and absence of author-path override variables checked. The standard self-hosted x2 + 0.15 s own-server adjustment is an assumption, not measured. Measured 24 Oct 2026, pod window 09:07-09:19Z, approximately USD 0.50 rental. Measurement container 307 s, run cap 2856 s, complete=true, deadline_hit=false, rc 0; scored loop 224.870576 s over 1,500 requests. Frozen v1.6-regular-20261012 self-hosted input sha256 dfd36720077ef7c41ace367524896507c7f44ce71306fb3e659e8b85069d995b, checked in the container. Raw output sha256 814d4afdcc638f3773e3b6263f98e1319e952a2115e6ff1f3cca6c18e98f88af. Node-shred and row cleanup verified; exact-id VM and volume teardown recorded, provider absent-or-discontinued and guard closed. GPU rental is not the benchmark's estimated model tariff.
Measured latency over 1,500 decisions: p50 0.028 s · p95 0.045 s
15 decisions over 10 s (15 of them answered correctly, counted as wrong).
A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.
Model and serving disclosure
THE AUTHORS' OWN ENGINE, run unmodified: upstream vllm-sr/d3-mini at bd0090dd758d32ba59ef5c749910a8e87fc4391f, the author's stdlib d3_server.py with d3_runtime.py, d3_format.py, d3_engine.py, d3_fast.py and d3_kernels.py. The measured repository is vllm-sr/d3-mini; zils-ai/z3-mini-v1 and zils-ai/d3-mini-4.5b-v1 are byte-identical weight mirrors, not separate measurements. The server, engine and kernels are byte-identical to the reviewed d3-turbo pin 689b3af198baaf6f11110be38a9daa7188dfd8ad; d3_runtime.py and d3_format.py are the older non-causal-only variants, reviewed separately. The d3-mini weight shards/readout share 0 LFS sha256 objects with the measured d3 and d3-turbo siblings (r25-card/sibling-weight-distinct-20261024.txt). Author defaults: --batch-size 8, --verify fast, warm-up on, device cuda:0, no input-length cap; no evaluator launcher, shim, patch or flag changes the model path. bf16 Qwen3.5-4B backbone, noncausal_full_attention (8 full-attention layers bidirectional, 24 Gated-DeltaNet layers causal), SDPA, plain CUDA / PyTorch path; the author's ROCm fast path does not apply on CUDA. fp32 255-way readout, temperature 1.0, permutation_average off, one forward pass per question, no generation. Python stack: torch 2.13.0, transformers 5.17.0, tokenizers 0.23.2, safetensors 0.8.0, accelerate 1.15.0, fla-core 0.5.2 and flash-linear-attention 0.5.2. Base image vllm/vllm-openai:v0.31.0@sha256:a4a4c0437bf7240089da5f08aa370c4aee17ae5290f7a3b468825ee26c4c3a6b; derived image sha256:c22d99ab6aca5d4f947d311da6af754a1d17b9183da92e85cd8a39e66c7fa057, hash-pinned wheel overlay installed offline. vLLM is in the image, not the serving backend. All 22 pinned snapshot files hash-checked in the measuring container; author MODEL_MANIFEST.json sha256 bc638e17982411f6ad0f524d33d0dc1f0db81ad7995c1bfdf93dba7674df5e48. The submitted code was reviewed and cleared at the exact pin; evaluator exposure, the device, pricing and context settings were cleared before the run. The source was reviewed with required safeguards; the runner was reviewed and findings fixed, with cold nonce probes and a maximum-of-repetitions fit check applied before the final review. pre-registration-D3MINI.md Amendments 1 and 2, including the token correction, govern this run. Weights: 4,539,265,536 bf16 backbone parameters plus fp32 readout; two weight shards plus readout.safetensors total 9,081,227,360 bytes. FIRST MEASUREMENT of vLLM-SR d3-mini (our own intake). AUTHOR'S OWN ENGINE on our rented hardware, one request at a time; the unchanged pinned pod_driver_v16 and TypeSafeAdapter drive the author's server. NATIVE OUTPUTS: choice, noul and score probabilities; no generated answer text. AUTHOR EXPOSURE: training data not disclosed by the card; no overlap or contamination audit was run by us. The author's Decision Index figures are a different measurement and are not independently verified by us. CONTEXT: as shipped, no input-length cap (decision_config.json max_length null and /health max_input_tokens null), no lower evaluator-imposed cap and no truncation; all 15 standard long items attempted, as required by ruling on file point 4. This differs from the published d3-turbo row, measured with its author's 32,768-token default cap; the published d3 row also had no cap. All 1,500 items answered HTTP 200, 0 refusals, preregistered class A_all_answered. 10 s RULE: exactly 15 decisions over 10 s, the standard long items at positions in the pre-registered long-item set, taking 10.543-17.503 s; these count wrong in the v1.7 share view (frozen v1.6 and v1.7-off views apply no time limit). Maximum 17.503080 s, median 0.028272 s, p95 0.044697 s. DETERMINISM: first 50 positions run twice, 50/50 bit-identical, 0 argmax flips, maximum probability difference 0.0. This is a distinct checkpoint from d3 and d3-turbo, with no shared LFS weight shards/readout; the zils-ai mirrors are not additional entries.