Torchcast Decision 4B (v1.1)
Data: JevBench v1.6.39, published 2026-10-22
Capability 68.7 · rank #29 of 139 on the open-weights board
Last measured: 2026-10-12 · measurement release: v1.6.12
CC BY-NC 4.0 (weights); base Qwen/Qwen3.5-4B Apache-2.0; serving package Apache-2.0
intelligence
54.2
calibration
83.1
speed
93.6
cost
64.8
Composite (secondary)
70.7 · Composite rank #8
Cost and measurement conditions
$0.015 per 1,000 decisions (estimated)
ESTIMATE, v1.5-M2 base-model reference rule: self-hosted open weights, no bookable tariff. Base model Qwen/Qwen3.5-4B (model card base_model). OpenRouter does not list Qwen3.5-4B (host snapshot source/public-docs/OPENROUTER-MODELS.json, fetched 2026-10-12T08:15:26Z, read 2026-10-12). Binding precedent INTERPRETATION-1 I-2/I-3: DeepInfra Qwen/Qwen3.5-4B, USD 0.03/M input, USD 0.15/M output, frozen 25 Sep 2026 price cut-off snapshot (listing later deprecated; frozen reference kept). Logit readout, no generation: server reports 0 output tokens. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them). They are the 15 items the Jev reference refused with max_tokens_exceeded, which the other rows that did not answer them refused or failed on; their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence. This row: USD 0.014895 per 1,000 decisions (0.231x the official cost cap USD 0.064595), against USD 0.019658 over all 1,500 items (v1.6.29).
Measured p50 latency: 0.021 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.19 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.021 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.19 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
evaluator-owned Lium GPU pod (H100), offline read-only container (--network none); author's torchcast_decision server unchanged at tag v1.1.0 with the bundled decision_config.json (bf16, one forward pass reading option letters, calibration temperature 0.9, yes/no commit floor 0.801) on the documented stack torch 2.9.1+cu128 / transformers 5.8.1 / flash-linear-attention 0.5.2 / causal-conv1d 1.7.0 (fast kernels active, CUDA graphs on) Paid fast-lane order 529be5d5 (public). Qwen3.5-4B fine-tune served by the author's own torchcast_decision package. Runtime: official pytorch/pytorch:2.9.1-cuda12.8-cudnn9-devel image (digest-pinned) plus an offline hash-locked overlay of the documented packages (Python 3.11 instead of the documented 3.12). decision_config.json sets a disclosed yes/no commit floor of 0.801: a yes/no probability with max(P, 1-P) below 0.801 is moved to 0.801 on its own side (answer unchanged; same mechanism as h2o-lightning-4b). Prompts over 16,384 tokens keep the head and the tail of the state (documented). Output tokens are 0 (no generation).