iota-decide-4b (Macrocosmos, Clef joint-head 4B, self-hosted)
Data: JevBench v1.6.40, published 2026-10-22
Capability 57.1 · rank #69 of 139 on the open-weights board
Last measured: 2026-10-21 · measurement release: v1.6.38
Apache-2.0 (HF metadata, LICENSE, NOTICE; README notes CC-BY-SA NLI training records), public, not gated
intelligence
29.4
calibration
84.8
speed
87.5
cost
63.0
Composite (secondary)
18.9 · Composite rank #99
Cost and measurement conditions
$0.017 per 1,000 decisions (estimated)
ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 a8a05890ddab2d39d1a1bd543f2b5c83842bdc70aa54766788bc89b7a8a448d7): base Qwen/Qwen3.5-4B-Base priced as Qwen/Qwen3.5-4B -> deepinfra:Qwen/Qwen3.5-4B USD 0.030000000000000002/M input, USD 0.15/M output. Estimated, not charged, not a quote from the author or any host. Tokens as reported by the server usage block: 1,094,798 input on 1,500 answered items, 0 output tokens (one forward pass, joint-head readout, no generation). COMMON COST BASIS (v1.6.30): common 1,485 items, excluding the 15 longest for every row; 849,038 input / 0 output tokens. All-item usage 1,094,798 input / 0 output over 1,500 items.
Measured p50 latency: 0.117 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.38 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.117 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.38 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Evaluator-owned rented Verda GPU VM, 1x NVIDIA A100-SXM4-80GB (FIN-03, Finland, EU, pay-as-you-go non-spot, USD 2.141/h). Container with --network none (cut proven from inside before the sealed input), Hub offline, every credential unset, read-only weights, all 16 weight files sha256-verified; driven one request at a time. The standard self-hosted x2 + 0.15 s adjustment applies. Measured 21 Oct 2026 on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720...); determinism 50/50 bit-identical; VM and volume destroyed afterwards.