pplx-decider-flash-v1.2 (Perplexity, Qwen3.5-4B full fine-tune, marker readout, self-hosted)
Data: JevBench v1.6.39, published 2026-10-22
Capability 62.5 · rank #47 of 139 on the open-weights board
Last measured: 2026-10-19 · measurement release: v1.6.31
Apache-2.0 (LICENSE + NOTICE in the repo and the HF metadata; base Qwen/Qwen3.5-4B Apache-2.0), public, not gated
intelligence
46.3
calibration
78.7
speed
91.8
cost
64.4
Composite (secondary)
56.6 · Composite rank #34
Cost and measurement conditions
$0.015 per 1,000 decisions (estimated)
ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 6202ef78a2c83cbffc9389cf1f37df70fe0c85ed03f4dd5c5f997f3d31aae13a): declared base Qwen/Qwen3.5-4B -> deepinfra:Qwen/Qwen3.5-4B USD 0.030000000000000002/M input, USD 0.15/M output. Estimated, not charged, not a quote from the author or any host. USD per 1,000 decisions over the full 1,500-item denominator. Tokens as reported by the author's own `Decider.decide()` usage block (input = every non-padding position the request used; output 0, nothing is generated - the logits are read at the markers in one forward pass). The offline count through the author's own protocol.Encoder predicted the server's numbers EXACTLY, item by item: 1,939,205 input tokens over all 1,500 answered items, and the same prediction held on the 231 public items (154,452) before any sealed item was dispatched. COMMON COST BASIS: the preceding full-1,500-item figure is an all-items diagnostic. The published cost in this release uses the same remaining 1,485 items as every other row, under the unchanged v1.6.30 cost rule; USD 0.015339595959595961 per 1,000 decisions. All items still count for Intelligence.
Measured p50 latency: 0.042 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.23 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.042 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.23 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Evaluator-owned rented Verda GPU VM, NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (compute capability 12.0, driver 580.178.04), FIN-02 (Finland, EU), pay-as-you-go non-spot, USD 2.395/h, instance id 0c5da6de-cf37-4754-ba2f-718e990499d4. The author's own `Decider.decide()` from the pinned commit, served behind our own 135-line stdlib HTTP shim (sha256 911747c9ed275c6f5734e4f0bf8466741600ef94960d80641139b1779822c349) on 127.0.0.1:8034 inside a network namespace with no route and a proven cut, offline and credential-free, one request at a time, reference path (fast=False, never fp8). Timed over loopback on the VM. Speed was measured while the host VM was downloading other weights; the measured server remained in its network namespace with the cut proven.