pplx-decider-v1.2 (Perplexity, Qwen3.8-27B full fine-tune, marker readout, self-hosted)
Data: JevBench v1.6.39, published 2026-10-22
Capability 81.7 · Not ranked in the current release
cost 6.6× Jev (v1.5 reference)
Last measured: 2026-10-20 · measurement release: v1.6.33
Apache-2.0 (LICENSE + NOTICE in the repo and the HF metadata; base Qwen/Qwen3.8-27B Apache-2.0), public, not gated
intelligence
72.7
calibration
90.8
speed
86.3
cost
30.0
Composite (secondary)
20.7 · Composite rank #90
Cost and measurement conditions
$0.215 per 1,000 decisions (estimated)
ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 6202ef78a2c83cbffc9389cf1f37df70fe0c85ed03f4dd5c5f997f3d31aae13a): declared base Qwen/Qwen3.8-27B -> qwen/qwen3.8-27b USD 0.42/M input, USD 3.0/M output. Estimated, not charged, not a quote from the author or any host. USD per 1,000 decisions on the denominator of the release basis (the same-items cost rule for this draw prices the common item set without the 15 very long items, for every row alike). Same pair as the published pplx-decider-v1-1-27b and vllm-sr-d3 rows. Tokens as reported by the author's own `Decider.decide()` usage block (input = every non-padding position the request used; output 0, nothing is generated - the logits are read at the markers in one forward pass). The offline count through the author's own protocol.Encoder predicted the server's numbers EXACTLY, item by item: 1,939,205 input tokens over all 1,500 answered items, and the same prediction held on the 231 public items (154,452) before any sealed item was dispatched. OVER THE OFFICIAL COST CAP on the common 1,485-item basis: USD 0.214754343 per 1,000 decisions (3.325x cap); the row stays RANKED, but is outside the official cost-capped views and visible in uncapped views. ALL-ITEMS DIAGNOSTIC: USD 0.542977 per 1,000 over all 1,500 answered items (8.41x cap). All 1,500 items count for Intelligence.
Measured p50 latency: 0.142 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.43 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.142 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.43 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Evaluator-owned rented Verda GPU VM, one NVIDIA H200 141 GB (compute capability 9.0, driver 580.178.04), FIN-02 (Finland, EU), pay-as-you-go non-spot, instance type 2H200.141S.88V at USD 11.32/h with the second GPU unbound from the driver before anything ran (nvidia-smi and torch both saw exactly one GPU; single-GPU, no tensor parallelism). The author's own `Decider.decide()` from the pinned commit, served behind our own 135-line stdlib HTTP shim (sha256 911747c9ed275c6f5734e4f0bf8466741600ef94960d80641139b1779822c349) on 127.0.0.1:8035 inside a network namespace with no route and a proven cut, offline and credential-free, one request at a time, reference path (fast=False, never fp8). Timed over loopback on the VM. Measured 20 Oct 2026.