Uniphore-4B-Decider
Data: JevBench v1.6.39, published 2026-10-22
Capability 57.1 · rank #70 of 139 on the open-weights board
Last measured: 2026-10-12 · measurement release: v1.6.11
not stated in the model card (no licence field; base Qwen/Qwen3.5-4B Apache-2.0)
intelligence
34.2
calibration
80.0
speed
87.7
cost
64.5
Composite (secondary)
27.2 · Composite rank #73
Cost and measurement conditions
$0.015 per 1,000 decisions (estimated)
v1.5-M2 base-model reference: frozen DeepInfra snapshot jevbench-v15-measure-20260925 receipts/DEEPINFRA-MODELS-20260925.json (SHA-256 84eb7926acd4e93b71d695be24898e2ca999b20011d1b66cc0348ce7c6c3546f); listing Qwen/Qwen3.5-4B (exact base, merge_info.json base_revision 851bf6e8), USD 0.03/M input and USD 0.15/M output. Note (METHOD-v1.5-ADDENDUM-PRICING-INTERPRETATION-1 I-3): the frozen DeepInfra Qwen/Qwen3.5-4B entry is flagged deprecated (replaced_by Qwen/Qwen3.5-9B) while still showing this price; it remains the frozen reference. The OpenRouter /api/v1/models snapshot in source/public-docs (fetched 2026-10-12T07:05:09Z) lists no Qwen3.5-4B route. Self-hosted GPU deployment has no provider tariff, so price_kind is estimate. The native readout is one causal forward pass with no generated tokens: usage.output_tokens is 0 by construction (logits_to_keep=1). COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them). They are the 15 items the Jev reference refused with max_tokens_exceeded, which the other rows that did not answer them refused or failed on; their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence. This row: USD 0.015205 per 1,000 decisions (0.235x the official cost cap USD 0.064595), against USD 0.015053 over all 1,500 items (v1.6.29).
Measured p50 latency: 0.101 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.35 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.101 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.35 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
evaluator-owned Lium GPU pod (H100), offline read-only container (--network none); author's inference.py DecisionModel.predict_jev_task unchanged (bf16, sdpa, one forward pass, softmax over single-token option codes, temperatures.json binary T 1.0353 / choice 1.0) behind a first-party loopback TypeSafe shim; transformers 5.17 reference PyTorch kernels (no flash-linear-attention / causal-conv1d in the image) Paid fast-lane order 4b3cb7b8 (public). Merged text-only Qwen3.5-4B LoRA-SFT checkpoint (Qwen3_5ForCausalLM). Runtime vllm/vllm-openai:v0.31.0 image (torch 2.13, transformers 5.17; vLLM unused) instead of the documented torch 2.8 / transformers 5.12 + fla kernels, so latency was measured on the reference PyTorch linear-attention path. Requests over the documented 16,384-token wrapper limit answer HTTP 422 (refused, not truncated). Choice options are presented in the harness's criteria order (labels are not sent), unlike the author's historical label-order evaluator. Output tokens are 0 (no generation). The author discloses that the public JevBench set was inspected during development.