Clef-omni
Data: JevBench v1.6.40, published 2026-10-22
Capability 58.3 · Not ranked in the current release
cost 8.5× Jev (v1.5 reference)
Last measured: 2026-10-10 · measurement release: v1.6.6
Apache-2.0
intelligence
34.5
calibration
82.2
speed
86.8
cost
26.8
Composite (secondary)
6.1 · Composite rank #142
Cost and measurement conditions
$0.276 per 1,000 decisions (estimated)
ESTIMATE (*est.), base-class reference. Self-hosted open weights; the declared base Qwen/Qwen3-Omni-30B-A3B-Instruct is not a key of any frozen pricing table. Method as the v1.6.5 Liquid rows: the nearest same-family entry of the frozen 25 Sep 2026 snapshot, deepinfra:Qwen/Qwen3-VL-30B-A3B-Instruct (the multimodal Qwen3 30B-A3B), at USD 0.15 per 1M input and USD 0.60 per 1M output. The authors' systemone() has no generation path, so output_tokens is a literal 0. This row's own usage on draw v1.6-regular-20261010-a2: 2,755,730 input and 0 output tokens over 1,500 decisions.
Measured p50 latency: 0.143 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.44 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.143 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.44 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
evaluator-owned Lium RTX PRO 6000 Blackwell 96 GB pod (Beltsville US, 29 min, USD 1.19/h); the authors' own load_release_model() + systemone() from the pinned repo's joint_schema_model.py behind the run-39 loopback shim (127.0.0.1, single-threaded, no answer arithmetic), pointed at the LOCAL PINNED SNAPSHOT DIRECTORY, defaults unchanged (max_length 64,000). Stack as the model card tested: torch 2.11.0+cu130, transformers 5.10.2, plus torchvision 0.26.0+cu130, which the Omni video processor requires to load. HF/transformers offline, no credentials, pod-level iptables egress cut installed and verified before the sealed input was uploaded. All 17 LFS files verified against the Hugging Face tree digests. GPU memory 62,729 MiB after load and at most 74,735 MiB during the run (per-minute log), of 97,887 MiB.