Messier One v0.4 (Qwen3.5-4B fine-tune, self-hosted)
Data: JevBench v1.6.39, published 2026-10-22
Capability 62.4 · rank #48 of 139 on the open-weights board
Last measured: 2026-10-14 · measurement release: v1.6.22
Weights CC BY-NC 4.0 (non-commercial: free for research and evaluation, commercial licence on request; listing a measured score is evaluation use); code Apache-2.0; base Qwen/Qwen3.5-4B Apache-2.0
intelligence
52.0
calibration
72.9
speed
89.7
cost
61.1
Composite (secondary)
66.2 · Composite rank #23
Cost and measurement conditions
$0.020 per 1,000 decisions (estimated)
ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 dcfcecb0..., the module in force; its base-model references are unchanged since the dae86740... module this row was first priced with - the two later re-freezes each added one hosted-API launch-list key and nothing else): declared base Qwen/Qwen3.5-4B -> deepinfra:Qwen/Qwen3.5-4B USD 0.030000000000000002/M input, USD 0.15/M output, the same basis the live messier-one-v0.2 row carries. The author's server reports NO usage block of any kind, so the input tokens are counted OFFLINE with the author's own Prompter (prompt.py at tag v0.4) at the exact serving settings (--max-state 7500 --max-total 8150, AA labels, qboth layout) and cross-checked against vLLM's own prompt-token counters from the measured pass; output tokens are 0 (the engine is asked for one token restricted to the option labels and nothing is generated for the caller). The reference listing was deprecated after the 25 Sep 2026 cut-off and replaced by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3). Estimated, not charged, not a quote from the author or any host. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them). They are the 15 items the Jev reference refused with max_tokens_exceeded, which the other rows that did not answer them refused or failed on; their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence. This row: USD 0.019767 per 1,000 decisions (0.306x the official cost cap USD 0.064595), against USD 0.022118 over all 1,500 items (v1.6.29).
Measured p50 latency: 0.066 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.28 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.066 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.28 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX A6000 48 GB - different hardware from the H100/RTX 5090 rows of this draw and from the RTX 3090 the author quotes. Started by the author's OWN ./serve_vllm.sh MODEL_DIR 8080 with its defaults, which is what the author insists on for v0.4: vLLM 0.30.0, --max-model-len 8192, chunked prefill and prefix caching OFF (this revision is non-causal), --max-logprobs 256, processed_logprobs, front end --max-state 7500 --max-total 8150 --gate 0.5 --noul-floor 0.801; transformers 5.18.0 exactly as pinned. One request at a time from the unchanged runner. Isolation: the server ran inside a network namespace with loopback only (unshare -n), the cut proven before the sealed input was staged and re-proven immediately before the first sealed request, with the synthetic smoke gate passed in between; a network namespace replaces docker --network none because the pod carried no local image (custody #19335, same property, proven the same way). The pod host kept its network for ssh.