Arcbright Elf 1 12B
Data: JevBench v1.6.39, published 2026-10-22
Capability 65.0 · rank #37 of 139 on the open-weights board
Last measured: 2026-10-20 · measurement release: v1.6.34
Apache-2.0 (model card, root LICENSE and serve/LICENSE; base google/gemma-4-12B-it under the Gemma 4 licence), public, not gated
intelligence
43.8
calibration
86.2
speed
92.0
cost
57.4
Composite (secondary)
48.8 · Composite rank #41
Cost and measurement conditions
$0.026 per 1,000 decisions (estimated)
ESTIMATE (*est.), nearest-listed-sibling reference: declared base google/gemma-4-12B-it has no market listing (UNLISTED in the frozen pricing_v15 module, sha256 6202ef78a2c83cbffc9389cf1f37df70fe0c85ed03f4dd5c5f997f3d31aae13a), priced through the frozen catalog's nearest listed sibling of the same family and size class: google/gemma-3-12b-it USD 0.05/M input, USD 0.15/M output (the fixed Gemma-4-12B precedent, same basis as the published metask_jev_rain_12b_trained row). Estimated, not charged, not a quote from the author or any host. USD per 1,000 decisions on the denominator of the release basis (the same-items cost rule for this draw prices the common item set without the 15 very long items the author's server refuses, for every row alike). Tokens as reported by the author's own server usage block (input_tokens = tokens of the rendered chat prompt incl. system prompt, options and the answer prefill; output_tokens = 1, exactly one vLLM forward pass per answered single-question row - nothing is generated, the option-code logits are read in that one pass). The offline count through the author's own render.py and the pinned tokenizer predicted the server's numbers EXACTLY, item by item: 777,809 input tokens over the 1,485 answered items, and 15 HTTP 422 refusals at the pre-registered ids and positions.
Measured p50 latency: 0.034 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.22 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.034 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.22 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Evaluator-owned rented Verda GPU VM, one NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (compute capability 12.0, driver 580.178.04), FIN-03 (Finland, EU), pay-as-you-go non-spot, instance type 1RTXPRO6000.30V at USD 2.477/h. The author's own `python3 -m elf_serve` from the pinned commit, ELF_MODEL pointed at the commit-pinned local snapshot, on 127.0.0.1:8000 inside a network-none Docker container with the cut proven from inside (loopback only, no default route, DNS and TCP fail), offline and credential-free, one request at a time. Timed over loopback on the VM. Measured 20 Oct 2026.