metask-jev-rain-31B-trained service v3
Data: JevBench v1.6.39, published 2026-10-22
Capability 79.6 · rank #6 of 139 on the open-weights board
Last measured: 2026-10-13 · measurement release: v1.6.17
Weights: Apache-2.0 per model card (base Gemma 4 under the Gemma 4 licence); service code (service/) MIT
intelligence
72.6
calibration
86.6
speed
84.7
cost
49.3
Composite (secondary)
67.7 · Composite rank #20
Cost and measurement conditions
$0.049 per 1,000 decisions (estimated)
v1.5-M2 base-model reference: exact base google/gemma-4-31B-it, OpenRouter google/gemma-4-31b-it list price USD 0.09/M input, USD 0.34/M output (the same reference as the published metask_jev_rain_31b_trained row; snapshot read 2026-10-13T12:10:25Z by the order's public-docs receipt) x the service's own measured usage tokens (840,049 in / 17,397 out); estimated, not charged. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them). They are the 15 items the Jev reference refused with max_tokens_exceeded, which the other rows that did not answer them refused or failed on; their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence. This row: USD 0.049062 per 1,000 decisions (0.760x the official cost cap USD 0.064595), against USD 0.054346 over all 1,500 items (v1.6.29).
Measured p50 latency: 0.181 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.51 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.181 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.51 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
evaluator-owned Lium GPU pod (RTX PRO 6000 96 GB; the reviewed recipe needs 96 GB), offline read-only container (--network none, verified by docker inspect before the input upload); the author's own service at revision 2feff076 (service v3), unchanged defaults. Paid public fast-lane order 4d9fef10. RE-SUBMISSION, DISCLOSED: the same LoRA weights as the published metask_jev_rain_31b_trained row (order 147c5e18, v1.6.10) with a changed service/ layer (e.g. its yes/no constant); those changes may have been tuned on that row's published v1.6.10 aggregates (custodian #18945). Both rows are shown; no third same-weights row is admitted on this draw. Weights file shas: model 1b9b1d62..., adapter 2b956899..., code tree 6a1234ee...