metask-jev-rain-12B-trained
Data: JevBench v1.6.39, published 2026-10-22
Capability 77.9 · rank #9 of 139 on the open-weights board
Last measured: 2026-10-12 · measurement release: v1.6.10
Weights: Apache-2.0 per model card (base Gemma 4 under the Gemma 4 licence); service code MIT (github.com/chenyuwang166-oss/metask-jev-rain-12B-trained@e36c7a54)
intelligence
67.8
calibration
87.9
speed
73.4
cost
52.7
Composite (secondary)
68.1 · Composite rank #17
Cost and measurement conditions
$0.038 per 1,000 decisions (estimated)
v1.5-M2 base-model reference: base google/gemma-4-12B-it (author int4 AWQ of google/gemma-4-12B-it-qat-q4_0-unquantized + own LoRA). google/gemma-4-12b-it is not listed on OpenRouter; nearest listed sibling of the same family and size class (fixed Gemma-4-12B precedent) OpenRouter google/gemma-3-12b-it USD 0.05/M input, USD 0.15/M output, read 2026-10-12T00:13:38Z from https://openrouter.ai/api/v1/models x the service's own measured usage tokens; estimated, not charged. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them). They are the 15 items the Jev reference refused with max_tokens_exceeded, which the other rows that did not answer them refused or failed on; their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence. This row: USD 0.037821 per 1,000 decisions (0.586x the official cost cap USD 0.064595), against USD 0.041404 over all 1,500 items (v1.6.29).
Measured p50 latency: 0.115 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.38 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.115 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.38 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
evaluator-owned Lium GPU pod (H100 80GB), offline read-only container (--network none); author's own service (serve.sh -> launch.py -> server.py on vLLM 0.31.0, pinned digest image), unchanged defaults Paid fast-lane order 66c7b6d2 (public). Own LoRA adapter (rank 16) on the author's int4 Gemma 4 12B base, applied unmerged by vLLM. Replaces nothing: the earlier metask_jev_rain_12b row (third-party AWQ wrapper, order f1b81eec) is a different system.