H2O-Lightning-31B v1.0.7 (H2O.ai, Gemma-4-31B-it fine-tune, vLLM + h2oai overlay, one-engine FP8 profile, self-hosted)
Data: JevBench v1.6.39, published 2026-10-22
Capability 85.2 · rank #2 of 139 on the open-weights board
Last measured: 2026-10-22 · measurement release: v1.6.39
H2O Model Non-Commercial Licence v1.1 (weights; gated on Hugging Face; benchmarking and publishing results are allowed by the licence). Base Gemma 4 under its own licence.
intelligence
80.2
calibration
90.2
speed
89.8
cost
50.6
Composite (secondary)
73.4 · Composite rank #2
Cost and measurement conditions
$0.044 per 1,000 decisions (estimated)
ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 a8a05890ddab2d39d1a1bd543f2b5c83842bdc70aa54766788bc89b7a8a448d7): declared base google/gemma-4-31B-it -> reference() -> google/gemma-4-31b-it, USD 0.09 per million input tokens, USD 0.34 per million output tokens (frozen price cut-off snapshot), the same reference as the other Gemma-4-31B-it rows of this board. The customer asked for the cost of the base model. Estimated, not charged, not a quote from the author or any host. Tokens as reported by the author's server usage block (the full record, 1 output token per decision for the readout). Over all 1,500 items the frozen resolve() gives USD 0.115415 per 1,000 decisions; the official cost axis uses the common 1,485-item basis (below), where the row is under the official cost cap. COMMON COST BASIS (v1.6.30): common 1,485 items, excluding the 15 longest for every row; 727,319 input / 1,485 output tokens. All-item usage 1,917,924 input / 1,500 output over 1,500 items; the 15 longest items, which this row answered in full, count for no row's cost.
Measured p50 latency: 0.063 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.28 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.063 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.28 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Paid public fast-lane order 5fa1a5f0 (H2O.ai). Evaluator-owned rented Verda GPU VM (1RTXPRO6000.30V, FIN-03, Finland, EU, pay-as-you-go, USD 2.535/h), 1x NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (97,887 MiB). Run image built on the digest-pinned official vllm/vllm-openai:v0.30.0 with the author's overlay and shim installed before any test item existed, all 26 weight files sha256-verified at revision ca6e6f6e, then run in an offline container (network none), the cut proven from inside before the sealed input and again before the scored run. Driven serially, one request at a time, by the unchanged self-hosted runner. Measured 22 Oct 2026 on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720...); determinism 50/50 bit-identical; raw latency p50 0.065 s, p95 0.126 s, maximum 39.08 s. The standard self-hosted x2 + 0.15 s adjustment applies (assumption, not measured).