Image JevBench v0.3.0 · individual system

H2O-Lightning-31B v1.3.4 @c35ed950 (image)

LLM decoderBase model: google/gemma-4-31B-itsource

Published source

local/GPU evaluation

Image JevBench v0.3.0 composite score

50.965

Rank #7 of 65 ranked systems.

From the published full-benchmark aggregate. The composite combines intelligence, calibration, speed and cost.

Published axes

intelligence
74.0
calibration
86.7
speed
83.8
cost
43.7

Cost and speed evidence

Cost per 1,000 decisions
USD 0.075428
Cost basis
measured GPU seconds x $1.19/GPU-hour
Measured latency · p50 / p95
0.220 s / 0.279 s
Adjusted latency · p50 / p95
0.590 s / 0.707 s
Latency adjustment
v1.4 local latency adjustment
Public accuracy
81.7%
Sealed accuracy
82.9%
Measurement note
Official ImageJev v0.3 S1200+P300 pass 1 on 17 Nov 2026, one read of the 1,500-item self-hosted draw. Model h2oai/h2o-lightning-31b at c35ed9505f6744770dc0d74bd3f468791a7984af (v1.3.4), base google/gemma-4-31B-it; new safetensors LFS oids versus v1.0.7 (both shards differ). Served with unmodified serve.sh --profile two-way, rather than v1.0.7's one-engine profile. vLLM 0.30.0 image sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90; h2oai/vllm overlay @a36449a18b4465b4f70fa078f5403032de414de8; torch 2.13.0, transformers 5.20.0, torchcodec 0.17.0. Online FP8 W8A8 from BF16 checkpoint, dtype BF16, TRITON_ATTN, max_model_len 131072, two_way_window 8192, max_num_batched_tokens 9216, max_num_seqs 1024, GPU utilization 0.92. State budget 126976, selector v2, auto one-way above the 4096-token prompt/window threshold. One read, readout_orders 1, no image reread; images max 4, 560 soft tokens, 20 MiB each, data URIs only, no max_pixels override. Image temperature 0.9 (v1.0.7: one-way 0.59, two-way 0.54); one-way choice/score/noul temperatures 1.28/0.7/0.6, two-way 1.13/0.7/1.07 (v1.0.7 choice/score/noul: one-way 0.7/0.7/0.6, two-way 0.7/0.7/0.5); noul floor 0.801, cutoff 0.8. AR off, no reason/router/residual configuration; server default mode jevbench. One RTX PRO 6000 Blackwell, offline after hash verification; serial requests, no retries. 1,500 of 1,500 answered, 0 errors. Latency and raw gpu_s use the host performance-counter wall clock of each loopback HTTP request, including shim/server overhead, as for v1.0.7; other self-hosted rows generally use CUDA-event timing. The card discloses temperatures and yes/no floor fitted on v1.3.0 weights using Decision Index 0.3 public per-row reads; two-way choice confirmed on SuperGPQA (1,990), decision-index image temperature 0.96 confirmed on MMStar (1,477), while the measured other-profile image temperature 0.9 was not refitted. The inactive packaged router was fitted on HLE-style sets with public JevBench 231 held out; it was disabled here. Disclosure only; no claim of training-set exclusion. TypeSafe-20 not measured for this revision (TS20 stage stopped with a container error on 17 Nov 2026; no output). The v1.0.7 row remains a separate ranked, unchanged published snapshot.

Published 2026-11-17. Scores and ranks can change in a later release. See the method notes.