H2O-Lightning-31B v1.3.4 @c35ed950 (image)
LLM decoderBase model: google/gemma-4-31B-itsourcelocal/GPU evaluation
Image JevBench v0.3.0 composite score
50.965
Rank #7 of 65 ranked systems.
From the published full-benchmark aggregate. The composite combines intelligence, calibration, speed and cost.
Published axes
- intelligence
- 74.0
- calibration
- 86.7
- speed
- 83.8
- cost
- 43.7
Cost and speed evidence
- Cost per 1,000 decisions
- USD 0.075428
- Cost basis
- measured GPU seconds x $1.19/GPU-hour
- Measured latency · p50 / p95
- 0.220 s / 0.279 s
- Adjusted latency · p50 / p95
- 0.590 s / 0.707 s
- Latency adjustment
- v1.4 local latency adjustment
- Public accuracy
- 81.7%
- Sealed accuracy
- 82.9%
- Measurement note
- Official ImageJev v0.3 S1200+P300 pass 1 on 17 Nov 2026, one read of the 1,500-item self-hosted draw. Model h2oai/h2o-lightning-31b at c35ed9505f6744770dc0d74bd3f468791a7984af (v1.3.4), base google/gemma-4-31B-it; new safetensors LFS oids versus v1.0.7 (both shards differ). Served with unmodified serve.sh --profile two-way, rather than v1.0.7's one-engine profile. vLLM 0.30.0 image sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90; h2oai/vllm overlay @a36449a18b4465b4f70fa078f5403032de414de8; torch 2.13.0, transformers 5.20.0, torchcodec 0.17.0. Online FP8 W8A8 from BF16 checkpoint, dtype BF16, TRITON_ATTN, max_model_len 131072, two_way_window 8192, max_num_batched_tokens 9216, max_num_seqs 1024, GPU utilization 0.92. State budget 126976, selector v2, auto one-way above the 4096-token prompt/window threshold. One read, readout_orders 1, no image reread; images max 4, 560 soft tokens, 20 MiB each, data URIs only, no max_pixels override. Image temperature 0.9 (v1.0.7: one-way 0.59, two-way 0.54); one-way choice/score/noul temperatures 1.28/0.7/0.6, two-way 1.13/0.7/1.07 (v1.0.7 choice/score/noul: one-way 0.7/0.7/0.6, two-way 0.7/0.7/0.5); noul floor 0.801, cutoff 0.8. AR off, no reason/router/residual configuration; server default mode jevbench. One RTX PRO 6000 Blackwell, offline after hash verification; serial requests, no retries. 1,500 of 1,500 answered, 0 errors. Latency and raw gpu_s use the host performance-counter wall clock of each loopback HTTP request, including shim/server overhead, as for v1.0.7; other self-hosted rows generally use CUDA-event timing. The card discloses temperatures and yes/no floor fitted on v1.3.0 weights using Decision Index 0.3 public per-row reads; two-way choice confirmed on SuperGPQA (1,990), decision-index image temperature 0.96 confirmed on MMStar (1,477), while the measured other-profile image temperature 0.9 was not refitted. The inactive packaged router was fitted on HLE-style sets with public JevBench 231 held out; it was disabled here. Disclosure only; no claim of training-set exclusion. TypeSafe-20 not measured for this revision (TS20 stage stopped with a container error on 17 Nov 2026; no output). The v1.0.7 row remains a separate ranked, unchanged published snapshot.
Published 2026-11-17. Scores and ranks can change in a later release. See the method notes.