Bongard-mini (AgentBull, T5Gemma-2 4B-4B)
Data: JevBench v1.6.40, published 2026-10-22
Capability 54.8 · Not ranked in the current release
unpriced: the frozen pricing path has no market-listing decision for the declared base google/t5gemma-2-4b-4b and raises rather than estimating; one listing decision would make the row rankable.
Last measured: 2026-10-16 · measurement release: v1.6.24
Gemma terms of use
intelligence
25.2
calibration
84.4
speed
90.6
cost
—
Composite (secondary)
— · Not ranked on this board
Cost and measurement conditions
not published (not published)
UNPRICED. The frozen pricing_v15 module (sha256 dcfcecb0...2835, and every later re-freeze, which only add hosted-API launch-list keys) RAISES on the declared base google/t5gemma-2-4b-4b ('base model lacks an explicit market-listing decision'); no price is invented here. Axes only: Intelligence, Calibration and Speed are computed; Cost, composite and rank are void. Tokens as reported by the author's server (input 786,538, output_tokens 0).
Measured p50 latency: 0.070 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.29 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.070 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.29 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Evaluator-owned rented Lium GPU pod (measurement receipts on file), 1x NVIDIA RTX 5090 32 GB (sm_120, driver 595.99.02, Ukraine node substituted at rental, pre-registration Amendment 2) - different hardware from the RTX PRO 6000 / H100 of the earlier runs, so Speed is as measured on this card and not comparable to the earlier rows. Python packaging: uv venv from a hashed lock at the old pins (declared deviation). Isolation: author server started inside a network namespace with loopback only (unshare -n), the cut proven before and after the sealed input was staged, synthetic smoke gate passed in between; runner unchanged (run_v15.py --resume --resume-keep-failures); one use of input-selfhosted-1500 (custody #20150, under the pre-registration). Published Speed uses the standing x2 + 0.15 s self-hosted adjustment. Author's own `bongard serve` with their temperatures.json, default token budgets (64,000 request / 32,768 sequence), bf16.