Jez-large-1.0.0 (pragmaticfrontier, Gemma-4-E4B label-logit decision model, author server, self-hosted)v1.7 · 10 s rule
Data: JevBench v1.7.0, published 2026-10-24 · method v1.7 (10 s rule), measured under the rule
Capability 54.4 · rank #95 of 157 on the open-weights board
Last measured: 2026-10-22 · measurement release: v1.7.0
Apache-2.0 (HF card metadata; the card also links the Gemma 4 licence terms for the base; the HF repo has no LICENSE file; author code repo is MIT), public, not gated
intelligence
25.6
calibration
83.2
speed
90.8
cost
70.5
Composite (secondary)
13.8 · Composite rank #140
Cost and measurement conditions
$0.0096 per 1,000 decisions (estimated)
ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 a8a05890ddab2d39d1a1bd543f2b5c83842bdc70aa54766788bc89b7a8a448d7; this base resolves identically under the earlier 1f4d1f7b...): base string google/gemma-4-E4B-it (the card's stated base_model, unverified author statement) priced as google/gemma-4-E4B-it -> deepinfra:google/gemma-4-E4B-it USD 0.02/M input, USD 0.1/M output. Estimated, not charged, not a quote from the author or any host. Tokens as reported by the author server's own usage block: 712,537 input on the 1,485 answered items (equal to our offline count item by item, pre-registered), 0 output tokens (the model reads label probabilities at the answer position, no generation); the author count bills the state block once plus each given-order question block, not the chat template and not the reversed-order second probe, and that native field is used as reported; the 15 refused items count 0 tokens.
Measured p50 latency: 0.060 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.27 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.060 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.27 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Evaluator-owned rented Verda GPU VM, 1x NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (FIN-03, Finland, EU, pay-as-you-go non-spot, USD 2.525/h), shared by the two Jez rows as two separate back-to-back uses (sealed input shredded from the node between them). The author's own server as shipped (torch backend, no vLLM serving) in a pre-built image on the stock vllm/vllm-openai:v0.30.0 base (torch 2.13.0+cu130, transformers 5.19.0, tokenizers 0.23.3, flash-linear-attention 0.5.2), container with --network none (cut proven from inside before the sealed input), Hub offline, every credential variable empty, read-only bundle, all 14 files of the pinned revision (15,516,502,246 bytes) sha256-verified inside the container against manifest 11092a9e... and all 169 author code files verified; our first-party loopback proxy only maps the author's over-limit 400 to 422; driven by the unchanged pinned pod_driver_v16, one request at a time. The standard self-hosted x2 + 0.15 s adjustment applies. Measured 22 Oct 2026 (05:35Z start of this row) on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720...); determinism 50/50 bit-identical; VM and volume destroyed afterwards.
Measured latency over 1,500 decisions: p50 0.060 s · p95 0.081 s
0 decisions over 10 s.
A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.
Model and serving disclosure
Author server pragmaticfrontier/jez at bf7c158169e5ff1db75a4b3ef3e1161cd5f50e91 (MIT; python3 -m jez.server, torch backend, --no-auth), started directly as shipped from an unchanged source copy (independent static review PASS WITH FINDINGS; no subprocess, eval/exec, pickle or trust_remote_code in the reviewed server closure; no branch on item text, task id or benchmark name). The review findings were the runtime-download fallbacks in the author code (hf_hub_download for the calibration file and from_pretrained base resolution when the checkpoint is a Hub id); they were never reached: the merged checkpoint was loaded from the local pinned /model with its own /model/calibration.json, Hub offline, container --network none, every credential variable empty; the owner cleared the code for one offline measurement on those terms. Author settings kept: order-debiased scoring (JEZ_DEBIAS=1, given and reverse order averaged), JEZ_MAX_BATCH=16, the checkpoint's calibration temperatures unchanged; no score fitting, no prompt transformation. FIRST MEASUREMENT (found by our own intake of new public Hugging Face decision models; not a free-queue submission, not in the Decision Index at pick-up). CONTEXT CAP: the author server refuses a question whose state+question exceeds 32,000 tokens and a request above 64,000 tokens (HTTP 400 max_tokens_exceeded); our reviewed loopback status proxy maps exactly that answer to 422 and passes everything else through unchanged; on this draw that is exactly the 15 items of 76k-81k tokens, pre-registered by id and position; they count as wrong, like elf-1-12b, seb-9b and the NeoHorse rows on this draw. SAME FAMILY: jez-large-1.0.0 (Gemma-4-E4B base) and jez-small-1.0.0 (Qwen3.5-0.8B base) are two checkpoints of one author measured back to back on one VM. PROVENANCE, author statements (unverified): LoRA r64/alpha128 merged into google/gemma-4-E4B-it (card); trained on 278,999 gold-label examples from 20 public sources (reports/mix-v2-manifest.json, not audited by us), no teacher model; the card reports its own acceptance gate 2 (Tier 1 vs base) as not met and states no JevBench or Decision Index result. No rule against public items was found in the reviewed code. LATENCY: maximum single decision 0.118 s on this card; no decision came near the 10 s limit. Determinism probe on the first 50 positions, two passes: 50/50 bit-identical.