Jev vs H2O-Lightning-31B v1.3.4: published benchmark comparison

Data: JevBench v1.8.18, published 2026-11-20 · method v1.7 (10 s rule), re-scored from the recorded per-decision times

Rank #3 on the open-weights board in v1.8.18.

Compare Jev-compatible models by separate measures and their measurement conditions. A historical measurement is not a current rank.

Jev and H2O-Lightning-31B v1.3.4: published axes and category radars

Pick any two. Radars for the score axes, capability by subject topic and use cases, chance-corrected competence per request type on the open and sealed sets, and competence per tier on each set. Further out is better on every spoke; the link keeps the pair.

  • A: H2O-Lightning-31B v1.3.4 — Open weights · LLM decoder · Score 79.3 (#3)
  • B: Jev 1.13.0REFERENCE · TypeSafe — Jev — reference (TypeSafe, closed) · Score 71.4 (ranked, not ranked)

The four score axes

Radar: the four score axes, two systemsThe four score axes, H2O-Lightning-31B v1.3.4 vs Jev 1.13.0. Intelligence: 85.0 vs 63.6; Calibration: 90.7 vs 90.6; Speed: 90.3 vs 91.2; Cost: 60.2 vs 54.7.50100Intelligence: H2O-Lightning-31B v1.3.4: 85.0; Jev 1.13.0: 63.6Intelligence85.0 · 63.6Calibration: H2O-Lightning-31B v1.3.4: 90.7; Jev 1.13.0: 90.6Calibration90.7 · 90.6Speed: H2O-Lightning-31B v1.3.4: 90.3; Jev 1.13.0: 91.2Speed90.3 · 91.2Cost: H2O-Lightning-31B v1.3.4: 60.2; Jev 1.13.0: 54.7Cost60.2 · 54.7
0–100, the values in the table. An axis a system has no published value for is left as a gap (it counts as 0 in the composite).

Capability by subject topic

H2O-Lightning-31B v1.3.4: subject topics: not labelled for this draw yet (top-up pending)

Radar: capability by subject topic, two systemsCapability by subject topic, H2O-Lightning-31B v1.3.4 vs Jev 1.13.0. Rules, policy & law: subject topics: not labelled for this draw yet (top-up pending) vs 49.4; Coding & software: subject topics: not labelled for this draw yet (top-up pending) vs 64.3; Math & numbers: subject topics: not labelled for this draw yet (top-up pending) vs 10.4; Finance & commerce: subject topics: not labelled for this draw yet (top-up pending) vs 40.9; Support & operations: subject topics: not labelled for this draw yet (top-up pending) vs 48.8; Everyday language: subject topics: not labelled for this draw yet (top-up pending) vs 77.5; Safety & security: subject topics: not labelled for this draw yet (top-up pending) vs 52.2. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Rules, policy & law: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 49.4 — Rules, policy & law: applying written rules: company policies, contracts, regulations, eligibility. 1616 items (151 open, 1465 sealed).Rules, policy& lawn/a · 49.4Coding & software: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 64.3 — Coding & software: code, SQL, repositories, developer tools and IT systems. 382 items (62 open, 320 sealed).Coding &softwaren/a · 64.3Math & numbers: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 10.4 — Math & numbers: a calculation decides the answer: arithmetic, word problems, probability, dates, units. 370 items (37 open, 333 sealed).Math &numbersn/a · 10.4Finance & commerce: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 40.9 — Finance & commerce: money: payments, refunds, invoices, orders, expenses, insurance payouts. 223 items (12 open, 211 sealed).Finance &commercen/a · 40.9Support & operations: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 48.8 — Support & operations: support tickets, incidents, logistics, scheduling desks and routing work to a team. 160 items (21 open, 139 sealed).Support &operationsn/a · 48.8Everyday language: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 77.5 — Everyday language: short everyday messages: intents, assistant requests, reading a detail out of a text. 145 items (11 open, 134 sealed).Everydaylanguagen/a · 77.5Safety & security: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 52.2 — Safety & security: untrusted or injected instructions, fraud, moderation, access and security triage. 109 items (6 open, 103 sealed).Safety &securityn/a · 52.2

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Measured subject-topic competence on this release’s item set. Chance-corrected competence per category (0 = at or below chance; negative averages are reported as 0, 100 = perfect), H2O-Lightning-31B v1.3.4: open+sealed; Jev 1.13.0: S+P+L1+L2+L3 items pooled; Breakdown values are shown for cells with at least 30 scored items; more items are being added.. Hover a category for its definition and item count.

H2O-Lightning-31B v1.3.4: subject topics: not labelled for this draw yet (top-up pending)

What each category means · items per category
  • Rules, policy & law — applying written rules: company policies, contracts, regulations, eligibility. 1616 items (151 open / 1465 sealed)
  • Coding & software — code, SQL, repositories, developer tools and IT systems. 382 items (62 open / 320 sealed)
  • Math & numbers — a calculation decides the answer: arithmetic, word problems, probability, dates, units. 370 items (37 open / 333 sealed)
  • Finance & commerce — money: payments, refunds, invoices, orders, expenses, insurance payouts. 223 items (12 open / 211 sealed)
  • Support & operations — support tickets, incidents, logistics, scheduling desks and routing work to a team. 160 items (21 open / 139 sealed)
  • Everyday language — short everyday messages: intents, assistant requests, reading a detail out of a text. 145 items (11 open / 134 sealed)
  • Safety & security — untrusted or injected instructions, fraud, moderation, access and security triage. 109 items (6 open / 103 sealed)

Use cases (TypeSafe categories)

Radar: use cases (typesafe categories), two systemsUse cases (TypeSafe categories), H2O-Lightning-31B v1.3.4 vs Jev 1.13.0. Model routing: 90.3 vs 73.0; Legal & compliance: 93.4 vs 68.6; Customer support: 93.0 vs 49.6; Other: 62.2 vs 40.1; E-commerce: 86.0 vs 40.5; Insurance claims: 93.1 vs 46.1; Risk assessment: 83.6 vs 32.8; Financial crime: 87.0 vs 35.8; Feature extraction: 84.9 vs 28.3; Lead generation: 95.7 vs 49.2; Recruiting: 91.3 vs 0.0; Knowledge graphs: 94.2 vs 28.8; LLM guardrails: 98.8 vs 68.0; Moderation: 93.3 vs 51.6; Code linting: 94.1 vs 29.1; Search & retrieval: 100.0 vs 80.9; Science: 96.5 vs 21.2; Advertising: 95.1 vs 16.4; Gaming: 84.6 vs 5.1; Demand forecasting: 60.8 vs 0.0. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Model routing: H2O-Lightning-31B v1.3.4: 90.3; Jev 1.13.0: 73.0 — Model routing: deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 524 items (86 open, 438 sealed).Modelrouting90.3 · 73.0Legal & compliance: H2O-Lightning-31B v1.3.4: 93.4; Jev 1.13.0: 68.6 — Legal and compliance: contracts, policies, regulations, eligibility rules, checking documents against written requirements. 515 items (74 open, 441 sealed).Legal &compliance93.4 · 68.6Customer support: H2O-Lightning-31B v1.3.4: 93.0; Jev 1.13.0: 49.6 — Customer support: classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 240 items (37 open, 203 sealed).Customersupport93.0 · 49.6Other: H2O-Lightning-31B v1.3.4: 62.2; Jev 1.13.0: 40.1 — Other: none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 174 items (24 open, 150 sealed).Other62.2 · 40.1E-commerce: H2O-Lightning-31B v1.3.4: 86.0; Jev 1.13.0: 40.5 — E-commerce marketplaces: product listings, product attributes, orders and deliveries, seller catalog policy. 119 items (9 open, 110 sealed).E-commerce86.0 · 40.5Insurance claims: H2O-Lightning-31B v1.3.4: 93.1; Jev 1.13.0: 46.1 — Insurance claims: first-notice-of-loss reports, claim complexity, missing information, claim triage. 119 items (9 open, 110 sealed).Insuranceclaims93.1 · 46.1Risk assessment: H2O-Lightning-31B v1.3.4: 83.6; Jev 1.13.0: 32.8 — Risk assessment: estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 118 items (8 open, 110 sealed).Riskassessment83.6 · 32.8Financial crime: H2O-Lightning-31B v1.3.4: 87.0; Jev 1.13.0: 35.8 — Financial crime: suspicious transactions, KYC, fraud alerts, entity matching for investigations. 101 items (9 open, 92 sealed).Financialcrime87.0 · 35.8Feature extraction: H2O-Lightning-31B v1.3.4: 84.9; Jev 1.13.0: 28.3 — Feature extraction for predictive modeling: turning natural-language data into probabilistic features or estimates for a downstream prediction. 99 items (5 open, 94 sealed).Featureextraction84.9 · 28.3Lead generation: H2O-Lightning-31B v1.3.4: 95.7; Jev 1.13.0: 49.2 — Lead generation: matching companies or inbound messages to an ideal customer profile, buyer intent, prioritising and routing sales leads. 95 items (4 open, 91 sealed).Leadgeneration95.7 · 49.2Recruiting: H2O-Lightning-31B v1.3.4: 91.3; Jev 1.13.0: 0.0 — Recruiting: resumes, applications, interview feedback, matching candidates to roles. 94 items (1 open, 93 sealed).Recruiting91.3 · 0.0Knowledge graphs: H2O-Lightning-31B v1.3.4: 94.2; Jev 1.13.0: 28.8 — Graphs and knowledge graphs: entity types, relationships between records, contradictions between facts, multi-step lookups across linked facts. 94 items (3 open, 91 sealed).Knowledgegraphs94.2 · 28.8LLM guardrails: H2O-Lightning-31B v1.3.4: 98.8; Jev 1.13.0: 68.0 — LLM guardrails: checking LLM inputs, outputs and tool calls: prompt injection, jailbreaks, policy violations, tool-call errors, answer quality. 93 items (5 open, 88 sealed).LLMguardrails98.8 · 68.0Moderation: H2O-Lightning-31B v1.3.4: 93.3; Jev 1.13.0: 51.6 — Moderation and trust and safety: user content moderation: toxicity, harassment, spam, unsafe advice, personal data exposure. 93 items (7 open, 86 sealed).Moderation93.3 · 51.6Code linting: H2O-Lightning-31B v1.3.4: 94.1; Jev 1.13.0: 29.1 — Semantic code linting: checking code or writing against conventions and guidelines, as in CI review of code. 90 items (3 open, 87 sealed).Code linting94.1 · 29.1Search & retrieval: H2O-Lightning-31B v1.3.4: 100.0; Jev 1.13.0: 80.9 — Search and retrieval: scoring query-to-candidate relevance, reranking results, selecting useful context for RAG. 90 items (5 open, 85 sealed).Search &retrieval100.0 · 80.9Science: H2O-Lightning-31B v1.3.4: 96.5; Jev 1.13.0: 21.2 — Scientific discovery: screening papers, labelling research passages or survey answers, checking that citations support claims, methodology checks. 90 items (2 open, 88 sealed).Science96.5 · 21.2Advertising: H2O-Lightning-31B v1.3.4: 95.1; Jev 1.13.0: 16.4 — Advertising: ad creatives, campaign copy, brand safety, prohibited claims in ads. 88 items (3 open, 85 sealed).Advertising95.1 · 16.4Gaming: H2O-Lightning-31B v1.3.4: 84.6; Jev 1.13.0: 5.1 — Gaming: player reports, in-game chat, game support. 86 items (3 open, 83 sealed).Gaming84.6 · 5.1Demand forecasting: H2O-Lightning-31B v1.3.4: 60.8; Jev 1.13.0: 0.0 — Demand forecasting: purchase intent, product interest, demand and supply signals for forecasting. 83 items (3 open, 80 sealed).Demandforecasting60.8 · 0.0

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Measured application-category competence on this release’s item set. Chance-corrected competence per category (0 = at or below chance; negative averages are reported as 0, 100 = perfect), H2O-Lightning-31B v1.3.4: open+sealed; Jev 1.13.0: S+P+L1+L2+L3 items pooled; Breakdown values are shown for cells with at least 30 scored items; more items are being added.. Hover a category for its definition and item count.A point on the bold 0 ring printed 0.0 is a measured value, not a gap: these published cells clip below-chance results to 0, so 0 means at or below chance. Their score markers cannot fall inside that ring.
What each category means · items per category
  • Model routing — deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 524 items (86 open / 438 sealed)
  • Legal and compliance — contracts, policies, regulations, eligibility rules, checking documents against written requirements. 515 items (74 open / 441 sealed)
  • Customer support — classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 240 items (37 open / 203 sealed)
  • Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 174 items (24 open / 150 sealed)
  • E-commerce marketplaces — product listings, product attributes, orders and deliveries, seller catalog policy. 119 items (9 open / 110 sealed)
  • Insurance claims — first-notice-of-loss reports, claim complexity, missing information, claim triage. 119 items (9 open / 110 sealed)
  • Risk assessment — estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 118 items (8 open / 110 sealed)
  • Financial crime — suspicious transactions, KYC, fraud alerts, entity matching for investigations. 101 items (9 open / 92 sealed)
  • Feature extraction for predictive modeling — turning natural-language data into probabilistic features or estimates for a downstream prediction. 99 items (5 open / 94 sealed)
  • Lead generation — matching companies or inbound messages to an ideal customer profile, buyer intent, prioritising and routing sales leads. 95 items (4 open / 91 sealed)
  • Recruiting — resumes, applications, interview feedback, matching candidates to roles. 94 items (1 open / 93 sealed)
  • Graphs and knowledge graphs — entity types, relationships between records, contradictions between facts, multi-step lookups across linked facts. 94 items (3 open / 91 sealed)
  • LLM guardrails — checking LLM inputs, outputs and tool calls: prompt injection, jailbreaks, policy violations, tool-call errors, answer quality. 93 items (5 open / 88 sealed)
  • Moderation and trust and safety — user content moderation: toxicity, harassment, spam, unsafe advice, personal data exposure. 93 items (7 open / 86 sealed)
  • Semantic code linting — checking code or writing against conventions and guidelines, as in CI review of code. 90 items (3 open / 87 sealed)
  • Search and retrieval — scoring query-to-candidate relevance, reranking results, selecting useful context for RAG. 90 items (5 open / 85 sealed)
  • Scientific discovery — screening papers, labelling research passages or survey answers, checking that citations support claims, methodology checks. 90 items (2 open / 88 sealed)
  • Advertising — ad creatives, campaign copy, brand safety, prohibited claims in ads. 88 items (3 open / 85 sealed)
  • Gaming — player reports, in-game chat, game support. 86 items (3 open / 83 sealed)
  • Demand forecasting — purchase intent, product interest, demand and supply signals for forecasting. 83 items (3 open / 80 sealed)

Competence per request type, open / sealed

H2O-Lightning-31B v1.3.4 has no published values for this view.

Radar: competence per request type, open / sealed, two systemsCompetence per request type, open / sealed, H2O-Lightning-31B v1.3.4 vs Jev 1.13.0. Choice · open: — vs 79.0; Choice · sealed: — vs 75.8; Noul · open: — vs 54.5; Noul · sealed: — vs 46.1; Score · open: — vs 63.3; Score · sealed: — vs 63.0. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Choice · open: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 79.0Choice · openn/a · 79.0Choice · sealed: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 75.8Choice ·sealedn/a · 75.8Noul · open: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 54.5Noul · openn/a · 54.5Noul · sealed: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 46.1Noul · sealedn/a · 46.1Score · open: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 63.3Score · openn/a · 63.3Score · sealed: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 63.0Score ·sealedn/a · 63.0

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Chance-corrected competence for Choice, Noul and Score on the 300 open and 1200 sealed decisions.

Competence per tier — open set

H2O-Lightning-31B v1.3.4 has no published values for this view.

Radar: competence per tier — open set, two systemsCompetence per tier — open set, H2O-Lightning-31B v1.3.4 vs Jev 1.13.0. Easy: — vs 93.9; Standard: — vs 63.2; Judge: — vs 67.7; Hard: — vs 65.6. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Easy: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 93.9Easyn/a · 93.9Standard: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 63.2Standardn/a · 63.2Judge: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 67.7Judgen/a · 67.7Hard: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 65.6Hardn/a · 65.6

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Per-tier competence, the three request types pooled by their published decision counts.

Competence per tier — sealed set

H2O-Lightning-31B v1.3.4 has no published values for this view.

Radar: competence per tier — sealed set, two systemsCompetence per tier — sealed set, H2O-Lightning-31B v1.3.4 vs Jev 1.13.0. Easy: — vs 78.4; Standard: — vs 68.7; Judge: — vs 66.9; Hard: — vs 58.9. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Easy: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 78.4Easyn/a · 78.4Standard: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 68.7Standardn/a · 68.7Judge: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 66.9Judgen/a · 66.9Hard: H2O-Lightning-31B v1.3.4: No measured result; Jev 1.13.0: 58.9Hardn/a · 58.9

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Per-tier competence on the sealed decisions, types pooled the same way; item text stays private.
How the categories were made

O1S chance-corrected competence: equal mean across the request types present, clipped to 0–100. Zero means the equal-weight mean across present request types was at or below baseline; negative means are clipped. It does not mean every type scored zero. Choice uses a random-option baseline; Noul uses a fixed 50% accuracy baseline (abstentions count as wrong); Score uses the midpoint-guess error (random-guess fallback when every gold is at the midpoint). Compare models within a category; row pools and supported types can differ.

Categories use ruled subject-topic labels and authoring use-case precedence across the answered pool union, with stable item identities counted once.

  • Official O1S; recorded failures count where supported; no missing response is imputed; stable identity dedupe parent first.
  • S-based category cells pool recorded supported responses from S+P and the answered L1/L2/L3 supplements. API overlay category cells keep their separately recorded A4/A5+P union. Headline and per-type/tier scores are unchanged.
  • Breakdown values are shown for cells with at least 30 scored items; more items are being added.
All values as a table
SpokeA: H2O-Lightning-31B v1.3.4B: Jev 1.13.0
The four score axes
Intelligence85.063.6
Calibration90.790.6
Speed90.391.2
Cost60.254.7
Capability by subject topic
Rules, policy & lawsubject topics: not labelled for this draw yet (top-up pending)49.4
Coding & softwaresubject topics: not labelled for this draw yet (top-up pending)64.3
Math & numberssubject topics: not labelled for this draw yet (top-up pending)10.4
Finance & commercesubject topics: not labelled for this draw yet (top-up pending)40.9
Support & operationssubject topics: not labelled for this draw yet (top-up pending)48.8
Everyday languagesubject topics: not labelled for this draw yet (top-up pending)77.5
Safety & securitysubject topics: not labelled for this draw yet (top-up pending)52.2
Use cases (TypeSafe categories)
Model routing90.373.0
Legal & compliance93.468.6
Customer support93.049.6
Other62.240.1
E-commerce86.040.5
Insurance claims93.146.1
Risk assessment83.632.8
Financial crime87.035.8
Feature extraction84.928.3
Lead generation95.749.2
Recruiting91.30.0
Knowledge graphs94.228.8
LLM guardrails98.868.0
Moderation93.351.6
Code linting94.129.1
Search & retrieval100.080.9
Science96.521.2
Advertising95.116.4
Gaming84.65.1
Demand forecasting60.80.0
Competence per request type, open / sealed
Choice · open—79.0
Choice · sealed—75.8
Noul · open—54.5
Noul · sealed—46.1
Score · open—63.3
Score · sealed—63.0
Competence per tier — open set
Easy—93.9
Standard—63.2
Judge—67.7
Hard—65.6
Competence per tier — sealed set
Easy—78.4
Standard—68.7
Judge—66.9
Hard—58.9

Published values

MeasureJev 1.13.0 (TypeSafe AI)H2O-Lightning-31B v1.3.4 (@c35ed950)
Capability Score77.187.9
Rank (open-weights board)Reference, not ranked#3
Composite (secondary)71.479.3
intelligence63.685.0
calibration90.690.7
speed91.290.3
cost54.760.2
USD per 1,000 decisions$0.032 per 1,000 decisions$0.021 per 1,000 decisions
p50 latency (seconds)0.20.052 measured on-GPU, no network · 0.25 est. incl. network (2x + 150 ms)
Methodv1.7 · 10 s rulev1.7 · 10 s rule
p95 (measured) · decisions over 10 sp95 0.311 s (measured) · 0 of 1,500 over 10 sp95 0.107 s (measured) · —
Before the 10 s rule (v1.6)v1.6.43: A 71.49 · #18—
Last measured2026-10-052026-11-16

* Self-hosted rows: the first value is the p50 we measured on our own hardware over loopback (on-GPU, or on-CPU where marked, no network). It is not directly comparable with API latency, which we time end to end from our measuring host, network included. “est.” is the est. incl. network (2x + 150 ms) that the ranking and the latency cap use for every self-hosted row. Model authors can ask us to time a hosted endpoint instead (on request).

Jev 1.13.0 (TypeSafe AI)

proprietary API

Published source
Measured on
the operator's hosted API
Cost basis (public list price)
TypeSafe list tariff USD 0.042 per 1M input tokens, output free; measured on the v1.6.1 common cost basis (all items except those outside the Jev reference input range) from this run's token usage
Latency adjustment
none (API)

H2O-Lightning-31B v1.3.4 (@c35ed950)

H2O Model Non-Commercial Licence v1.1 (weights; gated on Hugging Face; benchmarking and publishing results are allowed by the licence). Base Gemma 4 under its own licence.

Published source
Measured on
Evaluator-owned Verda GPU VM (RTX PRO 6000), offline container (--network none; lo only), author's serving stack at model commit c35ed9505f6744770dc0d74bd3f468791a7984af. Unmodified serve.sh --profile two-way, one vLLM 0.30.0 engine with h2oai/vllm overlay a36449a18b4465b4f70fa078f5403032de414de8. Online FP8 W8A8 from BF16 weights, TRITON_ATTN; max model length 131072, state budget 126976, two-way window 8192, selector v2 with automatic one-way fallback for the record/window or summed prompt threshold 4096. One read, readout_orders=1; AR and router off. PyTorch 2.13.0, transformers 5.20.0, torchcodec 0.17.0. Each scored stage is one fresh compile read once (no best-of); determinism is assessed only within the public preparation compile. Standard self-hosted x2 + 0.15 s latency adjustment.
Cost basis (estimated)
GPU-time estimate from v1.8.0: one monthly-rented RTX PRO 6000 96 GB, USD 1.5235/hour; average load 70%. Architecture: dense gemma, 31B active parameters; throughput 10106.516129 tokens/s (architecture-class calibration, efficient-engine prefill). GPU-time cost USD 0.0212260058867847/1,000; old list-price estimate USD 0.0321852518468771/1,000 (google/gemma-4-31b-it at USD 0.09/M input and USD 0.34/M output). Applied min(old list-price estimate, GPU-time cost) = USD 0.0212260058867847/1,000. Estimated, not a bill.
Latency adjustment
x2 + 0.15 s (assumption, not measured)

Using these models through an API

Decision Models offers a Jev-compatible API; its comparison page lists hosted options and prices, checked on the date shown there.

Jev-compatible APIs compared (Decision Models)

Decision Models is run by the same company as Benchmark Heaven. This link is the same on every comparison page and never affects scores or ranks.

Frequently asked questions

What does JevBench show for Jev and H2O-Lightning-31B v1.3.4?
Jev 1.13.0 has Capability 77.1 and is the unranked reference (the open-weights board ranks 251 open-weights systems; hosted API offerings are ranked on the API leaderboard). H2O-Lightning-31B v1.3.4 (@c35ed950) has Capability 87.9. Rank #3 on the open-weights board in v1.8.18. Historical scores use their original method and are not ranked against the current release.
Is there a time limit per decision?
Yes, from JevBench v1.7.0 (method v1.7): a decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item. Each row shows its p95 and how many decisions were over the limit; the v1.6 values stay visible.
How does Capability differ from Composite?
Capability is the mean of Intelligence and Calibration; headline ranks require both official cost and median-latency caps. Composite additionally scores Speed and Cost and is secondary.
Can I compare the prices as actual bills?
Jev 1.13.0 (TypeSafe AI): public list price, $0.032 per 1,000 decisions. H2O-Lightning-31B v1.3.4 (@c35ed950): estimated, $0.021 per 1,000 decisions. Estimates and self-reported vendor prices are not measured charges.
Is H2O-Lightning-31B v1.3.4 open source?
The published licence note for H2O-Lightning-31B v1.3.4 (@c35ed950) is H2O Model Non-Commercial Licence v1.1 (weights; gated on Hugging Face; benchmarking and publishing results are allowed by the licence). Base Gemma 4 under its own licence.. Jev 1.13.0 is listed as proprietary API. Check the linked sources for terms.