What does JevBench show for Jev and MetaDecision-31B-Rain?

Jev 1.13.0 has Capability 77.1 and is the unranked reference (the open-weights board ranks 243 open-weights systems; hosted API offerings are ranked on the API leaderboard). MetaDecision-31B-Rain has Capability 88.3. Rank #2 on the open-weights board in v1.8.11. Historical scores use their original method and are not ranked against the current release.

Is there a time limit per decision?

Yes, from JevBench v1.7.0 (method v1.7): a decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item. Each row shows its p95 and how many decisions were over the limit; the v1.6 values stay visible.

How does Capability differ from Composite?

Capability is the mean of Intelligence and Calibration; headline ranks require both official cost and median-latency caps. Composite additionally scores Speed and Cost and is secondary.

Can I compare the prices as actual bills?

Jev 1.13.0 (TypeSafe AI): public list price, $0.032 per 1,000 decisions. MetaDecision-31B-Rain: estimated, $0.032 per 1,000 decisions. Estimates and self-reported vendor prices are not measured charges.

Jev vs MetaDecision-31B-Rain: published benchmark comparison

Data: JevBench v1.8.11, published 2026-11-15 · method v1.7 (10 s rule), re-scored from the recorded per-decision times

Rank #2 on the open-weights board in v1.8.11.

Compare Jev-compatible models by separate measures and their measurement conditions. A historical measurement is not a current rank.

Jev and MetaDecision-31B-Rain: published axes and category radars

Pick any two. Radars for the score axes, capability by subject topic and use cases, chance-corrected competence per request type on the open and sealed sets, and competence per tier on each set. Further out is better on every spoke; the link keeps the pair.

System A

System B

A: MetaDecision-31B-Rain — Open weights · LLM decoder · Score 73.0 (#2)
B: Jev 1.13.0REFERENCE · TypeSafe — Jev — reference (TypeSafe, closed) · Score 71.4 (ranked, not ranked)

The four score axes

0–100, the values in the table. An axis a system has no published value for is left as a gap (it counts as 0 in the composite).

Capability by subject topic

MetaDecision-31B-Rain: No per-category values are published for this system.

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

What the decision is about — math and dates, coding, rules, money, support work, everyday messages, safety. Chance-corrected competence per category (0 = at or below chance; negative averages are reported as 0, 100 = perfect), MetaDecision-31B-Rain: open+sealed; Jev 1.13.0: S+P+L1+L2+L3 items pooled; every category with at least 15 items is a spoke; points under 30 answered items are hollow and marked n=…. Hover a category for its definition and item count.

MetaDecision-31B-Rain: No per-category values are published for this system.

What each category means · items per category

Rules, policy & law — applying written rules: company policies, contracts, regulations, eligibility. 1616 items (151 open / 1465 sealed)
Coding & software — code, SQL, repositories, developer tools and IT systems. 382 items (62 open / 320 sealed)
Math & numbers — a calculation decides the answer: arithmetic, word problems, probability, dates, units. 370 items (37 open / 333 sealed)
Finance & commerce — money: payments, refunds, invoices, orders, expenses, insurance payouts. 223 items (12 open / 211 sealed)
Support & operations — support tickets, incidents, logistics, scheduling desks and routing work to a team. 160 items (21 open / 139 sealed)
Everyday language — short everyday messages: intents, assistant requests, reading a detail out of a text. 145 items (11 open / 134 sealed)
Safety & security — untrusted or injected instructions, fraud, moderation, access and security triage. 109 items (6 open / 103 sealed)

Use cases (TypeSafe categories)

MetaDecision-31B-Rain: No per-category values are published for this system.

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

The real-world application area of each decision, after the TypeSafe use-case map; an item can count in two. Chance-corrected competence per category (0 = at or below chance; negative averages are reported as 0, 100 = perfect), MetaDecision-31B-Rain: open+sealed; Jev 1.13.0: S+P+L1+L2+L3 items pooled; every category with at least 15 items is a spoke; points under 30 answered items are hollow and marked n=…. Hover a category for its definition and item count.A point on the bold 0 ring printed 0.0 is a measured value, not a gap: these published cells clip below-chance results to 0, so 0 means at or below chance. Their score markers cannot fall inside that ring.

MetaDecision-31B-Rain: No per-category values are published for this system.

What each category means · items per category

Model routing — deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 524 items (86 open / 438 sealed)
Legal and compliance — contracts, policies, regulations, eligibility rules, checking documents against written requirements. 515 items (74 open / 441 sealed)
Customer support — classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 240 items (37 open / 203 sealed)
Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 174 items (24 open / 150 sealed)
E-commerce marketplaces — product listings, product attributes, orders and deliveries, seller catalog policy. 119 items (9 open / 110 sealed)
Insurance claims — first-notice-of-loss reports, claim complexity, missing information, claim triage. 119 items (9 open / 110 sealed)
Risk assessment — estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 118 items (8 open / 110 sealed)
Financial crime — suspicious transactions, KYC, fraud alerts, entity matching for investigations. 101 items (9 open / 92 sealed)
Feature extraction for predictive modeling — turning natural-language data into probabilistic features or estimates for a downstream prediction. 99 items (5 open / 94 sealed)
Lead generation — matching companies or inbound messages to an ideal customer profile, buyer intent, prioritising and routing sales leads. 95 items (4 open / 91 sealed)
Recruiting — resumes, applications, interview feedback, matching candidates to roles. 94 items (1 open / 93 sealed)
Graphs and knowledge graphs — entity types, relationships between records, contradictions between facts, multi-step lookups across linked facts. 94 items (3 open / 91 sealed)
LLM guardrails — checking LLM inputs, outputs and tool calls: prompt injection, jailbreaks, policy violations, tool-call errors, answer quality. 93 items (5 open / 88 sealed)
Moderation and trust and safety — user content moderation: toxicity, harassment, spam, unsafe advice, personal data exposure. 93 items (7 open / 86 sealed)
Semantic code linting — checking code or writing against conventions and guidelines, as in CI review of code. 90 items (3 open / 87 sealed)
Search and retrieval — scoring query-to-candidate relevance, reranking results, selecting useful context for RAG. 90 items (5 open / 85 sealed)
Scientific discovery — screening papers, labelling research passages or survey answers, checking that citations support claims, methodology checks. 90 items (2 open / 88 sealed)
Advertising — ad creatives, campaign copy, brand safety, prohibited claims in ads. 88 items (3 open / 85 sealed)
Gaming — player reports, in-game chat, game support. 86 items (3 open / 83 sealed)
Demand forecasting — purchase intent, product interest, demand and supply signals for forecasting. 83 items (3 open / 80 sealed)

Competence per request type, open / sealed

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Chance-corrected competence for Choice, Noul and Score on the 300 open and 1200 sealed decisions.

Competence per tier — open set

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Per-tier competence, the three request types pooled by their published decision counts.

Competence per tier — sealed set

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Per-tier competence on the sealed decisions, types pooled the same way; item text stays private.

How the categories were made

O1S chance-corrected competence: equal mean across the request types present, clipped to 0–100. Zero means the equal-weight mean across present request types was at or below baseline; negative means are clipped. It does not mean every type scored zero. Choice uses a random-option baseline; Noul uses a fixed 50% accuracy baseline (abstentions count as wrong); Score uses the midpoint-guess error (random-guess fallback when every gold is at the midpoint). Compare models within a category; row pools and supported types can differ.

Categories use ruled subject-topic labels and authoring use-case precedence across the answered pool union, with stable item identities counted once.

Official O1S; recorded failures count where supported; no missing response is imputed; stable identity dedupe parent first.
S-based category cells pool recorded supported responses from S+P and the answered L1/L2/L3 supplements. API overlay category cells keep their separately recorded A4/A5+P union. Headline and per-type/tier scores are unchanged.

All values as a table

Spoke	A: MetaDecision-31B-Rain	B: Jev 1.13.0
The four score axes
Intelligence	85.0	63.6
Calibration	91.5	90.6
Speed	71.9	91.2
Cost	55.0	54.7
Capability by subject topic
Rules, policy & law	No per-category values are published for this system.	49.4
Coding & software	No per-category values are published for this system.	64.3
Math & numbers	No per-category values are published for this system.	10.4
Finance & commerce	No per-category values are published for this system.	40.9
Support & operations	No per-category values are published for this system.	48.8
Everyday language	No per-category values are published for this system.	77.5
Safety & security	No per-category values are published for this system.	52.2
Use cases (TypeSafe categories)
Model routing	No per-category values are published for this system.	73.0
Legal & compliance	No per-category values are published for this system.	68.6
Customer support	No per-category values are published for this system.	49.6
Other	No per-category values are published for this system.	40.1
E-commerce	No per-category values are published for this system.	40.5
Insurance claims	No per-category values are published for this system.	46.1
Risk assessment	No per-category values are published for this system.	32.8
Financial crime	No per-category values are published for this system.	35.8
Feature extraction	No per-category values are published for this system.	28.3
Lead generation	No per-category values are published for this system.	49.2
Recruiting	No per-category values are published for this system.	0.0
Knowledge graphs	No per-category values are published for this system.	28.8
LLM guardrails	No per-category values are published for this system.	68.0
Moderation	No per-category values are published for this system.	51.6
Code linting	No per-category values are published for this system.	29.1
Search & retrieval	No per-category values are published for this system.	80.9
Science	No per-category values are published for this system.	21.2
Advertising	No per-category values are published for this system.	16.4
Gaming	No per-category values are published for this system.	5.1
Demand forecasting	No per-category values are published for this system.	0.0
Competence per request type, open / sealed
Choice · open	87.4	79.0
Choice · sealed	95.4	75.8
Noul · open	84.8	54.5
Noul · sealed	94.0	46.1
Score · open	71.6	63.3
Score · sealed	94.4	63.0
Competence per tier — open set
Easy	97.1	93.9
Standard	73.8	63.2
Judge	84.4	67.7
Hard	80.6	65.6
Competence per tier — sealed set
Easy	97.8	78.4
Standard	95.8	68.7
Judge	100.0	66.9
Hard	89.7	58.9

Published values

Measure	Jev 1.13.0 (TypeSafe AI)	MetaDecision-31B-Rain
Capability Score	77.1	88.3
Rank (open-weights board)	Reference, not ranked	#2
Composite (secondary)	71.4	73.0
intelligence	63.6	85.0
calibration	90.6	91.5
speed	91.2	71.9
cost	54.7	55.0
USD per 1,000 decisions	$0.032 per 1,000 decisions	$0.032 per 1,000 decisions
p50 latency (seconds)	0.2	0.7
Method	v1.7 · 10 s rule	v1.7 · 10 s rule
p95 (measured) · decisions over 10 s	p95 0.311 s (measured) · 0 of 1,500 over 10 s	p95 4.398 s (measured) · 0 of 1,500 over 10 s
Before the 10 s rule (v1.6)	v1.6.43: A 71.49 · #18	—
Last measured	2026-10-05	2026-10-27

Jev 1.13.0 (TypeSafe AI)

proprietary API

Published source

Measured on: the operator's hosted API
Cost basis (public list price): TypeSafe list tariff USD 0.042 per 1M input tokens, output free; measured on the v1.6.1 common cost basis (all items except those outside the Jev reference input range) from this run's token usage
Latency adjustment: none (API)

MetaDecision-31B-Rain

Weights: Apache-2.0 per model card (base Gemma 4 under the Gemma 4 licence); service code (service/) MIT

Published source

Measured on: Measured on one Verda RTX PRO 6000 Blackwell Server Edition GPU, 96 GB, FIN-03, the author's own server with network access cut. The vLLM image was vllm/vllm-openai@sha256:a4a4c0437bf7240089da5f08aa370c4aee17ae5290f7a3b468825ee26c4c3a6b; scored elapsed time was 2,422 seconds. Serving code is MeTask-Rain/MetaDecision-31B-Rain@068cb1420d40b06e48895f2fc75f29e6da37748b; weights are MeTask-Rain/metask-jev-rain-31B-trained@2feff076f07711fe2fefe7a6fdcedfe4132b773e. The weights are the same as the live rows metask_jev_rain_31b_trained and metask_jev_rain_31b_trained_v3, bound by SHA-256 (model.safetensors 1b9b1d622a93f02c0d33f98e502f233b5d707443af6ddc464ed0bf5498506c20; adapter_model.safetensors 2b9568998ffed5d133223ffdff03056be297285c643122db4d45365d0bd86970); only service/ changed. The ordered repo is the serving-code repo. Engine-reported token usage includes drafting and readout passes: 753,547 input and 72,602 output tokens across 1,500 decisions; no usage-estimated rows. The initial-prompt count was 597,680 processed tokens versus 964,173 untrimmed, with 5 trimmed items. The service extracts query-relevant state chunks, protects explicit rule sections, and marks omissions; inputs above 6,653 tokens are answered from a shortened context, and a required hard cut retains 60% of the head and 40% of the tail. A 50-item, two-pass determinism check had 1/50 argmax flip. Drafting has a time-based 9-second per-decision deadline, so answers can depend on timing. The served base is Google's W4A16 4-bit QAT checkpoint (google/gemma-4-31B-it-qat-w4a16-ct) with the unmerged rank-16 LoRA adapter applied at serving time. The service keeps state across decisions: drafting passes are capped at 49% of all decisions so far, and from the 101st decision a cost fuse limits drafting when the running engine cost nears USD 0.0646 per 1,000 decisions, priced at USD 0.0598 per million tokens (this benchmark's GPU-time rate for the architecture). Answers and cost can therefore depend on the order of decisions.
Cost basis (estimated): GPU-time estimate from v1.8.0: one monthly-rented RTX PRO 6000 96 GB, EUR 999/month / 730 hours, ECB USD 1.1133/EUR dated 28 Oct 2026, USD 1.5235/hour; average load 70%. Architecture: dense gemma, 31B active parameters; throughput 10106.516129 tokens/s (architecture-class calibration, 29 Oct 2026, efficient-engine prefill; capped at 100,000 tokens/s). The row's own scored cost basis: 717,572 input + 72,041 output tokens over 1,489 decisions = 530.29751511 tokens/decision. Generated tokens are counted at the same throughput as prefill. GPU-time cost USD 0.0317219617003379/1,000; old list-price estimate USD 0.0598223102753526/1,000 (google/gemma-4-31b-it via frozen pricing_v15 reference at USD 0.09/M input and USD 0.34/M output). Applied min(old list-price estimate, GPU-time cost) = USD 0.0317219617003379/1,000. Estimated, not a bill. USD 0.0317219617 per 1,000 decisions on the frozen 1,489-item common set of this draw (USD 0.0329463697 over all 1,500 items).
Latency adjustment: Self-hosted latency estimate under the shared method (assumption, not measured)

Using these models through an API

Decision Models offers a Jev-compatible API; its comparison page lists hosted options and prices, checked on the date shown there.

Jev-compatible APIs compared (Decision Models)

Decision Models is run by the same company as Benchmark Heaven. This link is the same on every comparison page and never affects scores or ranks.

Frequently asked questions

What does JevBench show for Jev and MetaDecision-31B-Rain?: Jev 1.13.0 has Capability 77.1 and is the unranked reference (the open-weights board ranks 243 open-weights systems; hosted API offerings are ranked on the API leaderboard). MetaDecision-31B-Rain has Capability 88.3. Rank #2 on the open-weights board in v1.8.11. Historical scores use their original method and are not ranked against the current release.
Is there a time limit per decision?: Yes, from JevBench v1.7.0 (method v1.7): a decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item. Each row shows its p95 and how many decisions were over the limit; the v1.6 values stay visible.
How does Capability differ from Composite?: Capability is the mean of Intelligence and Calibration; headline ranks require both official cost and median-latency caps. Composite additionally scores Speed and Cost and is secondary.
Can I compare the prices as actual bills?: Jev 1.13.0 (TypeSafe AI): public list price, $0.032 per 1,000 decisions. MetaDecision-31B-Rain: estimated, $0.032 per 1,000 decisions. Estimates and self-reported vendor prices are not measured charges.
Is MetaDecision-31B-Rain open source?: The published licence note for MetaDecision-31B-Rain is Weights: Apache-2.0 per model card (base Gemma 4 under the Gemma 4 licence); service code (service/) MIT. Jev 1.13.0 is listed as proprietary API. Check the linked sources for terms.