Jev vs H2O-Lightning-4B v1.1: published benchmark comparison

• Rules, policy & law — applying written rules: company policies, contracts, regulations, eligibility. 1011 items (151 open / 860 sealed)
• Coding & software — code, SQL, repositories, developer tools and IT systems. 330 items (62 open / 268 sealed)
• Math & numbers — a calculation decides the answer: arithmetic, word problems, probability, dates, units. 234 items (37 open / 197 sealed)
• Support & operations — support tickets, incidents, logistics, scheduling desks and routing work to a team. 113 items (21 open / 92 sealed)
• Finance & commerce — money: payments, refunds, invoices, orders, expenses, insurance payouts. 99 items (12 open / 87 sealed)
• Everyday language — short everyday messages: intents, assistant requests, reading a detail out of a text. 52 items (11 open / 41 sealed)
• Safety & security — untrusted or injected instructions, fraud, moderation, access and security triage. 48 items (6 open / 42 sealed)

• Model routing — deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 471 items (86 open / 385 sealed)
• Legal and compliance — contracts, policies, regulations, eligibility rules, checking documents against written requirements. 466 items (74 open / 392 sealed)
• Customer support — classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 179 items (37 open / 142 sealed)
• E-commerce marketplaces — product listings, product attributes, orders and deliveries, seller catalog policy. 65 items (9 open / 56 sealed)
• Insurance claims — first-notice-of-loss reports, claim complexity, missing information, claim triage. 63 items (9 open / 54 sealed)
• Risk assessment — estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 62 items (8 open / 54 sealed)
• Financial crime — suspicious transactions, KYC, fraud alerts, entity matching for investigations. 51 items (9 open / 42 sealed)
• Moderation and trust and safety — user content moderation: toxicity, harassment, spam, unsafe advice, personal data exposure. 47 items (7 open / 40 sealed)
• Semantic code linting — checking code or writing against conventions and guidelines, as in CI review of code. 33 items (3 open / 30 sealed)
• Lead generation — matching companies or inbound messages to an ideal customer profile, buyer intent, prioritising and routing sales leads. 31 items (4 open / 27 sealed)
• Feature extraction for predictive modeling — turning natural-language data into probabilistic features or estimates for a downstream prediction. 31 items (5 open / 26 sealed)
• LLM guardrails — checking LLM inputs, outputs and tool calls: prompt injection, jailbreaks, policy violations, tool-call errors, answer quality. 30 items (5 open / 25 sealed)
• Search and retrieval — scoring query-to-candidate relevance, reranking results, selecting useful context for RAG. 30 items (5 open / 25 sealed)
• Advertising — ad creatives, campaign copy, brand safety, prohibited claims in ads. 30 items (3 open / 27 sealed)
• Demand forecasting — purchase intent, product interest, demand and supply signals for forecasting. 30 items (3 open / 27 sealed)
• Recruiting — resumes, applications, interview feedback, matching candidates to roles. 30 items (1 open / 29 sealed)
• Graphs and knowledge graphs — entity types, relationships between records, contradictions between facts, multi-step lookups across linked facts. 30 items (3 open / 27 sealed)
• Gaming — player reports, in-game chat, game support. 30 items (3 open / 27 sealed)
• Scientific discovery — screening papers, labelling research passages or survey answers, checking that citations support claims, methodology checks. 30 items (2 open / 28 sealed)

Not drawn: Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 148 items (24 open / 124 sealed) — not a use case of its own, so it is counted but not drawn.