Gemini 3.1 Pro Preview

AA-AnalystAgent published 2026-09-10 → · Published board — Tests analyst tasks using agentic Python execution across fourteen domains.

APEX-Agents-AA published 2026-09-10 → · Published board — Tests professional-service tasks that require agents to produce locally graded deliverables.

AA-Briefcase published 2026-09-10 → · Published board — Tests multi-week professional knowledge-work projects with linked tasks and large source collections.

GDPval-AA v2 v2 → · Published board — Tests professional knowledge-work deliverables across occupations using AA's Stirrup harness.

Harvey LAB-AA published 2026-09-10 → · Published board — Tests legal-work deliverables across practice areas on Harvey's private task set. Artificial Analysis' run, graded by one LLM judge against task rubrics — not the same run or scale as Vals AI's HLAB row.

ITBench-AA published 2026-09-10 → · Published board — Tests root-cause diagnosis from offline Kubernetes incident snapshots.

Terminal-Bench Hard (AA) pinned revision 74221fb → · Published board — Tests a pinned 44-task hard subset of terminal-based work using Terminus 2.

Terminal-Bench v2.1 (AA) v2.1 → · Published board — Tests terminal-based work on the 89-task verified refresh using Terminus 2.

Terminal-Bench v4.0 (AA) v4.0 → · Published board — Tests terminal-based work on the 66-task release using mini-SWE-agent v2.4.6.

EBR-bench (Earthborne Rangers, Epoch AI) published 2026-09-18 → · Published board — Learning-capability test: models repeatedly play the obscure campaign board game Earthborne Rangers with note-taking, and the score measures whether results improve across playthroughs.

τ²-Bench Airline (OpenRouter run) published 2026-09-15 → · Published board — Multi-turn service agents making tool calls under strict policy constraints. (OpenRouter's own reproducible run, as published on openrouter.ai/benchmarks.)

AA Agentic Index published 2026-09-18 → · Published board — Artificial Analysis publishes one Agentic Index per model, measured on its primary configuration and relayed by OpenRouter's Benchmarks API; the value is attached at family scope on the deterministic representative. Artificial Analysis publishes this board without a version number; we keep the date each result was retained.

SciCode (AA subproblems) v1.0.1 → · Published board — Tests scientific Python programming with scientist-annotated background information.

React Native Evals 91-evals → · Published board — Coding agents build React Native app features — navigation, animation, async state, lists, native APIs and Skia graphics — and an LLM judge checks the generated code against each task's written requirements.

AA Coding Index v4.x · frozen 2026-10-27 → · Published board — Artificial Analysis withdrew the Coding Index with Intelligence Index v5.0 (2026-10-28). These are the last v4.x values, frozen with their own date (2026-10-27).

DesignArena Web Apps (agentic) published 2026-11-03 → · Published board — Published Elo. Exact source attachment is retained; fewer than 200 battles excludes a row from radar normalization and anomaly peers.

DesignArena Full-Stack published 2026-11-03 → · Published board — Published Elo. Exact source attachment is retained; fewer than 200 battles excludes a row from radar normalization and anomaly peers.

GPQA Diamond (OpenRouter run) — measured cost per task published 2026-09-15 → · Published board — Mean USD OpenRouter actually spent per task while running GPQA Diamond for this configuration.

τ²-Bench Airline (OpenRouter run) — measured cost per task published 2026-09-15 → · Published board — Mean USD OpenRouter actually spent per task while running τ²-Bench Airline for this configuration.

IFBench (AA single-turn) published 2026-09-10 → · Published board — Tests precise single-turn instructions with deterministic rule checks.

Humanity's Last Exam (AA text-only) published 2026-09-10 → · Published board — Tests expert-level knowledge on AA's text-only Humanity's Last Exam subset.

AA-Omniscience Index published 2026-09-10 → · Published board — Tests factual reliability while rewarding correct answers and penalizing hallucinations.

GDP.pdf (AA) published 2026-09-10 → · Published board — Tests professional reasoning over long PDFs with AA document preparation and grading.

AA-LCR v1.1 v1.1 → · Published board — Tests reasoning across multiple long documents with corrected answer keys and grading.

MLCR-AA published 2026-09-10 → · Published board — Tests medical-record synthesis and reasoning across long, fragmented case documents.

FrontierMath Tier 4 v2 (Epoch AI) v2 → · Published board — The hardest, research-level tier of Epoch AI's unpublished FrontierMath problems, run by Epoch AI on its private v2 set.

FrontierMath Tiers 1–3 v2 (Epoch AI) v2 → · Published board — Unpublished, expert-written mathematics problems from undergraduate to research level with automatically checkable answers, run by Epoch AI on its private v2 set.

ArXivMath 06/2026 (MathArena) 2026-06 → · Published board — Research-level math problems with a checkable final answer, drawn from arXiv papers submitted in June 2026; each edition is restricted to papers published within the last month to reduce training-data contamination.

BrokenArXiv 06/2026 (MathArena) 2026-06 → · Published board — Plausible but false proof statements taken from June 2026 arXiv papers; a model scores by refusing to prove them and saying the statement is false as written.

HMMT February 2026 (MathArena) 2026-02 → · Published board — The 33 final-answer problems of the February 2026 Harvard-MIT Mathematics Tournament, one of the largest high-school math competitions in the United States, run by MathArena on the contest problems.

Apex 2025 (MathArena) v2025 → · Published board — Twelve final-answer problems from 2025 competitions that MathArena selected because strong models of the time failed them; the selection is biased against the four models used to choose it (Grok 4, GPT-5, Gemini 2.5 Pro, GLM 4.5).

Apex Shortlist 2025 (MathArena) v2025 → · Published board — 47 final-answer problems from 2025 competitions on which at least one of two reference models (Grok 4 Fast, GPT-5 mini) got at least one of four attempts wrong, run by MathArena.

AIME 2026 (MathArena) v2026 → · Published board — The 30 integer-answer problems of the 2026 American Invitational Mathematics Examination (AIME I and II), the qualifying exam for the USA Mathematical Olympiad, run by MathArena on the contest problems.

USAMO 2026 (MathArena) v2026 → · Published board — The six proof problems of the 2026 USA Mathematical Olympiad, the final round of the American Mathematics Competitions, with each model's proofs graded by MathArena's LLM judges.

Chess Puzzles (Epoch AI) published 2026-09-18 → · Published board — Best-move selection on 100 novel chess positions generated programmatically by Epoch AI, each with a single Stockfish-verified best move; probes spatial reasoning and planning.

SimpleBench published 2026-09-10 → · Published board — A multiple-choice text benchmark of over 200 questions covering spatio-temporal reasoning, social intelligence, and linguistic adversarial robustness, on which a non-specialized human baseline outperforms every tested LLM.

SimpleBench published 2026-09-26 → · Published board — A multiple-choice text benchmark of over 200 questions covering spatio-temporal reasoning, social intelligence, and linguistic adversarial robustness, on which a non-specialized human baseline outperforms every tested LLM.

CritPt (AA) published 2026-09-10 → · Published board — Tests research-level physics reasoning with Python, symbolic and numerical answers.

GPQA Diamond (AA) published 2026-09-10 → · Published board — Tests graduate-level biology, physics and chemistry knowledge on the Diamond subset.

GPQA Diamond (Epoch AI run) published 2026-09-18 → · Published board — Epoch AI's own inspect-ai runs of GPQA Diamond, the 198-question graduate-level science multiple-choice set.

GPQA Diamond (OpenRouter run) published 2026-09-15 → · Published board — Graduate-level science questions that resist retrieval and reward careful reasoning. (OpenRouter's own reproducible run, as published on openrouter.ai/benchmarks.)

AutomationBench-AA v1.0.6 → · Published board — Tests multi-app SaaS workflows through REST tools on a held-out AutomationBench split.

EnterpriseOps-Gym-AA published 2026-09-10 → · Published board — Tests enterprise workflows through MCP tools against resettable application environments.

τ²-Bench Telecom (AA) published 2026-09-10 → · Published board — Tests dual-control telecom agents that coordinate tool use with a simulated user.

τ³-Banking (AA) v1.0.1 → · Published board — Tests banking support agents that retrieve policies and change account state through tools.

MMMU Pro (AA) published 2026-09-10 → · Published board — Tests multimodal understanding using challenging ten-option questions.

EQ-Bench Creative Writing v3 v3 → · Published board — A LLM-judged creative writing benchmark with vocabulary and GPT-slop controls.

EQ-Bench Longform Creative Writing v1.11 → · Published board — LLM-judged benchmark in which models plan and write a short story/novella over 8x 1000-word turns, scored 0-100 across 14 rubric dimensions.

LLM Creative Story-Writing Benchmark published 2026-09-10 → · Published board — Pairwise comparison of short stories written to the same constrained creative briefs, with LLM evaluator choices combined into a relative comparison score.