Claude Fable 5 (Max, Opus 4.8 Fallback)

AA-AnalystAgent published 2026-09-10 → · Published board — Tests analyst tasks using agentic Python execution across fourteen domains.

AA-Briefcase published 2026-09-10 → · Published board — Tests multi-week professional knowledge-work projects with linked tasks and large source collections.

GDPval-AA v2 v2 → · Published board — Tests professional knowledge-work deliverables across occupations using AA's Stirrup harness.

Harvey LAB-AA published 2026-09-10 → · Published board — Tests legal-work deliverables across practice areas on Harvey's private task set. Artificial Analysis' run, graded by one LLM judge against task rubrics — not the same run or scale as Vals AI's HLAB row.

Terminal-Bench Hard (AA) pinned revision 74221fb → · Published board — Tests a pinned 44-task hard subset of terminal-based work using Terminus 2.

Terminal-Bench v2.1 (AA) v2.1 → · Published board — Tests terminal-based work on the 89-task verified refresh using Terminus 2.

Terminal-Bench v4.0 (AA) v4.0 → · Published board — Tests terminal-based work on the 66-task release using mini-SWE-agent v2.4.6.

ApprenticeBench API (NeoCognition) published 2026-09-14 → · Claude Code — An agent completes the same 100 sequential accounts-payable tasks in the Odoo ERP with diminishing mentoring, using dedicated application API calls instead of the screen interface.

ApprenticeBench CUA (NeoCognition) published 2026-09-14 → · Claude Code — A computer-use agent operates the Odoo ERP through its screens across 100 sequential accounts-payable tasks with diminishing mentoring.

EBR-bench (Earthborne Rangers, Epoch AI) published 2026-09-18 → · Published board — Learning-capability test: models repeatedly play the obscure campaign board game Earthborne Rangers with note-taking, and the score measures whether results improve across playthroughs.

PostTrainBench v1.1 → · Claude Code — Agents retrain four small base models (gemma-3-4b-pt, SmolLM3-3B-Base, Qwen3-1.7B-Base, Qwen3-4B-Base) for each of 7 benchmark families; the leaderboard value is the weighted mean over base models × benchmarks, aggregated over the 2–3 listed runs per agent.

Terminal-Bench v4.0 → · Claude Code — A benchmark to measure and evolve with the frontier of agent work, whose homepage leaderboard reports resolution rate on Terminal-Bench 4.0 tasks.

Terminal-Bench v4.0 → · Claude Code in --bare mode — A benchmark to measure and evolve with the frontier of agent work, whose homepage leaderboard reports resolution rate on Terminal-Bench 4.0 tasks.

Finance Agent v2 (Vals Index v2) v2 → · Published board — Multi-step financial reasoning tasks.

HLAB — Harvey's Legal Agent Benchmark (Vals Index v2) v2 → · Published board — Long-horizon legal work product creation. The index column is Harvey's Legal Agent Benchmark's own published standalone score, reported as accuracy — not comparable with Artificial Analysis' Harvey LAB-AA row.

Terminal-Bench 2.1 (Vals Index v2) v2 → · Published board — Command-line interface problem solving; the page states that each index column is that benchmark's own published standalone score, so the runner is not asserted here.

Vals Index v2 (Vals AI) v2 → · Published board — GDP-weighted average of agentic model performance across finance, coding and legal tasks.

Artificial Analysis Coding Agent Index v1.4 v1.4 → · Claude Code — The retained AA Coding Agent Index measures coding-agent systems using the earlier three-component implementation.

SciCode (AA subproblems) v1.0.1 → · Published board — Tests scientific Python programming with scientist-annotated background information.

DeepSWE (Datacurve, via Epoch AI) published 2026-09-15 → · mini-SWE-agent — Pass rate of coding agents on original, long-horizon software engineering tasks, run by Datacurve with the mini-swe-agent harness.

FrontierCode 1.1 Main (Cognition) v1.1 → · Claude Code — Whether a maintainer would merge the agent's pull request, on tasks crafted by open-source maintainers and graded with tests, rubrics and verifiers.

SWE-bench Multilingual published 2026-09-10 → · Published board — A 300-instance SWE-bench variant with tasks from 42 repositories across 9 programming languages.

SWE-bench Multimodal published 2026-09-10 → · Published board — A 480-instance SWE-bench variant whose issue descriptions include visual elements.

Code Migration (Vals Index v2 subset) v2 → · Published board — Porting projects to another language, including COBOL modernization.

Vibe Code Bench (Vals Index v2) v2 → · Published board — End-to-end app-building tasks.

FrontierCode 1.1 Extended (Cognition) v1.1 → · Claude Code — Whether a maintainer would merge the agent's pull request, on the full 150-task FrontierCode set crafted by open-source maintainers and graded with tests, rubrics and verifiers.

AA Coding Index v4.x · frozen 2026-10-27 → · Published board — Artificial Analysis withdrew the Coding Index with Intelligence Index v5.0 (2026-10-28). These are the last v4.x values, frozen with their own date (2026-10-27).

ApprenticeBench API cost per task (NeoCognition) published 2026-09-14 → · Claude Code — ApprenticeBench's published USD cost per task for each API-board model-harness-effort configuration on the 100-task accounts-payable job.

ApprenticeBench CUA cost per task (NeoCognition) published 2026-09-14 → · Claude Code — ApprenticeBench's published USD cost per task for each CUA model-harness-effort configuration on the 100-task accounts-payable job.

FrontierCode 1.1 Main cost per rollout (Cognition) v1.1 → · Claude Code — Cognition's published mean USD spend per rollout for each FrontierCode 1.1 Main model and reasoning effort.

Vals Index v2 cost per test (Vals AI) v2 → · Published board — Vals AI's published USD cost per test for each model on the Vals Index v2.

FrontierCode 1.1 Extended cost per rollout (Cognition) v1.1 → · Claude Code — Cognition's published mean USD spend per rollout for each FrontierCode 1.1 Extended model and reasoning effort.

IFBench (AA single-turn) published 2026-09-10 → · Published board — Tests precise single-turn instructions with deterministic rule checks.

Humanity's Last Exam (AA text-only) published 2026-09-10 → · Published board — Tests expert-level knowledge on AA's text-only Humanity's Last Exam subset.

AA-Omniscience Index published 2026-09-10 → · Published board — Tests factual reliability while rewarding correct answers and penalizing hallucinations.

Legal Research Bench (Vals Index v2) v2 → · Published board — Case and statute research with citation-backed answers.

GDP.pdf (AA) published 2026-09-10 → · Published board — Tests professional reasoning over long PDFs with AA document preparation and grading.

AA-LCR v1.1 v1.1 → · Published board — Tests reasoning across multiple long documents with corrected answer keys and grading.

MLCR-AA published 2026-09-10 → · Published board — Tests medical-record synthesis and reasoning across long, fragmented case documents.

FrontierMath Tier 4 v2 (Epoch AI) v2 → · Published board — The hardest, research-level tier of Epoch AI's unpublished FrontierMath problems, run by Epoch AI on its private v2 set.

FrontierMath Tiers 1–3 v2 (Epoch AI) v2 → · Published board — Unpublished, expert-written mathematics problems from undergraduate to research level with automatically checkable answers, run by Epoch AI on its private v2 set.

ArXivMath 06/2026 (MathArena) 2026-06 → · Published board — Research-level math problems with a checkable final answer, drawn from arXiv papers submitted in June 2026; each edition is restricted to papers published within the last month to reduce training-data contamination.

BrokenArXiv 06/2026 (MathArena) 2026-06 → · Published board — Plausible but false proof statements taken from June 2026 arXiv papers; a model scores by refusing to prove them and saying the statement is false as written.

FrontierMath Erdős (Epoch AI) published 2026-09-26 → · Published board — 68 open problems posed or studied by Paul Erdős, formalised in Lean 4, where a model agent must produce a complete Lean proof or disproof that passes the Comparator checker.

Chess Puzzles (Epoch AI) published 2026-09-18 → · Published board — Best-move selection on 100 novel chess positions generated programmatically by Epoch AI, each with a single Stockfish-verified best move; probes spatial reasoning and planning.

LiveBench 2026-06-25 → · Published board — A periodically refreshed suite of objectively graded tasks with separate category scores and an overall category mean.

Mystery Game Puzzles (Epoch AI) published 2026-09-18 → · Published board — Best-move selection on 100 mid-game positions of a well-known game whose identity Epoch deliberately keeps undisclosed, generated programmatically like Chess Puzzles.

CritPt (AA) published 2026-09-10 → · Published board — Tests research-level physics reasoning with Python, symbolic and numerical answers.

GPQA Diamond (AA) published 2026-09-10 → · Published board — Tests graduate-level biology, physics and chemistry knowledge on the Diamond subset.

GPQA Diamond (Epoch AI run) published 2026-09-18 → · Published board — Epoch AI's own inspect-ai runs of GPQA Diamond, the 198-question graduate-level science multiple-choice set.

AutomationBench-AA v1.0.6 → · Published board — Tests multi-app SaaS workflows through REST tools on a held-out AutomationBench split.

EnterpriseOps-Gym-AA published 2026-09-10 → · Published board — Tests enterprise workflows through MCP tools against resettable application environments.

τ²-Bench Telecom (AA) published 2026-09-10 → · Published board — Tests dual-control telecom agents that coordinate tool use with a simulated user.

τ³-Banking (AA) v1.0.1 → · Published board — Tests banking support agents that retrieve policies and change account state through tools.

Excel Modeling Benchmark (Vals Index v2) v2 → · Published board — Building and editing financial models in spreadsheets.

MCPMark Verified verified → · Published board — AI agents complete realistic multi-step tasks through Model Context Protocol servers for Notion, GitHub, a filesystem, Postgres and a Playwright browser, each checked by the task's own verification script; Verified is the maintainers' version-pinned, stabilized 127-task set.

Chartography 100-tasks → · Published board — Can a model read the charts professionals rely on — Kaplan-Meier curves, candlesticks, contour maps, Sankey diagrams, Bode plots, wind roses — from the image alone, to an expert-set precision?

GDP.pdf 100-tasks → · Published board — Can a model answer real professional questions about the PDFs work runs on — manuals, wiring diagrams, filings, forms — so that every rubric criterion an expert wrote is met?

Furniture Assembly Benchmark (Epoch AI) published 2026-09-26 → · Published board — Spatial-reasoning test by Epoch AI: given a photo of a partly or fully assembled IKEA product and its official manual, the model must say whether the build is correct and, if not, name the step with the mistake.