Inkling (Xhigh)

AA-AnalystAgent published 2026-09-10 → · Published board — Tests analyst tasks using agentic Python execution across fourteen domains.

AA-Briefcase published 2026-09-10 → · Published board — Tests multi-week professional knowledge-work projects with linked tasks and large source collections.

GDPval-AA v2 v2 → · Published board — Tests professional knowledge-work deliverables across occupations using AA's Stirrup harness.

Terminal-Bench v2.1 (AA) v2.1 → · Published board — Tests terminal-based work on the 89-task verified refresh using Terminus 2.

Terminal-Bench v4.0 (AA) v4.0 → · Published board — Tests terminal-based work on the 66-task release using mini-SWE-agent v2.4.6.

τ²-Bench Airline (OpenRouter run) published 2026-09-15 → · Published board — Multi-turn service agents making tool calls under strict policy constraints. (OpenRouter's own reproducible run, as published on openrouter.ai/benchmarks.)

Scale SEAL: SWE-Bench Pro V2 HARD published 2026-09-22 → · Published board — Scale SEAL: SWE-Bench Pro V2 HARD results published by Scale AI.

AA Agentic Index published 2026-09-18 → · Published board — Artificial Analysis publishes one Agentic Index per model, measured on its primary configuration and relayed by OpenRouter's Benchmarks API; the value is attached at family scope on the deterministic representative. Artificial Analysis publishes this board without a version number; we keep the date each result was retained.

SciCode (AA subproblems) v1.0.1 → · Published board — Tests scientific Python programming with scientist-annotated background information.

FrontierSWE v2 v2 → · proximus — 34 hand-written real-world software tasks: every model is evaluated with the site’s own Proximus harness at its maximum reasoning effort, 5 trials per task under a 20-hour budget, scored by the site’s own review protocol; the headline value is mean@5 in percent with best@5/worst@5 bounds.

Scale SEAL: SWE-Bench Pro V2 Full published 2026-09-26 → · Published board — Scale AI’s refreshed public SWE-Bench Pro split (642 long-horizon tasks in 11 copyleft repositories) run under a locked protocol, scored by resolve rate.

AA Coding Index v4.x · frozen 2026-10-27 → · Published board — Artificial Analysis withdrew the Coding Index with Intelligence Index v5.0 (2026-10-28). These are the last v4.x values, frozen with their own date (2026-10-27).

DesignArena Web Apps (agentic) published 2026-11-03 → · Published board — Published Elo. Exact source attachment is retained; fewer than 200 battles excludes a row from radar normalization and anomaly peers.

GPQA Diamond (OpenRouter run) — measured cost per task published 2026-09-15 → · Published board — Mean USD OpenRouter actually spent per task while running GPQA Diamond for this configuration.

τ²-Bench Airline (OpenRouter run) — measured cost per task published 2026-09-15 → · Published board — Mean USD OpenRouter actually spent per task while running τ²-Bench Airline for this configuration.

Humanity's Last Exam (AA text-only) published 2026-09-10 → · Published board — Tests expert-level knowledge on AA's text-only Humanity's Last Exam subset.

AA-Omniscience Index published 2026-09-10 → · Published board — Tests factual reliability while rewarding correct answers and penalizing hallucinations.

GDP.pdf (AA) published 2026-09-10 → · Published board — Tests professional reasoning over long PDFs with AA document preparation and grading.

AA-LCR v1.1 v1.1 → · Published board — Tests reasoning across multiple long documents with corrected answer keys and grading.

MLCR-AA published 2026-09-10 → · Published board — Tests medical-record synthesis and reasoning across long, fragmented case documents.

LiveBench 2026-06-25 → · Published board — A periodically refreshed suite of objectively graded tasks with separate category scores and an overall category mean.

AA Intelligence Index v5.0 · snapshot-2026-11-03 → · Published board — Artificial Analysis Intelligence Index v5.0. Models AA has not rescored on v5.0 are listed as estimates from their frozen v4.x value (isotonic fit over the models measured on both versions), never as measurements.

EQ-Bench 4 v4 → · Published board — Measures active emotional and social intelligence of LLMs in multi-turn roleplay chats with simulated personas, ranked by an LLM-judge panel Elo.

CritPt (AA) published 2026-09-10 → · Published board — Tests research-level physics reasoning with Python, symbolic and numerical answers.

GPQA Diamond (AA) published 2026-09-10 → · Published board — Tests graduate-level biology, physics and chemistry knowledge on the Diamond subset.

GPQA Diamond (OpenRouter run) published 2026-09-15 → · Published board — Graduate-level science questions that resist retrieval and reward careful reasoning. (OpenRouter's own reproducible run, as published on openrouter.ai/benchmarks.)

AutomationBench-AA v1.0.6 → · Published board — Tests multi-app SaaS workflows through REST tools on a held-out AutomationBench split.

EnterpriseOps-Gym-AA published 2026-09-10 → · Published board — Tests enterprise workflows through MCP tools against resettable application environments.

τ³-Banking (AA) v1.0.1 → · Published board — Tests banking support agents that retrieve policies and change account state through tools.

MCP Atlas (Scale AI) published 2026-09-21 → · Published board — How reliably a model completes realistic multi-step workflows over real Model Context Protocol servers: discovering the right tool in a noisy tool menu, calling it with correct parameters, recovering from errors and synthesising the results into an accurate final answer.

Toolathlon-Verified (HKUST NLP) 2026-06-30 → · Published board — An agent completes 108 real-world tool-use tasks across 32 MCP servers and 7 local toolkits, and each task is graded by executing the resulting state against the task's ground truth.

MMMU Pro (AA) published 2026-09-10 → · Published board — Tests multimodal understanding using challenging ten-option questions.

Chartography 100-tasks → · Published board — Can a model read the charts professionals rely on — Kaplan-Meier curves, candlesticks, contour maps, Sankey diagrams, Bode plots, wind roses — from the image alone, to an expert-set precision?

EQ-Bench Creative Writing v3 v3 → · Published board — A LLM-judged creative writing benchmark with vocabulary and GPT-slop controls.

EQ-Bench Longform Creative Writing v1.11 → · Published board — LLM-judged benchmark in which models plan and write a short story/novella over 8x 1000-word turns, scored 0-100 across 14 rubric dimensions.