Model Benchmark Atlas

Token prices and actual task cost

Standard API rates in USD per million tokens, recorded on 15 September 2026. Subscription plans are separate. Token prices are inputs to a bill; the cost-per-task chart also reflects how many tokens each evaluation consumes.

Astra rates shown for inputs up to 272k tokens. Cache writes, tool fees, long-context tiers and other service modes may differ. Consult the linked provider documentation before budgeting.

SciCode: scientific knowledge expressed as working code

The full benchmark contains 80 main scientific problems divided into 338 subproblems. Artificial Analysis uses the 288 test-set subproblems, supplies scientist-written background, and scores Python code through unit tests. The measured skill combines understanding a scientific specification, translating it into an algorithm, and implementing that algorithm correctly.

Illustrative task, not a quoted benchmark item: implement a numerical routine with the right units, array shapes and boundary conditions. Explaining the physics beautifully would not pass if the function returns incorrect values.

A 56% subproblem score does not mean the model completes 56% of research projects. It also does not directly measure C++, engine architecture, shader quality or frame time. The later SciCode-Verified audit reports defects in the original benchmark; the chart here retains the published Artificial Analysis SciCode values rather than substituting results from the revised test.

GDPval: professional work products, not a knowledge quiz

GDPval-AA v2 evaluates 220 public tasks spanning 44 occupations. Agents use tools to produce files such as reports, spreadsheets and presentations. A panel of model judges compares submitted work; the reported Elo scale is anchored to human expert work at 1000. The v2 environment permits up to 250 turns.

Illustrative task: read reference files, calculate a justified recommendation, and deliver a usable spreadsheet and memo. Factual knowledge helps, but file handling, following requirements, checking calculations and communicating the result also affect the outcome.

Elo describes relative performance in this evaluation. A score of 1700 is not 70% better than a human and is not a percentage of jobs an agent can replace. The environment, tools and judging procedure are part of the result.

Why can a flagship model lag on knowledge work?

“Flagship” names a product position; it does not mean winning every test. At xhigh in these snapshots, Astra and Fable 5.1 both score 53 on the rounded overall index, while Opus 5 scores 50. On GDPval-AA, however, Opus 5 is ahead of Astra. Astra also leads Opus on GDP.pdf and AutomationBench-AA at that setting. Professional deliverables, document reasoning and workflow automation are distinct capabilities.

These observations establish the pattern, not its internal cause. Training emphasis, tool behavior, effort allocation and the judging rubric are possible contributors; the published scores do not isolate which explains the gap. Choose by the work you need done, then verify on representative tasks.