Artificial Analysis Decision Index v2 Scores Calibration: What the Models Gating Your Agent Actually Cost

Artificial Analysis released the second version of its Decision Index on Nov. 8, 2026, and the change that matters is not a reshuffled leaderboard. It is a new headline metric. Every question now scores from 0 to 100 instead of right or wrong: correct answers score above 50, incorrect answers below 50, and the model’s confidence in its top answer moves the score within whichever half it earned. If you route agent actions through a cheap decision model and act only when confidence clears a threshold, that calibration is now part of the grade rather than an unmeasured assumption: Artificial Analysis describes the Decision Score as a headline metric that rewards model calibration.

The second structural change targets benchmark gaming: 45% of the Index is now built on private datasets, up from 25% in v1, and WildGuardTest was removed for high saturation.

What v2 weighs

Component	v2 weight	What changed
AA-Paperclip	30%	New private benchmark: 1,500 resumes, receipts, invoices and research papers
AA-Symmetry	15%	New: 120 questions reworded, reordered, label-swapped and negated
AA-Omniscience-Decision v1.1	15%	Down from 25%; the “I don’t know” option now uses ten phrasings
HLE-MultiChoice v1.1	15%	Down from 25%; adds a decline option in the same ten phrasings
EvalSafe-ONET-AA	15%	New, Artificial Analysis’ implementation of TypeSafe’s ONET
When2Call	10%	Down from 25%, retained
WildGuardTest	0%	Removed as saturated

Those weights sum to 100. Two of the new tests measure failures operators actually see. AA-Symmetry asks whether an equivalent question earns an equivalent probability — the property you need before a reworded prompt silently changes what an agent does. AA-Paperclip scores a whole document as a single decision, correct only if all 15 of its answers are right.

That last rule deserves its own arithmetic.

All-or-nothing scoring compounds: 0.99 to the 15th power is about 0.86, 0.95 is about 0.46, and 0.90 is about 0.21. That is our arithmetic on the rule Artificial Analysis states, not a published score. Two models five points apart per answer (95% vs 90%) are 25 points apart per document (46% vs 21%); near the top, four points apart per answer (99% vs 95%) become a 40-point gap (86% vs 46%). Any pipeline that reads an invoice, a form or a screenshot and needs every field right will feel that gap long before a leaderboard average hints at it.

It also explains the request shape decision models use: a state plus typed questions, answered in one pass with a probability for every option instead of a token stream to parse.

{
  "state": { "document": "invoice-4419.pdf" },
  "questions": [
    { "id": "topic", "options": ["billing", "bug", "shipping"] },
    { "id": "escalate", "type": "yes_no" },
    { "id": "severity", "type": "scale", "levels": [1, 2, 3, 4, 5] }
  ]
}

Illustrative, built from the request structure the Drex paper describes — not a wire spec for any vendor.

The 60x cost spread is a billing-shape story

Cost per 1M tasks spans more than 60x across decision models in Artificial Analysis’ numbers: Celeris-1.1 Decision at $14, Decider v1.2 at $68 and Drex Max v1.1 at $102 sit on the Decision Index versus cost Pareto frontier, while Clef costs $859. The instructive case is GLiDE v1.2 at $780 — nearly 8x Drex Max v1.1 despite list prices only 33% higher. The cause is structural: most decision models, including Drex Max v1.1 and Decider v1.2, bill a document’s input once for all of its questions, while GLiDE v1.2 and Decider Flash v1.2 bill it for every question. Ask that in writing before comparing token prices. It is the bill-shape trap again, and the reason cost per completed task beats $/MTok.

General language models still score higher and cost far more. Claude Haiku 5.5 (Max) scores 77 and GPT-6 Luna (Max) 76, about three points ahead of Drex Max v1.1. GPT-6 Luna (Max) costs $1,530 per 1M tasks against $102, 15x, and takes 11.34 seconds per task against 275 milliseconds, roughly 40x slower. Nace’s product page reproduces the same table to one decimal — 77.1, 76.4, 73.4 — and puts Claude Haiku 5.5 (Max) at $6.2k per 1M tasks.

Two different things are called “Decision Index”

Nace AI followed on Nov. 9 claiming Drex Max v1.1 is #1 on Decision Index and going open-weight soon, linking its paper and product page.

The paper, submitted Nov. 5, 2026 — three days before v2 — reports Max at 67.52 on Decision Index 0.4.1’s public suite against 60.70 for Jev 1.14.0. That is the community Public Decision Index hosted on Hugging Face, chance-corrected so a random guess scores zero, and Nace’s page notes Drex Max is listed there as “Drex Max v1.” So 73.4 and 67.5 are both real numbers for the same model, from two different indexes. Write the edition next to any score you paste into a procurement doc; the same discipline applies to reading agent benchmarks generally.

The #1 claim also needs precision. Artificial Analysis’ own post says three decision models sit within one point: Drex Max v1.1 and GLiDE v1.2 both at 73, Mammoth 27B at 72. Nace’s reproduced table separates the leaders at 73.4 and 72.8. The lead is real, and it is 0.6 of a point on a table published by the vendor selling the model. That page also reports When2Call two ways — 91.8 against 81.7 in one table, 95.3 against 84.4 in its FAQ — without saying which edition each comes from. One more line is easy to miss: Drex Max v1.1’s $0.03 per 1,000 decisions is quoted “at its 50% launch price,” exactly the kind of figure that belongs in a price-expiry register.

What to do with it

• Re-derive your gate threshold. If you act above a fixed confidence, check the model’s AA-Symmetry score first. A gate on a wording-sensitive model is a confidence gate you should not trust at any threshold.
• Reword your decline option. v2 varies “I don’t know” across ten phrasings because a single fixed refusal string is a memorization surface. Test your own refusal wording before it guards a tool.
• Ask how document input is billed — per document or per question — then multiply by your real questions-per-document figure before believing any per-token comparison.
• Record the index edition and the price date with every score you cite internally.
• Treat the private 45% as unverifiable. It is a reasonable anti-gaming move and it also means you cannot reproduce the ranking, so jaggedness on your own tasks still has to be measured.

Every number here comes from the publishers: Artificial Analysis’ announcement post for the index and its results, and Nace AI’s post, paper and product page for Drex. Nace sells the model it places against rivals, so read its reproductions of Artificial Analysis’ table as a claim rather than a record.

Load X post

Loading this post connects to X.

The original source is available above.

Load X post

Loading this post connects to X.

The original source is available above.