Evals for AI Agents: How to Measure Software That Acts

Three weeks ago you moved your coding agent to a newer model. It felt sharper for two days. Then it quietly stopped rerunning the test suite before committing, and a regression landed that nobody could trace — because nothing was measured, so there was nothing to trace it to. So, what are evals in AI? Evals are systematic, repeatable tests that score an AI system’s outputs against defined expectations, so you can measure quality, catch regressions, and compare changes with numbers instead of vibes.

Shipping prompt, model, or tool changes without evals is manual QA on a non-deterministic system — the one testing strategy known to fail everywhere it’s tried. If you build agentic software for a living, an eval suite is the difference between “the agent got worse” being a bug report and being a mood.

This guide covers what to measure and how to score it: the straight definition, why agents are harder to evaluate than chat models, the four grader types, a five-step plan for your first suite, and the pass@k statistics that make the numbers honest. The sandbox that safely runs agent evals is its own subject — that’s the test harness for agentic software, and we’ll point there when the two meet.

What are evals in AI? The straight answer

AI evals are systematic, repeatable tests that score an AI system’s outputs against defined expectations, so teams can measure quality, catch regressions, and compare prompt, model, or tool changes objectively — the difference between knowing an agent improved and feeling like it did.

Every eval, however fancy the tooling, has three parts:

• Dataset — the tasks: real tickets, real repos, real questions, with expected outcomes.
• Target — the thing under test: a model, a prompt, or a whole agent with its tools.
• Grader — the scorer: code, a judge model, or a human.

Evals get confused with two neighbors. Unit tests are deterministic and binary — the same input yields the same output, pass or fail. Benchmarks — SWE-bench Verified, GAIA, τ-bench — compare models on public task sets, and reading them well is its own skill. Evals measure your system on your tasks.

	Unit tests	Benchmarks	Evals
What’s tested	Deterministic code	Public models on public tasks	Your system on your tasks
Scoring	Binary, exact	Leaderboard percentage	Statistical, graded
Who runs them	Your CI	Labs and researchers	Your CI

Two one-line examples. A support bot scored against 500 real historical tickets, judged on resolution and tone. A coding agent scored on 50 tasks from your own repos, where the grader checks out the agent’s branch and reruns the test suite.

Where the word came from: OpenAI Evals and the lineage

The term escaped the labs with OpenAI Evals, the open-source repository OpenAI published alongside GPT-4 in March 2023. It gave the practice a shape — YAML-templated test definitions, a registry of community-contributed suites — and made “write evals” standard advice for LLM builders, the way “write tests” once spread through software teams.

What the repo is today: a usable OSS harness for model-level checks, receiving maintenance rather than momentum, and not an agent-evaluation platform. OpenAI’s active investment moved to hosted evals in its platform dashboard — dataset management, trace grading, regression comparisons against logged production traffic. If you’re building on that stack, the hosted product is the natural home for what this article describes; our review of OpenAI’s agentic stack covers where it fits. Either way, the ideas below are vendor-neutral — the repo’s real legacy is the vocabulary.

Why agent evals are harder than model benchmarks

A chat model yields one output you can score in isolation. An agent yields a trajectory — a sequence of tool calls, decisions, and recoveries — and that changes the job:

• Trajectories, not outputs. An agent can be right for wrong reasons (a lucky diff after ignoring the failing test) or wrong despite a sane path (correct diagnosis, botched final edit). Practice that answers it: graders that see the transcript, not just the final artifact.
• Side effects. Agents edit files, run shell commands, and open PRs. Scoring requires a sandbox that absorbs and inspects those effects — snapshotted repos, disposable containers, recorded network calls. That sandbox is the harness layer covered in our QA-for-agents guide.
• Non-determinism. The same task yields five different trajectories on five runs. A single-run pass/fail is mostly noise, which forces statistical pass criteria — pass@k and friends, covered below.
• Partial credit. An agent that locates the right bug but ships a wrong fix is meaningfully better than one that never finds it. Binary graders erase that signal; rubric graders and step-level checks preserve it.

None of this makes agent evals impractical. It makes them specific: transcript-aware graders, sandboxed execution, multiple runs per task. That’s the toolbox.

The eval type toolbox

There are four grader families, and mature suites use all of them. Ordered by cost and determinism: programmatic assertions (cheap, deterministic), LLM-as-judge (cheap-ish, stochastic), human review (expensive, high signal), and online evals (production-priced, continuous). The rule of thumb: assert what code can verify, judge what needs a rubric, sample humans for taste and risk, watch production for the rest. Grader choice is per task type, not per suite — one suite happily mixes all four.

The four grader families. Start in the bottom-left and add the others as the suite matures.

Programmatic assertions

Deterministic code checks: the test suite passes, the diff applies cleanly, the JSON validates, the exit code is 0, no forbidden strings appear in the transcript. For a “fix the failing test” task, the grader reruns pytest and asserts two things — the target test now passes, and nothing previously green broke:

def grade(workspace) -> bool:
    # Golden 0007: fix test_retry_backoff without breaking the suite
    target = run(["pytest", "tests/test_webhooks.py::test_retry_backoff", "-q"], cwd=workspace)
    suite = run(["pytest", "-q"], cwd=workspace)
    return target.returncode == 0 and suite.returncode == 0

Strengths: cheap, fast, zero variance — you can rerun the suite hourly without a budget meeting. The boundary: assertions can’t judge code quality or approach sanity; a hideous-but-green diff passes. Design for assertions first anyway. Tasks with programmatically checkable outcomes keep the whole suite cheap to rerun, and cheap-to-rerun is what makes everything else in this article stick.

LLM-as-judge

A model grades outputs in one of two modes: rubric scoring (1–5 against written criteria) or pairwise comparison (which of two outputs is better) — with pairwise generally more reliable for catching regressions between versions. The judge approach was formalized in the MT-Bench / Chatbot Arena work, which also documented the biases you must design around:

• Position bias — judges favor the first answer shown; mitigate by scoring each pair twice with order swapped.
• Verbosity bias — longer answers score higher; mitigate by capping or normalizing length.
• Self-preference — models favor their own family’s outputs; mitigate by judging with a different model family than the target.

Calibration is not optional: spot-check judge scores against human labels before trusting them, and re-calibrate whenever the judge model changes. A rubric fragment for grading an agent’s PR description:

Score the PR description 1–5:
5 — Every claim matches the diff; breaking changes flagged; test evidence linked.
4 — Accurate but incomplete; minor omissions, nothing misleading.
3 — Mostly accurate; one unsupported claim or a missing breaking-change note.
2 — Describes intent rather than the actual change; no test evidence.
1 — Contradicts the diff or claims work that didn't happen.

Use a strong, cheap-enough judge from a different family than the agent under test, and pin its version.

Human review and online evals

Humans are the expensive, high-signal grader — reserve them for taste, safety-adjacent behavior, and bootstrapping new eval types. A workable cadence: 10–20 sampled trajectories per week, binary verdict plus a one-line reason, half an hour total. The flywheel is the point: human labels calibrate your LLM judges, and as judges earn trust, the human queue shrinks to spot checks.

Online evals score live traffic instead of a fixed dataset: retry rate, human-intervention rate, task-completion rate, cost per completed task. They’re the graduation step, not the starting point — offline evals catch regressions before rollout; online evals catch what your dataset didn’t cover. The mechanics are ordinary A/B discipline: canary the new prompt or model to 10% of sessions, compare intervention and completion rates against control, then roll forward or back. These metrics belong on the same dashboard as spend and reliability — that’s the AgentOps view.

Building your first agent eval suite

A useful v1 is bounded: 20–50 golden tasks, one CI job, one part-time week. The five steps: collect golden tasks from real sessions → define graders → set pass thresholds → run in CI → track regressions. The anti-goal is starting by shopping for a platform. Start with tasks and graders as files in your repo; adopt tooling when experiment volume hurts, not before.

Throughout, we’ll use one concrete running example: a team whose agent works on payments-service, building a suite of 30 goldens.

Step 1: mine golden tasks from real sessions

A golden task is a real task your agent has faced, packaged to be rerun: representative of actual work, self-contained enough to execute from a snapshot, and verifiable by a grader. Never invent toy problems — your own history is strictly better. The migration the agent aced is a golden. The refactor where it broke CI is a better one.

Selection heuristics: cover your task mix roughly by frequency (if 40% of agent work is test fixing, ~40% of goldens should be), over-sample past failures, and hold out ~20% of tasks that never influence prompt tuning — you’ll want an untouched set to detect overfitting later. Package each golden as prompt + starting state + expected outcome + grader reference, in version control:

# evals/goldens/0007-fix-flaky-webhook-test.yaml
prompt: >-
  tests/test_webhooks.py::test_retry_backoff fails intermittently on main.
  Find the cause and fix it without changing the public API.
starting_state: snapshots/payments-service@9c41f2e
expected: target test passes 5/5; full suite green; diff confined to retry logic
grader: graders/pytest_rerun.py
runs: 5

Product note: The best goldens are sessions you already ran. Automater Lite archives every session from 10+ CLIs locally, full-text searchable — mine last month’s wins and failures for golden tasks and regression cases. Free on automater.ai.

Steps 2–5: graders, thresholds, CI, and regression tracking

Graders (step 2). Match grader to task type per the toolbox: pytest reruns for code tasks, schema checks for extraction tasks, a judge with a rubric for prose artifacts like PR descriptions. Version graders beside the prompts they test — a grader change can move scores as much as a model change, and you want both in the same diff.

Thresholds (step 3). Set them empirically, never aspirationally. Our example team runs the current agent five times on each of the 30 goldens: 123 of 150 runs pass, so the baseline is 82%. With n=150, the 95% confidence interval is about ±6 points, so the gate becomes “fail the build below 76%” — no worse than baseline minus noise. Ratchet the gate up deliberately as the agent improves; never let it drift down silently.

CI (step 4). Two tiers. A per-commit smoke: 8 high-signal goldens, single run each, minutes of wall clock, cents of spend. A nightly full suite: all 30 goldens at five runs, budget-capped. Failures must link the transcript and the diff, not just emit a score — a red number nobody can investigate gets ignored within a month.

# .github/workflows/agent-evals.yml — nightly tier
on:
  schedule: [{ cron: '0 3 * * *' }]
jobs:
  full-suite:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python evals/run_suite.py --goldens evals/goldens --runs 5 --budget-usd 12
      - run: python evals/gate.py results/ --min-pass 0.76 --attach transcripts

Regression tracking (step 5). Every escaped defect becomes a permanent golden the same day it’s diagnosed — the agent-era version of the regression test. Six months in, the suite is a fossil record of everything that ever bit you, which is precisely what makes it valuable. The sandboxing and replay mechanics that make reruns trustworthy live in the agentic test harness.

pass@k and statistical pass criteria for engineers

pass@k is the probability that at least one of k attempts succeeds — a capability metric with its lineage in the HumanEval / Codex paper. Its strict sibling pass^k — all k attempts succeed — is the reliability lens τ-bench pushed for tool-using agents, and it’s the right metric whenever users get one shot.

The spread between them is the most underrated number in agent engineering. Take a golden with a 60% single-run success rate:

Single-run success	pass@5 (≥1 of 5 succeeds)	pass^5 (all 5 succeed)
50%	96.9%	3.1%
60%	99.0%	7.8%
80%	99.97%	32.8%
90%	~100%	59.0%
95%	~100%	77.4%

The same agent, three different truths. pass@5 flatters; pass^5 is what a one-shot user experiences.

The 60% agent looks nearly perfect at pass@5 and nearly useless at pass^5 — and both numbers are true. Which one matters depends on the product: a background fixer that can retry five times lives on pass@k; an interactive assistant whose user walks away after one bad answer lives on pass^k. State which number you’re reporting, every time. It is remarkable how much optimism hides in an unlabeled percentage.

Working statistics for the rest: run each golden 5–10 times (five runs quantize a task’s rate to 20-point steps — fine for gating, too coarse for fine comparisons). Report suite success with a confidence interval, as in the worked threshold above (82% ± 6 on 150 runs). And only call something a regression when it lands outside the noise band — chasing two-point wiggles inside the interval is how teams lose faith in their own suite.

Metrics beyond accuracy

Accuracy misses failure modes that cost real money. Record these per run, so every experiment reports quality and efficiency together:

• Cost per completed task — tokens × pricing, divided by successful runs. Moves when models get chattier or loop longer, even at flat accuracy.
• Latency per task — wall clock, end to end. Moves with model speed, retry storms, and tool timeouts.
• Tool-error rate — malformed calls, schema violations, retries. The earliest warning that a model swap changed tool-calling behavior.
• Human-intervention rate — how often a person had to step in. The closest single proxy for “is this agent actually autonomous.”
• Steps to completion — trajectory length. Quiet doubling here predicts cost and latency pain before the invoice does.

The concrete failure this catches: a team swaps models, success holds at 82%, and mean tokens per task go from 180K to 540K. Accuracy dashboards show nothing; the cost-per-task chart shows a cliff. On live traffic, these same metrics are the operations dashboard — the AgentOps view of a fleet is this list, aggregated.

Eval-driven development: write the eval first

The practice that makes all of this compound: before changing a prompt, a model, or tool wiring, write the eval cases that would prove the change worked. It’s the agent-era mirror of test-driven development, and it produces the same discipline — no prompt, model, or tool change merges without an attached eval delta. Make that a PR rule, not a norm.

The side effect is the real payoff: writing a grader forces the team to define “good” precisely. Half the arguments about whether the agent got better dissolve once someone has to encode the criteria in a rubric or an assertion.

Where it pays fastest: model migrations, provider swaps, and “quick” system-prompt edits — the three leading sources of silent regression in agent systems. If a change is too small to eval, it’s too small to explain the incident it causes.

The tooling in 2026, honestly

One line each, no ranking. OpenAI’s hosted evals: dashboard-native trace grading, natural if you’re already on that platform, OpenAI-centric by design. LangSmith: datasets and experiments with first-class tracing, the path of least resistance on LangGraph — see our LangGraph review for that pairing. Braintrust: fast experiment loops and the Autoevals library, popular with teams iterating daily. Promptfoo: open-source, config-driven, built to run in CI — closest in spirit to this article. On the open-source bench beside it: Langfuse for self-hosted tracing with eval hooks, DeepEval for judge-based testing in pytest style, and Inspect, the UK AI Security Institute’s rigorous eval framework.

The unfashionable advice: your v1 suite needs none of them. Pytest, a directory of YAML goldens, and a cron job cover the first several months; adopt a platform when experiment volume — not enthusiasm — demands it. And guard the exit: your golden tasks and human labels are the asset, so prefer tools that export datasets and traces in open formats.

How eval suites rot: failure modes to design against

• Overfitting to goldens. Prompts get tuned against the suite until scores rise while real performance stalls. Countermeasure: the 20% holdout set, plus rotating fresh tasks in quarterly.
• Judge drift. A judge-model update silently shifts every rubric score; last month’s 4.2 and this month’s 4.2 aren’t the same number. Countermeasure: pin judge versions, log them with every result, re-calibrate against human labels on every change.
• Evals nobody reruns. A suite outside CI is decorative. If it neither blocks merges nor pages anyone on failure, it has already rotted. Countermeasure: the two-tier CI wiring above, with the gate set to actually fail builds.
• Stale goldens. The product moved; the tasks didn’t. Countermeasure: a quarterly review pass that retires dead tasks and mines the last quarter’s sessions for new ones.
• Benchmark substitution. Leaderboard deltas get cited in place of your own numbers — “the new model is +4 on SWE-bench” answering a question about your agent. Countermeasure: the suite itself; public benchmarks pick your shortlist, your evals pick your model.

Start this week

The arc in one line: evals turn “it feels smarter” into a number, and agents demand the statistical, transcript-aware version of that discipline. You don’t need a platform, a budget, or a quarter. This week: pull ten recent sessions from your archive, package five of them as goldens with programmatic graders, wire the smoke tier into CI, and record your baseline with five runs per task. That’s the whole entry fee — everything else in this article is compounding interest. If you run several coding agents side by side, do it once and point every harness at the same suite — the multi-agent command center approach — and let the numbers referee.

FAQ: evals in AI

What are evals in AI?

Evals are systematic, repeatable tests that score an AI system’s outputs against defined expectations. Every eval has three parts: a dataset of tasks, a target (the model, prompt, or agent under test), and a grader. Teams use them to measure quality, catch regressions, and compare changes objectively.

What is OpenAI Evals?

OpenAI Evals is the open-source repository OpenAI published alongside GPT-4 in 2023 — YAML-templated tests plus a community registry — which popularized “write evals” as standard advice. It remains a usable harness for model-level checks; OpenAI’s active investment moved to hosted evals in its platform dashboard.

Are evals the same as benchmarks?

No. Benchmarks — SWE-bench Verified, GAIA, τ-bench — compare models on public task sets, which is useful for shortlisting. Evals measure your system on your tasks: your prompts, tools, and codebase. A model that climbs a leaderboard can still regress your agent; only your own evals catch that.

What is LLM-as-a-judge?

LLM-as-a-judge uses a model to grade outputs against a written rubric or in pairwise comparison. It’s cheaper than human review and handles fuzzy criteria, but carries biases — position bias, for one, mitigated by scoring each pair twice with order swapped — so calibrate judges against human labels.

What does pass@k mean?

pass@k is the probability that at least one of k attempts succeeds — a capability metric from the HumanEval lineage. Its stricter sibling pass^k requires all k attempts to succeed, which measures reliability. An agent with 60% single-run success scores ~99% pass@5 but only ~8% pass^5.

Sources

• OpenAI Evals repository — github.com/openai/evals
• OpenAI hosted evals documentation
• Chen et al., “Evaluating Large Language Models Trained on Code” (HumanEval, pass@k)
• Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”
• Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”
• Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”
• Promptfoo
• LangSmith — LangChain
• Langfuse
• Braintrust