The Test Harness, Reinvented: QA for Agentic Software

Last month, one of our agents opened a pull request: 14 files changed, every check green — and the tests it passed were tests the same agent had written twenty minutes earlier. Whether merging that PR is routine or reckless has almost nothing to do with which model wrote it. It has everything to do with the harness around it.

This article serves two readers. If you came for the textbook definition of a test harness in software testing — the term QA teams have used for decades — it is one scroll down, in 49 words, with the component list and the harness-versus-framework table the glossaries never quite nail. If you ship agent-written code daily and suspect that textbook no longer covers non-deterministic output, real side effects, and token-metered runs, you are also right, and the rest of the piece rebuilds the harness for that world.

The claim we will defend along the way: the harness, not the model, separates teams that trust their agents from teams that babysit them. Consider this the QA chapter of our agentic software series — classic harness first, why agents break it, then the 2026 blueprint.

What is a test harness in software testing?

A test harness in software testing is the framework of supporting code and tools that executes tests automatically and collects the results: drivers that invoke the software under test, stubs and mocks that stand in for dependencies, fixtures that establish known starting state, and reporters that record what happened.

That is the test harness definition worth memorizing. In plainer terms, when an engineer says “test harness,” they mean the rig that holds software still so tests can act on it and record the outcome. The metaphor predates computing: a harness is the rigging that lets a driver control a horse — straps and instrumentation, not muscle. A test harness is control and instrumentation strapped around code.

Classically, the harness earns its keep in two places. First, unattended automation: CI runs the suite at 3 a.m., nothing prompts a human, and the results are waiting in the morning. Second, integration testing against components that do not exist yet: stubs answer for the unfinished payment service so checkout logic can be exercised today.

Test harness vs test framework vs test runner

The three terms blur because modern tools bundle all of them, but the separation matters the moment you start replacing pieces. A framework gives you structure and assertions. A runner discovers and executes tests. The harness is the whole rig. Which means: pytest alone is not your harness. pytest plus conftest fixtures, a docker-compose file of dependencies, seeded data, and the CI job that reports results — that is a harness.

Term	What it provides	Example
Test framework	Structure and assertions for writing tests	pytest, JUnit, Jest
Test runner	Discovers, executes, and reports tests	pytest CLI, Jest runner, gradle test
Test harness	The whole executing rig: framework + runner + fixtures, doubles, environments, data, CI glue	pytest + conftest + docker-compose Postgres + seeded rows + JUnit-XML in CI

You will also hear software harness in embedded and hardware-in-the-loop shops — a looser synonym for the same rig, often wrapping physical devices rather than services.

One more collision, and it is new: in AI engineering, an agent harness is the software wrapped around a language model to make it act — Claude Code, Codex CLI, and OpenCode are harnesses in that sense, and “harness engineering” is now a named discipline with its own awesome list. Same metaphor, different rig; our agent harness field map covers that meaning. (There is also a CI/CD vendor named Harness. The word is doing a lot of work in 2026.) This article uses test harness throughout: the rig that tests an agent — which is itself running inside an agent harness. Two harnesses, one nested in the other.

The components of a classic test harness

Four parts, each with a job you can point to:

• Driver — invokes the unit under test with controlled inputs: a pytest function calling parse_invoice() with a crafted PDF, or a script POSTing fixtures to a locally started service.
• Stubs and mocks — stand-ins for dependencies: unittest.mock replacing the payment client so no card is ever charged; WireMock impersonating a flaky third-party API with exactly the 429 response you want to exercise.
• Fixtures and test data — known starting state: testcontainers spinning up a throwaway Postgres; factory_boy seeding twenty orders in known states before each run.
• Assertions and reporting — judgment plus paper trail: plain asserts or Hamcrest matchers, feeding JUnit-XML that CI renders as an annotation on the exact failing line.

Hold that four-part structure in your head. The agentic blueprint below echoes it piece for piece.

Why agents break the classic test harness

Point that rig at an agent — software that plans, calls tools, edits files, and decides when it is done — and four classic assumptions die in the first week.

• Non-determinism kills exact-output assertions. The same task produces different trajectories run to run. Pass/fail stops being a bit you read and becomes a distribution you sample — the statistical machinery lives in our guide to evals for AI agents.
• Side effects escape the mock boundary. The thing under test edits files, runs shell commands, opens PRs, and calls live APIs. A mock at one function boundary contains none of that; containment has to move to the environment.
• Every run costs real money. A classic suite is free to rerun forever. A 500-case agent suite at frontier-model prices is an invoice, and “run everything on every commit” now needs an economics answer, not just a scheduler.
• The oracle problem widens. One bug admits many valid fixes. A golden-string assertion rejects correct work; graders have to accept equivalence classes (“the hidden test passes and nothing else broke”), not exact artifacts.

The agentic test harness blueprint

None of this argues for throwing the harness out. It argues for re-deriving each component for software that acts:

Classic component	Agentic equivalent
Fixture (known starting state)	Sandbox (disposable environment)
Stub / mock	Recorded tool responses, replayed
Assertion on final output	Trajectory assertions on the transcript
Binary pass/fail	Statistical criteria over n runs
Timeout	Budget cap (steps, tokens, dollars)

The classic harness re-derived: every column on the left has a working descendant on the right.

The design goal across all five: every run safe to leave unattended, cheap to repeat, and inspectable afterward. And note what transfers from classic practice unchanged — version everything, run it in CI, report legibly.

Sandboxed environments

The fixture’s descendant is the sandbox, and it comes in weights:

• Containers or devcontainers per run for most coding tasks; template-cloned throwaway repos so every scenario starts from a pinned commit; seeded databases (a dockerized Postgres with known rows) for anything stateful; microVM sandboxes (E2B, Modal, Daytona) when the agent runs arbitrary commands.
• The invariant: the agent must be unable to touch anything the harness cannot recreate. No shared credentials, no production DNS, no writable network mounts. This is the least-privilege discipline from securing AI agents, applied at test time.
• Reproducibility: scenarios start from a pinned commit and dataset, so any failure replays exactly.
• Cleanup: sandboxes are disposable by construction. If teardown needs a runbook, the design is wrong.

A minimal compose fragment says most of it:

services:
  workspace:
    build: .devcontainer/
    volumes: ['./scenario/repo:/workspace'] # template clone, pinned commit
    networks: [sandboxed]
  db:
    image: postgres:17
    volumes: ['./scenario/seed:/docker-entrypoint-initdb.d:ro']
    networks: [sandboxed]
networks:
  sandboxed:
    internal: true # no egress: nothing to leak, nothing to break

Recorded and replayed tool responses

The stub’s descendant has a lineage: vcrpy, Ruby’s VCR, and nock all record real HTTP responses once and replay them deterministically. The agentic version records tool-call responses — the web search, the API lookup, the registry query — and replays them on later runs.

• What replay buys: deterministic reruns for debugging, near-zero token and API cost for smoke suites, and immunity to third-party hiccups failing your build.
• Where to intercept: record at the tool boundary — the MCP server or tool-call layer — not inside the model call, so cassettes survive model swaps. The 2026-07-28 MCP spec, with its stateless request/response core, makes that interception point cleaner than it used to be.
• Where replay lies: state-dependent tools rot cassettes. A search whose results moved, a repo that drifted — both will replay a world that no longer exists. Set a refresh policy (re-record weekly) and mark replayed runs in reports so nobody mistakes them for live results.
• The working rhythm: record Monday’s live nightly run; replay it in every PR check all week.

Scenario suites and regression cases

A scenario is the agentic test case: seed + goal + grader. A repo with a planted bug to fix. A half-applied migration to complete. A failing pipeline to diagnose.

• Sourcing: the best scenarios are last month’s real failures. Every incident, wrong fix, and abandoned session becomes a permanent regression case — the agentic equivalent of “write a test for every escaped bug.”
• Composition: evergreen scenarios weighted by how often the task type actually occurs, plus a growing regression tail. Review quarterly for staleness.
• Graders: the harness executes what the eval suite defines — grader design (assertions, LLM judges, calibration) lives in our agent evals guide.

A scenario file needs no ceremony:

seed: repo@9f3ab21 + seed/orders.sql # pinned, replayable
goal: 'Checkout 500s when coupon_code is null — find it, fix it'
grader: hidden_test_passes && suite_green && no_new_migrations

Product note: Regression scenarios start life as real sessions. Automater Lite keeps a local, searchable archive of every session across 10+ CLIs — search “rolled back,” find the session that broke the migration, pin it as a permanent scenario. Free on automater.ai.

Trajectory assertions

The assertion’s descendant judges the path, not just the artifact: checks over the sequence of tool calls and decisions the agent made. The surface they run against is the session transcript — a JSONL log of tool calls the sandbox captures by default.

assert "rm -rf" not in t.commands              # unless the task says so
assert t.steps <= 40                           # a looping agent fails loudly
assert t.first("Edit") > t.first("Read")       # look before touching
assert t.index("run_tests") < t.index("done")  # verify before declaring victory

The discipline: assert invariants — safety, sanity, ordering — never the exact path. Agents legitimately vary, and a harness that punishes legitimate variation trains the team to ignore its failures.

Trajectory assertions run over the transcript: the path is inspectable even when the destination varies.

Statistical pass criteria and budget caps

Binary pass/fail is the last classic component to fall. In its place:

• Sampled results: run each scenario 5–10 times and report the success rate, gated with a noise band. pass@k is the probability at least one of k attempts succeeds — the capability lens, from the Codex/HumanEval lineage. pass^k is the probability all k succeed — the reliability lens your users actually experience.
• Budget caps as first-class citizens: per-run token and dollar ceilings kill runaway loops; per-suite caps fail the build loudly instead of silently draining the account.
• Flake handling: a quarantine list for unstable graders, a majority-of-three rerun policy, and every quarantine entry requires an owner and an expiry date.
• Cost in the report: every run publishes scenarios passed, success rates, and dollars spent as one artifact.

End state, one line in your CI config: ship if mean success ≥ 85% on n=5 and suite spend ≤ $12.

Testing both directions: agent-written code and the agent itself

“Testing agentic software” means two different things, and most teams need both. One sandbox serves both.

QA of the output — the code agents write. Hold it to stricter CI than human code, because review attention per line is lower:

• Mutation testing (mutmut for Python, Stryker for JS/TS) catches assert-nothing tests by seeding faults and checking the suite notices.
• Property-based testing (Hypothesis) hunts the edge cases nobody typed into an example.
• Coverage ratchets — coverage may never drop below its high-water mark — and CODEOWNERS gates on sensitive paths.

The trust problem deserves its name: when the same model writes the code and its tests, green tests are weak evidence — the tests may simply assert too little. Mutation score is the honest check; a seeded fault does not care who wrote the asserts.

QA of the agent — its behavior over time. That is the scenario suites and statistical gates above, rerun on every prompt, model, or tool change. Treat those changes like dependency upgrades, because that is what they are.

Wiring the harness into CI without burning the budget

The economics work when you tier the suite:

Tier	Trigger	Scope	Tool mode	Budget
Smoke	every PR	3–5 scenarios × 1 run	replayed cassettes	cents
Full	nightly cron	all scenarios × 5 runs	live tools in sandboxes	capped (say $40/night)
Deep	weekly	long-horizon scenarios; re-record cassettes	live	largest; reviewed monthly

Three habits keep the tiers honest. Rotate a random 20% of the full suite into PR runs, so coverage creeps upward without cost exploding. Publish spend next to pass rate on every run — a suite that costs more than the engineering time it saves gets redesigned, like any other tool. And give nightly failures an owner with a paging path, or the full tier becomes decorative; pipeline patterns live in agentic CI/CD and the ownership question in AgentOps.

A worked example: harness spec for a bug-fixing agent

Concrete enough to build this sprint. The inventory: a seeded repo with 20 known bugs across five classes — off-by-one, missing null check, wrong API usage, stale cache invalidation, race condition — each pinned at a reproducible commit with a hidden reference test the agent never sees.

Scoring per attempt: fixed (hidden test passes, suite green), partial (bug located, fix wrong), failed, and broke-something (a previously green test now red) — the last weighted worst, because it is the outcome that erodes trust in every future run.

Runs: k=5 attempts per bug. Report per-bug success rate, aggregate pass@5 and pass^5, and mean cost and steps per fixed bug. Illustrative results:

Bug class (4 bugs each)	Mean success (n=5)	Solved at least once	Broke-something runs	Mean cost per fix
Off-by-one	90%	4/4	0	$0.60
Missing null check	85%	4/4	1	$0.75
Wrong API usage	70%	4/4	1	$1.10
Stale cache invalidation	45%	3/4	2	$1.90
Race condition	30%	2/4	4	$2.40

Read the table like an operator: this agent is a safe bet on shallow bugs, a coin flip on cache bugs, and an active hazard on races. So route race conditions to humans and let it run on the rest. That routing decision is the harness paying rent.

The config is five lines plus discipline:

suite: bugfix-bench-v3
bugs: scenarios/bugs/*.yaml # 20 files: seed + goal + grader each
runs_per_bug: 5
budgets: { per_run_usd: 4, per_suite_usd: 60, max_steps: 40 }
quarantine: requires { owner, expiry }

Flakes get the same treatment as production: grader disagreement across reruns sends the bug to quarantine with a named owner, and no result counts without majority-of-three agreement.

Test harness tooling in 2026, briefly

• The SWE-bench harness remains the reference implementation for repo-level scenarios — containerized repos, hidden tests, patch grading. Worth reading even if you never run it.
• Inspect, from the UK AI Security Institute, is the strongest open-source option for structured agent evals — solvers, scorers, and sandboxing with opinions.
• promptfoo gets config-driven checks into CI in an afternoon.
• LangSmith and Braintrust offer hosted experiment tracking — reasonable when you want dashboards before infrastructure.
• Agent SDKs increasingly ship harness primitives — sandboxing, transcripts, replay; see the Claude Agent SDK and OpenAI Agents SDK docs.

The assembly advice has not changed all year: no off-the-shelf product is your harness. Teams that do this well assemble containers, pytest, transcript storage, and one eval tool — buy observability, build scenarios. And watch data gravity: transcripts and scenario suites are the asset, so keep them exportable and local-first. The same rule governs picking among the best agentic AI tools generally — the tool is replaceable; your scenario suite is not.

The harness is the asset

Models are rented; the harness is owned. Every model swap of the past year — and there were many — rewarded the teams whose harness could answer “is the new one better on our tasks?” by tomorrow morning, and punished the teams still arguing from vibes.

The this-sprint version: stand up one sandbox, write five scenarios from your last five real failures, and wire one budget-capped nightly job at n=5. That is a weekend of work, and it changes every model, prompt, and tool decision you make afterward from an argument into a number.

FAQ: test harness in software testing

What is a test harness in software testing?

A test harness is the supporting rig that runs tests automatically: drivers invoke the software under test, stubs and mocks replace dependencies, fixtures establish known starting state, and reporters collect results. It serves unattended CI automation and integration testing against components that are not finished yet.

What is the difference between a test harness and a test framework?

A framework supplies structure and assertions for writing tests — pytest, JUnit, Jest. The harness is the whole executing rig: framework plus runner, fixtures, test doubles, environments, seeded data, and CI reporting. You write tests in a framework; you trust results because of the harness around them.

What is an example of a test harness?

For a Python service: pytest as the framework, conftest fixtures for setup, testcontainers running a throwaway Postgres seeded by factory_boy, WireMock faking a third-party API, and a CI job publishing JUnit-XML results. No single tool is the harness — the assembled rig is.

How do you test AI agents?

Run them in sandboxed scenario suites: pinned starting state, a goal, and a grader. Assert on the trajectory — ordering, safety limits, step ceilings — rather than exact outputs, then run each scenario five or more times and gate on success rates under budget caps. Grader design is covered in evals for AI agents.

Sources

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv)
• Evaluating Large Language Models Trained on Code — the pass@k lineage (arXiv)
• Model Context Protocol: the 2026-07-28 specification
• awesome-harness-engineering (GitHub)
• vcrpy documentation
• Inspect — UK AI Security Institute
• promptfoo
• LangSmith — LangChain
• Claude Agent SDK documentation
• OpenAI platform documentation