The Agent Memory Benchmark You Can Run on a Tuesday

The demo is always the same. A vendor’s agent forgets a fact in session one, remembers it in session two, and the room nods. Nobody asks how long the lookup took at the 99th percentile, what a thousand lookups cost once the injected tokens are counted, or what happened the one time it remembered something that was no longer true.

An agent memory benchmark you can run yourself answers those questions in an afternoon, and the two public benchmarks that matter this year agree on how: grade retrieval deterministically and grade tasks by execution, never by a model’s opinion of a transcript. For an acting agent that is the only honest grading, because a wrong recall is not a miss; it is an action taken on a false premise, and it can pass the vendor’s demo while quietly failing your repo.

This runbook turns that into a Tuesday: a frozen suite of your own tasks, three arms, four numbers, a harm-case review, and a decision rule you write down before the run so the results cannot talk you into anything. The public numbers appear once, in the section that names them. After that, only yours matter.

Two benchmarks removed the judge: LongMemCode (Apr 21, 2026) and Agent Memory Bench

LongMemCode, published Apr 21, 2026 by Aurelian Jibleanu at ArgosBrain, evaluates “the retrieval component of memory systems used by AI coding agents” in isolation. Its benchmark is MIT-licensed and its paper is CC BY 4.0. The suite covers 20 open-source corpora, 16 languages, and roughly 8,000 scenarios across nine task categories (completion, bug-fix, refactor, test generation, feature addition, API discovery, control-flow, configuration-surface, safety-net), “scored deterministically without an LLM judge.” It reports three numbers per adapter: weighted accuracy, P99 latency, and dollars per 1,000 queries. Its two reference adapters bracket the field. A grep-backed text-search floor scores 6.3–54.4% weighted accuracy at 1.14–38.77 ms P99; a structural reference adapter scores 99.2–100% at 0.01–0.82 ms. Both cost $0 per 1,000 queries.

Screenshot: ArgosBrain, “LongMemCode: A Deterministic Benchmark for Code-Memory in AI Agents” (Apr 21, 2026), captured Sep 13, 2026.

Agent Memory Bench takes the other half, in its own words: “Existing memory benchmarks ask a model questions about synthetic conversations and let an LLM judge the answers. This one gives a real agent real work in a real repository, where success depends on something learned in earlier sessions, and grades the artifact by execution: tests pass or they do not. No judge anywhere in the primary endpoint.” Memory integrations are pluggable, and the leaderboard reports scores alongside token counts, putting task success and memory cost on the same page.

Screenshot: Hugging Face Spaces, “agent-memory-bench” overview, captured Sep 13, 2026.

Between them they make one claim: retrieval quality can be measured deterministically, task success can be measured by execution, and neither needs a judge. The original LongMemEval used synthetic chat histories. Its successor, LongMemEval-V2, instead tests web agents on curated questions whose histories can span 500 trajectories and 115 million tokens. Neither benchmark measures whether a coding agent finishes a migration in your repository, which is the gap this bakeoff closes.

What an agent memory benchmark must grade: four numbers, no judge

Before the suite, the numbers. Write them into the scoring sheet with thresholds you commit to before the run. The thresholds below are illustrative starting points, not industry figures.

Metric	Definition	How you get it	Illustrative threshold
Task success delta	Pass rate with memory minus pass rate without, on the same frozen suite, execution-graded	Run each task’s tests; it passes or it does not	Adopt only at ≥ +10 points
P99 lookup latency	Wall-clock from a memory lookup call to the answer landing in context, 99th percentile across every lookup in the run	Timestamp each lookup in the harness log	≤ 300 ms interactive; ≤ 2 s batch
Cost per 1,000 lookups	Vendor fee, plus the tokens each lookup injects into context, plus the write-side tokens the memory spent to exist	Meter delta between arms, divided by lookup count	Inside the run’s memory budget
Harm cases	Wrong recall, stale recall, cross-project leak; each counted per task and each failing the task regardless of tests	Planted fixtures plus a transcript read	Leaks: 0. Stale: ≤ 1 per 30 tasks

P99, not the average, and not P95. LongMemCode made P99 its headline for the reason your harness should: an agent loop that makes 300 lookups in a task hits the 99th percentile three times per task, and a lookup that stalls for four seconds stalls the whole worker while the meter runs. P95 hides one such stall in every twenty.

Cost is the number most sheets get wrong, because the vendor’s per-query price is the smallest part of it. A memory that injects 3,000 tokens of “relevant context” into every turn is billed at your model’s input rate on every turn, whether or not the recall helped. The mechanics of that tax are already written up in memory that burns quota; this sheet only insists the injected tokens go on the line.

LongMemCode’s reference adapters make one more point worth a chart: retrieval quality is a design property of the memory layer, and a deterministic benchmark separates the designs by an order of magnitude.

Source: ArgosBrain, LongMemCode reference adapter results (Apr 21, 2026). Bands span the 20 corpora. Latency ranges are P99; both adapters report $0 per 1,000 queries.

Read it as a floor and a ceiling rather than a product ranking. The text-search floor is roughly what “grep the transcripts” gets you; the structural ceiling is what an index that understands the code’s shape gets you. Where a vendor’s memory lands between them, on your corpus, is what Tuesday finds out.

Build the agent memory benchmark suite: freeze thirty tasks on Monday night

Step 1: pick the tasks from your repos, not the vendor’s. Twenty to forty tasks, each with the Agent Memory Bench shape: success depends on something a prior session learned. A test that only passes if the agent knows which of two similar modules is live. A refactor whose naming convention was decided in a session last month. A bug whose fix needs the workaround the team agreed on and never documented. Each task needs an executable pass condition (a test, a build, a linter rule); a task without one is not in the suite.

Step 2: write the prior sessions as fixtures. For each task, the “earlier session” is a transcript file you author: the decision, the workaround, the convention, realistic and slightly messy. Then plant the harm cases. In roughly one task in five, the fixture states a fact that was true and is now stale (the config moved). In one task in ten, a fixture from a different, fictional project mentions a similar file.

Step 3: freeze it. Hash the manifest, commit the repo snapshot the tasks run against, pin the model and harness version, and write the thresholds from the table into DECISION.md. Nothing changes after Monday night. A suite you edit after seeing arm B’s results is a demo again, and the point is to stop buying on demos. The threshold-margin discipline in the AWS AgentCore evaluation pattern (aws.amazon.com, Sep 8, 2026) is the CI-shaped version of the same rule, and its authors put the stakes plainly: “Without automated evaluation, agent quality is subjective.”

suite/
  manifest.json        # task ids, fixture ids, pass commands, sha256 of everything below
  tasks/t01..t30/      # prompt.md + pass.sh (exit 0 = pass)
  fixtures/s01..s30/   # prior-session transcripts, incl. planted stale and cross-project cases
  DECISION.md          # thresholds and the adopt / keep / re-run rule, dated

Step 4: define the three arms. Arm A, no memory: the agent gets the task and the repo, nothing else. Arm B, the candidate memory product, loaded with the fixtures through its own ingestion path. Arm C, local transcripts: the same fixtures on disk, searchable, injected only when the agent asks. Same model, same harness, same permission mode, same repo snapshot, same seed where the harness allows one. Arm A is the control. Arm C is the baseline the product has to beat, because it is what you already have if you keep transcripts at all; cross-CLI memory that survives the session covers what to keep and where.

Step 5: run, wiping between arms. All thirty tasks per arm, timestamps on every lookup, meter readings before and after each arm. Between arms, wipe the memory store, the repo worktree, and the harness’s own session state. Memory that leaks from arm B into arm C is the most common way a bakeoff produces a false result; the fix is a fresh worktree and a fresh store per arm, verified with a query before the first task, never a “clear history” button taken on faith.

Freeze, run, score, read, decide. The loop closes on the same frozen suite, so next quarter’s numbers are comparable.

Score the sheet and apply the rule you wrote on Monday

The sheet has three columns and five rows, and the fifth row is the verdict.

	Arm A: no memory	Arm B: vendor memory	Arm C: local transcripts
Task success (of 30)
P99 lookup latency (ms)	n/a
Cost per 1,000 lookups (fee + injected tokens)	n/a
Harm cases (wrong / stale / leak)
Verdict against DECISION.md	control		baseline

The decision rule, as written in DECISION.md before the run:

1. Adopt B only if its success delta over A clears the threshold, its delta over C is positive, P99 is inside budget, cost is inside budget, and leaks are zero.
2. Keep C if B beats A but not C, or if B’s only advantage is convenience. Transcripts you own are the cheaper win.
3. Re-run once if B misses one number by a small margin, after a configuration change the vendor can name in writing. A third attempt is a purchase decision made by fatigue.
4. Reject on any leak, whatever the other numbers say.

A memory layer that scores well on retrieval but does not move task success on your suite is a fine index of the wrong things. A layer that moves task success and injects 4,000 tokens per turn to do it may lose to arm C on cost by a factor you only see because the sheet made you count the tokens.

Harm cases: wrong, stale, and leaked recall are failed tasks

The fourth metric needs its own hour, because the tests will not catch all of it. A stale recall can still pass if the agent happened to check the file before trusting the memory. It is still a harm case: the memory asserted something false, and next time the agent may not check.

Read the transcripts for the planted tasks and record, per arm, what the memory returned and what the agent did with it: whether the memory surfaced the stale fact as current, whether the agent acted on it before verifying, and whether anything from the fictional project appeared in context. A yes on the third is a leak, and leaks end the evaluation. A store that cannot keep two projects apart across thirty tasks will not keep two clients apart across three hundred.

Harm cases are also where vendor-hosted memory changes what evidence you can keep. If the store lives in the vendor’s workspace, your only record of what it returned is the transcript on your side; export it before the vendor UI rotates, which is the whole argument of evidence when the harness is vendor-hosted.

Local transcripts versus vendor memory: what the bakeoff settles

Managed coordinators have made “memory” a cloud noun again. Cursor’s Sep 10, 2026 Projects post (cursor.com/blog/projects) says a Project’s shared context “grows with the Project, making the coordinator more effective over time”; the OpenAI Agents API overview (developers.openai.com, public beta Sep 10, 2026) says OpenAI manages “sessions, orchestration, context compaction, and recovery.” Both are real conveniences, and both are memory you cannot benchmark unless you bring the suite to them, which is what arms B and C do. The boundary itself is already drawn: what belongs in a vendor store, in local transcripts, or in neither is the protocol in the cross-CLI memory piece linked above, and the cost of replaying it is the quota piece. This runbook adds only the test that says, for your repo, whether the vendor store earns its arm.

What breaks a memory bakeoff, and the signal that tells you

The demo-suite trap. Signal: the vendor offers to “help set up the evaluation.” Cause: their tasks are the tasks their memory was built on. Fix: your repos, your fixtures, hashed before anyone from the vendor sees them.

Leakage between arms. Signal: arm C scores suspiciously close to arm B on the planted tasks. Cause: the store or the worktree was not wiped. Fix: fresh worktree and fresh store per arm; prove the store is empty with a query before the first task.

The judge creeps back in. Signal: someone proposes an LLM grader for “partial credit.” Cause: execution grading is harsh and the numbers look bad. Fix: harsh is the feature. AWS’s own guidance warns that judge variance forces padded thresholds; a pass condition does not vary.

The P50 slide. Signal: the vendor’s latency chart shows a median. Cause: medians hide the stall. Fix: your timestamps, your P99, or no latency claim at all.

Environment failures blamed on memory. Signal: arm A fails tasks that have nothing to do with recall. Cause: the environment, not the memory. GitTaskBench attributed 65.04% of failures to environment setup and dependencies (arxiv.org/abs/2508.18993). Fix: run the environment readiness preflight on arm A first; a task that fails there for an environment reason leaves the suite for all arms.

The cost line without injected tokens. Signal: cost per 1,000 lookups is a small round number. Cause: someone copied the price list. Fix: meter deltas per arm, or the number is not on the sheet.

The operating layer runs the bakeoff; the prompt cannot

None of this is a smarter prompt. A frozen suite, three isolated arms, lookup timestamps, meter deltas, and transcripts you can read afterwards are operating-layer infrastructure, the same layer that already has to know which sessions ran, what they cost, and what they touched. The bakeoff is one afternoon of that layer doing its job on purpose, and its output feeds the regression gate you should be running on every agent change anyway (agent evaluation in CI).

That is the fleet argument of agentic ops and the desk argument of the multi-agent command center. Agents act, so memory is a component with a latency budget and a bill. Measure it like one, on a Tuesday, with the thresholds written down on Monday.

FAQ: agent memory benchmarks

What is an agent memory benchmark?

An agent memory benchmark measures whether a memory layer helps an agent finish work it could not finish without prior context, and at what cost. The current public ones, LongMemCode and Agent Memory Bench, grade retrieval deterministically and tasks by execution, with no LLM judge in the primary metric.

How do you measure memory for a coding agent?

Freeze a suite of tasks from your own repos where success depends on an earlier session, run them with no memory, with the candidate memory, and with plain local transcripts, then compare execution-graded pass rates, P99 lookup latency, cost per 1,000 lookups including injected tokens, and counted harm cases.

Does memory make AI coding agents better?

Sometimes, on some suites, at a price. LongMemCode’s grep reference spans 6.3–54.4% weighted accuracy while its structural reference spans 99.2–100%, and only an execution-graded run on your tasks shows whether that difference turns into merged work. Stale and leaked recalls can make an acting agent worse than one that remembers nothing.

Sources

• ArgosBrain — LongMemCode: A Deterministic Benchmark for Code-Memory in AI Agents (Aurelian Jibleanu, Apr 21, 2026)
• Hugging Face Spaces — Agent Memory Bench: memory for coding agents, graded by execution
• arXiv — “LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues” (2605.12493)
• AWS Machine Learning Blog — Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions (Sep 8, 2026)
• arXiv — GitTaskBench (2508.18993)
• Cursor — Introducing Projects (Sep 10, 2026)
• OpenAI — Agents API overview (public beta, Sep 10, 2026)