Stop Trusting the Mermaid

Consider a representative review failure. An agent opens a large pull request with a tidy Mermaid diagram: requests flow through a validation service before writes reach the database. The actual diff gives one worker a direct write path. The diagram is not evidence of the architecture; it is a claim about the architecture.

That distinction applies even when the diagram is beautiful, source-derived, and mostly correct. A generated figure compresses a repository into selected nodes and edges. Compression is its job. Code review has to examine what the compression omitted.

The same rule applies to fleet views. A companion layer across harnesses can make supported sessions easier to find, and stall signals can direct attention to a quiet worker. Those surfaces help locate evidence; they do not establish that a diagram edge, code path, or completion claim is correct.

A diagram is a derived artifact

An agent audit trail is not one magic file. It is a linked set of primary and derived artifacts:

1. Repository state — base revision, changed files, untracked files, and final diff.
2. Executed checks — exact commands, exit codes, logs, and when they ran relative to the last edit.
3. Session record — prompts, tool calls, outputs, permission decisions, and summaries the harness persisted.
4. External state — issue, deployment, service, database, or browser evidence that may not be inside the session.
5. Derived communication — PR description, final summary, Mermaid graph, screenshot, or generated architecture page.

The list is not a guarantee that the first item is always truthful or the fifth is always wrong. It is an evidence hierarchy. Derived artifacts should point back to the observations that support them.

Annotate each important observation with four properties: scope (which files, service, or environment it covers), freshness (whether later edits could invalidate it), provenance (who or what produced it), and independence (whether it comes from a separate check or the same agent context). A current, narrowly scoped test may support one diagram edge without validating the rest of the graph. Three polished summaries from one session are still one evidence lineage. These labels keep “evidence exists” from becoming the broader and unsupported claim that the architecture is verified.

Archify illustrates the constructive version of generated architecture. Its project describes producing self-contained HTML documentation and emphasizes verifiable architecture rather than an opaque image. That is a good design goal. Verification still requires a reviewer to trace important nodes and edges to source evidence.

Mermaid is not the problem. Text-based diagrams are versionable, reviewable, and easy to regenerate. The mistake is assigning a diagram more authority than its inputs.

Why polished diagrams attract too much trust

They share the author’s blind spots

If the same model edits the worker and summarizes the result, a mistaken belief can appear consistently in code comments, PR prose, and the diagram. Agreement among artifacts generated from one context is not independent confirmation.

They omit the path that failed

The transcript may show a permission denial, abandoned implementation, retry, or test failure that changed the final approach. A clean graph usually renders the destination, not the wrong turns. Those wrong turns often explain the highest-risk lines in the diff.

Visual confidence is not semantic confidence

A wrong edge renders with the same typography, spacing, and arrowhead as a correct one. Design quality helps comprehension; it carries no truth signal.

Review order creates anchoring

Read the diagram first and the diff becomes an exercise in finding the promised boxes. Read the diff and checks first and the diagram becomes what it should be: a compact communication layer to verify against known behavior.

The 15-minute agent-PR ritual

For a normal pull request, use a fixed order. Large, security-sensitive, or production-critical changes need a deeper review; this ritual is the minimum, not a ceiling.

1. Fix the scope and revision

Record the base and head commits. Check the final working tree if the agent operated locally. List changed and untracked files before reading the PR narrative.

git status --short
git diff --stat <base>...HEAD
git diff --check <base>...HEAD
git log --oneline --decorate <base>..HEAD

This prevents a transcript from describing one checkout while the review examines another.

2. Inspect what actually ran

Find the final test, typecheck, lint, build, or migration command. Capture its exit code and timestamp. Then ask whether any relevant file changed afterward.

“Tests pass” is incomplete without:

• the exact command;
• the selected package or test scope;
• the revision or working-tree state;
• skipped suites and exclusions; and
• the actual result.

Run the smallest trustworthy existing check again when the evidence is stale or the change is risky. Re-running a check in controlled current state is reproduction. It is stronger than asking the original agent to repeat its summary.

3. Search the transcript for decision points

Do not read a long JSONL file linearly unless the risk warrants it. Search for failures, denials, retries, changes of plan, and scope negotiations:

rg -i 'error|failed|denied|retry|instead|workaround|skip|cannot' \
  ~/.claude/projects/<project>/<session-id>.jsonl

Claude Code currently documents local JSONL transcripts under ~/.claude/projects/<project>/. Other harnesses use their own locations and formats. Treat the command as a Claude example, not a universal path.

Read each hit with surrounding events. A failed authorization test followed by a weakened assertion is more important than twenty successful file reads. Also note missing evidence: persistence can be disabled, retention can expire, subprocess output can be truncated, and human terminal actions may never enter the transcript.

The local Automater Library and beta Session Explorer can reduce cross-provider archaeology by normalizing supported records. Automater Lite is free on automater.ai; Pro is $29/year.

4. Read the diff before the narration

Start with interfaces, schemas, migrations, permissions, tests, and error handling. Then inspect implementation. Leave the PR description and generated diagram until after the behavioral change is understood.

For every important diagram edge, find its code counterpart:

Diagram claim	Code evidence to inspect
“All writes pass through validation”	Every write call site, data-access boundary, and test for bypass paths
“Worker retries through the queue”	Retry owner, backoff, idempotency key, dead-letter behavior
“Service is read-only”	Credentials, route methods, tool permissions, database grants
“Data stays local”	Network clients, telemetry, upload paths, configuration defaults

If the edge cannot be traced, mark it unverified. Do not repair the diagram until the code question is settled.

Worked example: “all writes pass through validation”

Take the opening diagram claim and turn it into a review query. Start at the table or repository interface that performs the write. Search for every constructor, client, repository method, SQL statement, or SDK call that can reach it. Then work backward toward the validator.

The investigation may produce four paths:

1. the public API validates and writes;
2. a background worker consumes an already-validated message and writes;
3. an administrative repair command validates through a different function; and
4. a test utility writes directly.

The diagram’s single arrow may be an acceptable simplification if each production path has an equivalent enforced check. It is wrong if the background worker accepts an untrusted payload or the repair command is available in normal operation without the validation contract.

Now inspect tests. A unit test proving the validator rejects one bad payload does not prove all writes pass through it. Look for an integration test at the boundary and a negative test for the bypass path. Confirm the test ran after the final worker change.

Finally, use the transcript to understand intent. Did the agent notice the direct write and classify it as safe? Did a failing test cause the validation call to be removed? Did a tool output truncate before the relevant path was inspected? The transcript can explain how the mismatch arose, but the source and current test establish whether the mismatch still exists.

The review outcome should be precise:

• Verified: every production write path enforces the invariant, with current test evidence.
• Qualified: one trusted administrative path bypasses the shared service but applies an equivalent check; update the diagram to show or annotate it.
• Rejected: an untrusted or ordinary production path writes without validation.
• Unknown: the available source, runtime, or transcript evidence is incomplete; do not merge the claim as fact.

This vocabulary keeps a visual simplification from turning into a false binary. The diagram does not have to render every private helper. It does have to preserve the architecture’s consequential boundaries.

5. Reproduce the disputed behavior

When transcript and diff do not settle a question, reproduce the narrow behavior in a clean checkout or controlled test environment. Use known inputs and observe current outputs. If production state is involved, use the team’s approved diagnostic path rather than letting an agent improvise against live systems.

Resuming the original conversation can recover rationale and ask follow-up questions. It does not replay history deterministically. The filesystem and dependencies are now different, and the model may answer differently. Fleet replay means inspecting the recorded chronology; reproduction means executing a defined check again.

Only after those five steps should the diagram influence documentation or onboarding.

How to use generated diagrams well

Generated diagrams are excellent for:

• explaining a verified change to a wider audience;
• onboarding readers to a large repository;
• making missing boundaries visible for discussion;
• versioning architecture descriptions alongside code; and
• linking important nodes to source files and checks.

Apply four rules:

1. Generate from a named revision. Put the commit or release in the figure metadata.
2. Link claims to evidence. A service node should lead to source, configuration, or an architecture decision record.
3. Label inferred edges. Static analysis, naming inference, and model interpretation should not look like runtime observation.
4. Regenerate and diff. Treat diagram changes as reviewable documentation, not hand-maintained decoration.

Archify’s emphasis on verifiability points in this direction. Mermaid’s source text makes diffs practical. The reviewer’s responsibility is to preserve the chain from edge to evidence.

Session tools are evidence surfaces, not truth machines

Apache Maka uses an append-only event log for model messages, tool calls, results, permission decisions, and termination. Agenttrail provides a local visual canvas for plans, tool activity, and file changes from supported coding assistants. These designs can make chronology easier to inspect.

They cannot record what never passed through their capture boundary. A human hotfix, external service mutation, truncated subprocess, or disabled transcript can remain outside the trail. Audit tools improve visibility; they do not abolish missing data.

The review card

1. Pin base and head revisions.
2. Inspect final commands and their timing.
3. Read transcript decision points and gaps.
4. Review diff and tests before narrative.
5. Reproduce unresolved behavior in controlled current state.
6. Read or regenerate the diagram last.

The memorable line is still the title: stop trusting the Mermaid. The operational meaning is more precise: trust no generated summary without following its important claims back to source and current behavior.

FAQ

Are AI-generated diagrams reliable for code review?

They can be accurate, but they are derived claims. Use them to navigate after inspecting the revision, diff, tests, and session evidence. Trace security and data-flow edges back to code.

What counts as an agent audit trail?

Repository state, executed checks, persisted session events, and relevant external-system evidence together form the trail. PR summaries and diagrams are derived artifacts built from that evidence.

Is resuming an agent session the same as replaying it?

No. Resume continues a saved conversation in current state. Replay, in this article, means inspecting recorded events in order. Reproduction means re-running a defined command or scenario to observe current behavior.

Sources

• Archify — self-contained, source-derived architecture documentation with an explicit verifiability goal
• Mermaid — text-based diagram syntax and rendering
• Claude Code sessions — transcript format, location, persistence, and resume semantics
• Apache Maka — append-only local agent event logging
• Agenttrail — local visual inspection of supported coding-agent activity
• Automater — current Library, Session Explorer, privacy, and pricing claims