Harbor Agent Evals Now Run Inside Docker Sandboxes: MicroVM Isolation for Reproducible Coding-Agent Benchmarks

Diagram of three separate Docker Sandbox microVMs, each running a single Harbor eval task with its own kernel, fed by the Harbor eval framework and producing scores, traces and logs, with a phase one setup network policy allowing downloads and a phase two policy cutting internet access. — Original diagram of the Harbor plus Docker Sandboxes pipeline per Docker's announced architecture: each eval task runs in its own microVM with staged network policies; isolation shrinks the blast radius while the operator still controls credentials, tools and egress.

Agent benchmarks have a dirty secret: to test a coding agent, you let it install packages, edit files and run commands — thousands of times, on infrastructure you control. That code is untrusted by definition, and Docker’s answer, announced October 14, is to pair the open-source Harbor eval harness with Docker Sandboxes so every eval task runs in its own MicroVM.

Docker blog post titled Harbor plus Docker Sandboxes: Run Safe, Reproducible Agent Evals, showing the headline, the Posted Oct 14 2026 date, authors Srini Sekaran, Sohan Patil and John Mu, the Key Takeaways box, and the opening paragraphs about untrusted agent code and reward hacking. — Docker's announcement post as captured: headline, date, authors, key takeaways on running every Harbor eval task in Sandboxes, MicroVM isolation, network policies and local or cloud runs, plus the opening explanation of why evals need isolated environments. · Original source · Original source

Why isolation matters

Harbor supplies the framework half: it runs each task, lets the agent work, checks the result and collects scores and logs. What a benchmark also needs is a clean, isolated environment per task — and agents do not always play fair. Docker’s post calls out reward hacking: an agent can pass an eval by gaming the check rather than doing the work, say by tampering with tests or overwriting the score file. Docker’s design answer is a MicroVM per task with its own kernel, so agent work stays inside its own boundary. That sharply shrinks the blast radius, but it is not a blanket guarantee: credentials, tools or network access you deliberately permit inside the sandbox still need their own controls.

What you get

Per-task MicroVM isolation. Each task runs in its own microVM rather than a shared container, per Docker’s architecture.
Fair, reproducible evals. Network policies let a task download what it needs during setup, then cut the agent off from the internet, so every agent is tested under identical conditions.
Parallel runs. Docker reports testing Harbor with 64 concurrent workers on Cloud Sandboxes.
Local to cloud. The same tasks run locally for free or scale to Cloud Sandboxes unchanged.

Today’s limits: single-container CPU tasks; GPU and multi-container support are coming.

Try it

uv tool install "harbor[sbx]"

export SBX_DOCKER_USERNAME="your-docker-id"
export SBX_DOCKER_PAT="your-personal-access-token"

harbor run -t hello-world/hello-world -a oracle -e sbx

For local runs, install the Docker Sandboxes CLI, run sbx login, and add --ek target=local. Cloud runs need a Docker account with Cloud Sandboxes access and a personal access token, shown only once when created.

Original diagram summarizing LangChain's eval loop and where Harbor plus Docker Sandboxes fits: sandboxed task runs produce scored experiments, and failures flow back into the dataset for the next iteration.

The threat model the sandbox answers

Docker’s companion post, Top 5 AI Agent Security Risks, explains what the boundary is for: agents interpret instructions dynamically, choose tools at runtime and often run with your credentials, so one manipulated instruction becomes real damage. Its taxonomy:

Risk	Primary control
Prompt injection	Isolation, so a hijacked agent can’t reach what matters
Tool misuse and poisoning	Runtime-scoped tools from verified sources
Excessive autonomy	A sandbox boundary that makes auto-approval safer, not risk-free
Credential exposure	Short-lived credentials injected at runtime
Data exfiltration	Allow-listed egress and untrusted inter-agent handoffs

Docker also retells two incidents — a coding agent that allegedly wiped a developer’s home directory with an rm -rf ~/, and an agent with operator-level AWS credentials that reportedly caused a thirteen-hour outage. We could not independently verify these from the supplied sources, so treat the specifics as Docker’s accounts. The through-line stands either way: prompt hygiene alone is not a defense; you contain the blast radius and govern behavior. Isolation handles containment; it does not secure whatever credentials, tools or egress you allow inside. Our guide to securing AI agents covers the same split for production fleets.

Designing the eval, not just the sandbox

Isolation does not tell you what to measure. LangChain’s new guide, The Most Common Questions About Agent Evals, Answered, fills that gap:

LangChain blog post titled The Most Common Questions About Agent Evals, Answered, by Divya Santhanam, October 15 2026, showing the headline, author, date and the Key Takeaways box on evals, offline and online evals, and benchmarks. — LangChain's guide as captured: headline, author and date with the key takeaways on starting evals from traces, offline versus online evals, and benchmarks like Terminal-Bench run through Harbor and logged to LangSmith experiments. · Original source · Original source

Start from traces. Pick one behavior, build a ten-to-twenty-example dataset from real traces and failures, and run a baseline experiment.
Offline before you ship, online after. Offline evals compare versions against a fixed dataset; online evals score sampled production traces and feed failures back. Docker’s staged network policies fit the offline side: setup fetches dependencies, then the agent works with no internet.
Code evaluators first, LLM judges where code can’t. Judges need narrow questions, a pass-or-fail scale and periodic human checks. LangChain reports a cheap-judge experiment that matched human pass/fail labels for $0.34 total versus $28.17 for a frontier judge.
Check final state, not just final answers. For coding agents the reliable check is running the repo’s tests afterward, which needs a clean environment per run — the per-task setup Terminal-Bench-style benchmarks use. Harbor runs log to LangSmith experiments with LangChain’s integration; Harbor itself is open source, and the Sandboxes executor does not require LangSmith.

For a broader frame on benchmark numbers, see When SWE-bench Saturates and our guide to evals for AI agents.

The bottom line

The practical consequence: Harbor gains a Docker Sandboxes executor, so supported single-container CPU tasks run locally or in the cloud with per-task MicroVM isolation and staged network cutoffs. Pair Docker’s risk taxonomy and LangChain’s checklist with it, and you have an end-to-end pattern for actually trusting an agent benchmark’s score.