Microsoft Open-Sources Scope: A Self-Hosted Way to Test Whether Coding Agents Actually Use Your SDK, Docs or MCP Server
Microsoft's MIT-licensed Scope platform runs coding agents through real tasks against your own SDK, docs or MCP server, then grades the results for fixable gaps.
AI coding agents increasingly act as users of your product. They pick an SDK, follow your docs, call your CLI or your Model Context Protocol (MCP) server — and an experience that works fine for a human developer can still leave an agent calling the wrong API, missing a required step, or stalling before a task is done. On October 19, 2026, Microsoft open-sourced Scope, a Kubernetes-deployable, MIT-licensed platform built specifically to catch that gap — not by benchmarking which model is smartest, but by grading whether real agents can correctly use the thing your team ships.
What Scope actually does
Scope drives existing coding agents — Microsoft names GitHub Copilot, Claude Code and VS Code with Copilot — through a task you define against your own SDK, docs, CLI or MCP server, then scores the outcome. Success criteria are modeled as a directed acyclic graph rather than a flat checklist, so a check can depend on a prerequisite: a runtime test only runs, for example, if the build already succeeded. A “Judge” service uses an LLM to evaluate the agent’s output against that graph.
It’s FOSS’s walkthrough of the release gives a concrete, if illustrative, shape for that graph: ask an agent for a Node.js server, then check for a GET /health route that returns JSON, gated behind the build actually succeeding. Every run keeps live logs, generated files, workspace snapshots and the full sequence of tool calls, so a failed criterion traces back to the exact step that caused it — not just a final pass/fail.
Configurable personas let teams vary the simulated user’s perspective, and versioned profiles capture everything about a run — the worker, model, agent version, skills, MCP servers and extensions — so one piece (a prompt, an SDK sample, a doc page) can change while the rest of the setup stays fixed. Because agent behavior varies between attempts, Microsoft recommends repeated runs to separate a recurring failure from a one-off, reviewed through a Portal or pulled via CLI and REST API for CI pipelines.
The proof point: a real before-and-after number
Microsoft’s launch post leans on a concrete example rather than a demo. Jay Gordon, Senior Program Manager on Azure Cosmos DB, is quoted directly: “Using SCOPE, we compared bare models with our Agent Kit across eight end-to-end Azure Cosmos DB scenarios in five languages. The evaluations led to 20 Agent Kit issues and 27 documentation issues resolved. After one targeted Agent Kit update, hierarchical partition-key adoption rose from 47% to 100%, while OFFSET/LIMIT usage fell from 47% to 0%.” Microsoft frames those numbers as describing the Cosmos DB scenarios evaluated, not a general ranking of agents — but they are a specific, falsifiable before-and-after, not a changelog bullet.
Separately, Azure Cosmos DB’s own engineering blog describes a related but distinct closed-loop effort: five application scenarios run through GitHub Copilot against a live Cosmos DB Emulator in CI, repeated five-plus times per language for statistical confidence, which pushed the public Agent Kit skill to general availability with 120+ rules and over 200 automated test iterations. That post doesn’t name Scope directly, but it shows the same team investing heavily and independently in watching agents fail against its product repeatedly, then encoding the fix as a rule — the exact discipline Scope is built to generalize.
Where this is meant to be used
Microsoft frames three concrete investigation targets for teams adopting Scope:
| Area | What you check | What you do with a failure |
|---|---|---|
| Documentation | Do agents repeatedly miss an instruction or assume something wrong? | Rewrite the page, rerun the scenario, confirm the failure rate drops |
| SDK discovery and adoption | Does the agent pick your SDK, initialize it correctly, and call the right APIs? | Compare a baseline against revised samples or an agent skill |
| Tooling and MCP servers | Where does the agent stall on a CLI or MCP tool call? | Sharpen tool descriptions or error messages, then retest |
For a team shipping an SDK, a CLI, a docs site or an MCP server that agents are expected to call correctly, this turns “agents keep doing the wrong thing” from an anecdote collected through confused support tickets into a repeatable, versioned experiment you can rerun every time a doc, sample or tool description changes.
The catch: there’s no hosted version
Scope is software you deploy yourself, not a managed service. A local stack needs Docker with Compose v2, Node 22, pnpm, mkcert and the GitHub CLI; the default worker requires an active GitHub Copilot entitlement, and the Judge needs credentials for a model provider. Teams wanting a shared deployment run it on Kubernetes via a Helm chart. The Scope GitHub repository — MIT-licensed, 39 stars and 25 forks as of this week — is labeled a research preview, and contributing requires signing Microsoft’s Contributor License Agreement.
Microsoft is explicit about the limits of the method, too: repeatable configurations don’t guarantee identical agent behavior, and the LLM-based Judge can make mistakes. The company’s own guidance is to pair Scope with representative tasks, repeated runs and human review of the evidence before treating any result as a product decision — a caveat worth taking literally rather than as boilerplate, given that the entire pitch rests on trusting what the Judge reports.
Why it matters now
Most agent evaluation to date has graded models against public benchmarks — which model is smartest on a shared leaderboard. Scope flips the direction: it grades whether an agent, running against your actual product, discovers and uses what you ship. That’s a narrower and more useful question for anyone maintaining developer-facing surfaces agents are expected to operate — and Microsoft’s own dogfooding, imperfect as the attribution trail is, shows it surfacing fixable, specific bugs rather than vague advice to “test more.” Teams already running evals for AI agents or building MCP servers now have a free, versioned way to point that discipline at their own docs and tooling instead of only at the agent’s reasoning — one more piece of the broader shift toward testing agentic software as its own engineering practice.
