← AUTOMATER NEWSROOM

Stripe Open-Sources a Benchmark That Grades Coding Agents on Real Integration Scenarios

Stripe open-sourced a benchmark grading coding agents on realistic payment, subscription and Connect integration scenarios, with and without its agent plugin.

Diagram showing the Stripe Integration Evals benchmark structure with three categories: Payments, Subscriptions and Connect, each with top scores, how-it-works methodology, and open-source availability on GitHub.Diagram showing the Stripe Integration Evals benchmark structure with three categories: Payments, Subscriptions and Connect, each with top scores, how-it-works methodology, and open-source availability on GitHub.
Stripe Integration Evals tests realistic scenarios across three Stripe product categories, grading 14 harness-model combinations with and without the Stripe agent plugin.

Stripe published an open-source benchmark on November 19 that grades coding-agent harness-and-model combinations on realistic Stripe integration scenarios, giving teams a concrete way to compare agent setups before handing them production API keys.

The Stripe Integration Evals benchmark tests agent pairs across three categories — one-time payments, subscriptions and Connect — and publishes scores on a 0–100 scale both with and without the Stripe agent plugin, which provides Stripe-authored skills and MCP-based tool access. Stripe says it will refresh the results periodically.

Screenshot of the Stripe Integration Evals benchmark page at stripe.dev/evals showing the Claude Code section with performance scores for claude-fable-5-2, claude-fable-5-1 and claude-opus-5-5 across payments, subscriptions and Connect scenarios, with and without the Stripe agent plugin.
The Stripe Integration Evals page at stripe.dev/evals showing integration success rates for Claude Code harness-and-model combinations. Source: Stripe. · Original source

How the benchmark works

The scenarios cover realistic integration tasks — building a hosted checkout flow for a $50 photography book, wiring up a $29/month SaaS subscription with a billing portal, or architecting a white-label Connect onboarding flow for a franchise network. A coding agent works through each case while a separate role-playing agent simulates a developer who answers follow-up questions. Stripe then grades each evaluation criterion as Pass or Fail independently.

Flow diagram showing the benchmark evaluation pipeline: coding agent processes realistic integration scenarios, role-playing agent provides context, results graded per criterion, then scores with and without the Stripe plugin are compared and published.Flow diagram showing the benchmark evaluation pipeline: coding agent processes realistic integration scenarios, role-playing agent provides context, results graded per criterion, then scores with and without the Stripe plugin are compared and published.
The Stripe Integration Evals pipeline: a coding agent works through each scenario, a role-playing agent simulates a developer, each criterion is independently graded Pass or Fail, and scores with and without the plugin are compared before publication.

Two identical evaluation runs are performed for every harness-model pair: one without the plugin and one with it, both with web search enabled. The plugin gives agents access to Stripe-authored documentation skills and MCP server tools, and the results make its impact visible.

What the numbers show

Claude Code, Codex and OpenCode all appear on the leaderboard across 14 harness-model combinations, and the plugin consistently lifts scores — sometimes dramatically.

Payments (one-time checkout): Codex with GPT-6 Astra leads at 96% with the plugin, up from 90% without it. Claude Code with Claude Fable 5.2 reaches 90% with the plugin. Claude Sonnet 5.5 sees one of the largest single-category plugin gains, jumping from 84% to 90%.

Subscriptions: This is the hardest category. Codex with GPT-6.1 Sol scores 84% with the plugin, the top mark, while GPT-6 Astra scores 83% and Claude Opus 5.5 reaches 78%. No combination cracks 85%. Subscription scenarios demand more complex billing logic — price allowlisting, idempotency keys, webhook deduplication — and agents consistently leave points on the table.

Connect: This is where the plugin delivers its largest impact. Claude Code with Claude Fable 5.2 hits 97% with the plugin, an 8-point jump. OpenCode paired with Google Gemini 4 Argon reaches 95% — competitive with the best Claude Code and Codex entries. Claude Opus 5.5 scores 94%.

Across all three categories, the plugin’s benefit is largest on Connect scenarios, where Stripe’s authored skills appear to steer agents toward correct architecture choices — Accounts v2, embedded onboarding, and proper charge-pattern and fee-model selection.

Why this matters

Existing agent benchmarks like Artificial Analysis’s Coding Agent Index measure general coding ability, but they say little about whether an agent knows a specific API well enough to integrate it correctly. Stripe’s benchmark answers that question directly for one of the world’s most widely used payment platforms.

Claude Fable 5.2, which recently completed 41% of real Upwork projects on Scale’s Remote Labor Index, performs well here too — scoring 90% on payments and 97% on Connect with the plugin. The benchmark’s criteria target common integration mistakes: passing payment_method_types when the prompt does not request specific methods, or hardcoding an API version older than 2026-04-22.dahlia.

The domain-specific approach parallels what Scope found when benchmarking database agent experiences — general coding scores do not predict how an agent handles a specific platform’s conventions. Stripe is taking the same idea to payments infrastructure.

The benchmark scenarios and evaluation harness are released as open source on GitHub under stripe/forge. Teams can adapt the scenarios to their own integration patterns, or run the harness against models and agent configurations that are not on the public leaderboard. The approach follows a growing pattern: HuggingFace’s harness-hub similarly trains adapters for specific harnesses, reflecting an industry shift toward measuring agents in the context where they will actually run.

The engineering team described the benchmark approach in a March 2026 blog post on the Stripe blog, and the results page at stripe.dev/evals was last updated November 19, 2026. The benchmark datasets and eval harness are hosted in the stripe/forge repository on GitHub.

Sources

  1. Stripe | Integration evals
  2. Stripe Blog: Online Payment Solutions Blog
  3. Stripe · GitHub