← AUTOMATER NEWSROOM

OpenAI's Performance Factory: How Codex Agents Investigate and Fix ChatGPT Speed Regressions at Scale

OpenAI's Codex agents autonomously investigate and fix ChatGPT speed regressions at scale: 1,700+ investigations and 600+ root causes in three months.

A flowchart-style diagram showing the Performance Factory's regression workflow pipeline: alert fires, Codex agent investigates, adversarial review challenges the diagnosis, agent proposes a fix, engineer reviews and decides, agent verifies recovery, and task is resolved. A dark banner at the bottom shows key metrics: 1,700+ investigations, 600+ root caused, 17 per day.A flowchart-style diagram showing the Performance Factory's regression workflow pipeline: alert fires, Codex agent investigates, adversarial review challenges the diagnosis, agent proposes a fix, engineer reviews and decides, agent verifies recovery, and task is resolved. A dark banner at the bottom shows key metrics: 1,700+ investigations, 600+ root caused, 17 per day.
The Regression Workflow pipeline in OpenAI's Performance Factory, from alert to resolution, with key throughput metrics.

OpenAI blog post titled 'How OpenAI's Performance Factory keeps ChatGPT fast' showing the November 19, 2026 publication date, Engineering category tag, and the introductory description of the Codex-powered system that automates finding and fixing performance issues.
OpenAI's November 19, 2026 engineering post detailing the Performance Factory, an autonomous Codex-powered system for investigating and resolving ChatGPT performance regressions. · Original source

OpenAI published an unusually detailed engineering case study on November 19, 2026, on its engineering blog: the Performance Factory, an internal system where Codex agents autonomously investigate ChatGPT performance regressions, propose fixes, and verify recoveries — with engineers stepping in only for the merge decision. The post, listed on both OpenAI News and the engineering index, describes a system that over roughly three months tracked more than 1,700 investigations and root-caused over 600 cases, averaging 17 automated investigations per day.

This is one of the most concrete production-scale accounts of coding agents operating as a “software factory” — a pattern where multiple specialized agent turns pass work through explicit states, with adversarial reviewers challenging conclusions before a human makes the final call. It is directly relevant to any team running Codex, Claude Code, or similar agents in CI and operations pipelines.

From Manual Investigations to an Agent Factory

The post, authored by OpenAI engineers Okay Zed and Cyrus Forbes, traces a six-stage evolution that many teams will recognize:

  1. Pre-Codex: Engineers manually examined metrics, logs, traces, and profilers — work that was time-consuming and required deep system knowledge.
  2. Codex as a partner: Engineers could ask “Why did this get slower?” and Codex would search codebases, find metrics, and follow traces across services, but humans still steered the process.
  3. Reusable skills: The team captured their investigation playbook in a shared skill called perf-forensics, telling Codex which tools to use and when. They tested and refined it using evals based on past regressions.
  4. The human proxy: Engineers became a copy-paste relay between alerts and Codex — an alert would fire, someone would paste it into Codex with the right skill, then paste findings back into Slack.
  5. Automated investigations: Investigations moved to the cloud so alerts could trigger Codex sessions automatically, but the automation stopped at posting findings in Slack.
  6. The factory: The Performance Factory now runs end-to-end: it tracks what each active task needs next, dispatches agents when work is ready, and passes tasks between agents and engineers through state transitions.

A diagram showing six stages of evolution from manual pre-Codex performance work to the fully automated Performance Factory, arranged in a stepped path with color-coded boxes.A diagram showing six stages of evolution from manual pre-Codex performance work to the fully automated Performance Factory, arranged in a stepped path with color-coded boxes.
OpenAI's six-stage progression from manual performance investigations to the autonomous Performance Factory, as described in the November 2026 engineering post.

Four Components That Make It Work

The Performance Factory is built from four parts:

  • A task service with configurable workflows where engineers define state machines and Markdown prompts for agents at each step.
  • A dispatch loop that periodically checks which tasks are ready and assigns a Codex turn.
  • Cloud Codex execution via an internal API, running under a service identity with scoped permissions, with access to metrics, logs, traces, CPU profiles, and memory snapshots.
  • Subscriptions that wake agents when something changes — a GitHub review comment, a failed CI check, or a completed deployment — so agents don’t waste tokens polling.

Adversarial Review: A Second Agent Checks the First

One of the most instructive design decisions is the adversarial review step. Early on, agents sometimes stopped at the first plausible explanation, and some diagnoses were wrong. To counter this, a separate Codex session tries to disprove the proposed diagnosis, rechecking assumptions and testing counterfactuals.

In one cited example, an agent traced failing ChatGPT conversations to errors in image downloads and marked the cause as found. The reviewer noticed that the internal file-locating service had already recovered while image requests kept failing — the diagnosis didn’t explain the continuing failures, so the reviewer reopened the investigation.

In a manual review of 100 root cause suggestions, 70% were graded as accurate, while 30% needed further refinement. Different Codex turns investigate, challenge, and propose fixes to improve accuracy through checks and balances.

Fixing Is Harder Than Finding

Getting fixes right has proven harder than diagnosing problems. Agent-proposed changes can be more complex than necessary or introduce new issues. An adversarial review of proposed fixes was added to catch bugs and unintended consequences. Some fixes require architectural decisions — like adding a cache that improves performance but risks stale data — where an engineer’s judgment remains essential.

The team is building an evaluation loop by comparing agent-proposed fixes with what engineers ultimately merge, and exploring synthetic performance tests so agents can measure and revise fixes before human review. They are also investigating whether low-risk changes could eventually ship without a human in the loop, as detailed in the full engineering post.

Lessons for Teams Building Agent Workflows

The post offers practical takeaways for any team building agentic workflows in production:

  • Protect engineers’ attention: Raw Codex output can be a wall of text. The workflow now links related incidents, groups them by metrics teams care about, and ranks estimated impact so engineers spend attention on decisions, not reconstructing the agent’s work.
  • Evaluate changes to your agents: Updating a skill or upgrading the model can make agents better at some cases and worse at others. OpenAI backtests against past production incidents and separately evaluates the usefulness of agent-to-engineer summaries.
  • Turn recurring failures into self-repair: A daily agent scans recent runs for tool failures and access errors. When the same tool error blocks multiple investigations, the Performance Factory creates a task to fix itself — then uses evals to check whether agents can now get past the step that previously blocked them.

Why This Matters

The Performance Factory is significant not just for its scale — 17 investigations per day, 600 root causes in a quarter — but because it represents a concrete, running instance of what the “agent factory” pattern looks like in production. Tasks move through explicit states. Different agents handle different stages. Separate reviewers challenge conclusions. Engineers step in only where judgment is required.

This pattern — already visible in emerging external tools like Devin Bots and LangChain’s proactive-agent design patterns — is likely to become the default operating model for service reliability in organizations running large-scale AI-powered products. OpenAI’s post makes clear that the bottleneck today is waiting for an engineer to review and merge the fix, and they are already exploring whether agents can handle that step themselves.

For teams adopting Codex’s sandbox permission profiles or building custom agent workflows, the Performance Factory’s architecture — task service, dispatch loop, cloud execution, and event subscriptions — offers a reference model for turning ad-hoc agent use into a production system that runs 17 investigations a day while protecting the one resource that doesn’t scale: engineer attention. The full case study is available on OpenAI’s engineering blog, listed alongside recent posts on OpenAI News and the engineering index.

Sources

  1. How OpenAI’s Performance Factory keeps ChatGPT fast | OpenAI
  2. OpenAI News | OpenAI
  3. OpenAI News | OpenAI