# OpenAI automates performance engineering with Codex-powered factory that fixes regressions

> OpenAI's Performance Factory uses Codex agents to find and fix ChatGPT slowdowns, averaging 17 automated investigations daily and identifying root causes for over 600 cases in three months.

- Type: AI news
- Page: https://completeaitraining.com/news/openai-automates-performance-engineering-with-codex-powered/
- Published: 2026-11-20
- Updated: 2026-11-20
- Author: Jeroen Erne
- Jobs: Creatives, Customer support, Healthcare, Human resources, IT and development, Management, Operations, Product development, Science and research
- Reading time: 3 min

OpenAI has automated the entire process of finding and fixing performance regressions in ChatGPT, building a cloud-based system called the Performance Factory that runs on its Codex models. The system has tracked more than 1,700 investigations and identified root causes for over 600 cases in roughly three months - averaging 17 automated investigations each day - without requiring engineers to manually sift through metrics, traces, or logs.

The Performance Factory moves work through explicit stages: when an alert fires, an agent investigates the slowdown, records its findings, proposes a fix, and hands off to an engineer for review. After deployment, another agent checks whether performance returned to baseline. The factory pattern can coordinate agents across multiple workflows, with tasks passing between automated agents and human engineers as needed.

#### How the factory works

The system has four components. A task service holds configurable workflows defined through configuration files and Markdown prompts. A dispatch loop periodically checks which tasks are ready and assigns a Codex agent to each one. The agents run in the cloud via an internal API, with access to code repositories and observability tools like metrics, logs, traces, CPU profiles, and memory snapshots. Subscriptions wake agents when external events occur - a GitHub review comment, a failed CI check, or a completed deployment - so agents do not need to keep running while waiting.

For the Regression Workflow, the first workflow built inside the factory, an agent starts by determining which metrics degraded, when the change began, and which service requests were affected. It uses a skill called **perf-forensics** that gives Codex a playbook of available tools and investigation steps. The agent examines deployments and feature flag changes, reads relevant code, and checks whether the timing of changes aligns with the affected requests.

#### Adversarial review catches faulty diagnoses

Early agents sometimes stopped at the first plausible explanation, and some diagnoses were wrong. To address this, the team added an adversarial review step: a separate Codex session tries to disprove the proposed cause by rechecking assumptions and testing counterfactuals. In one case, an agent blamed failing ChatGPT conversations on errors in image downloads. The reviewer noticed the internal service responsible for locating files had already recovered while image requests were still failing - the diagnosis did not explain the continuing failures, so the reviewer reopened the investigation.

In a manual review of 100 root cause suggestions, 70% were graded as accurate, while 30% needed further refinement or did not fully establish causality. The team said different Codex turns handle investigation, adversarial review, and fix proposals to improve accuracy through checks and balances.

#### Fixes remain the harder problem

Once a diagnosis passes review, the workflow asks Codex to implement a fix and open a pull request. Each task includes an auto-generated summary with metrics time series, CPU flame graphs, and traces so engineers can quickly scan the evidence. Engineers review the PR, edit it directly, or write their own fix using the diagnosis. If they leave comments, the workflow wakes an agent to address them.

Getting fixes right has proven harder than finding causes. Agent-proposed changes can be more complicated than necessary or introduce new issues. The team added adversarial review for proposed fixes as well and is building an evaluation loop that compares agent proposals with what engineers ultimately merge. They are also exploring synthetic performance tests so agents can measure and revise fixes before asking for human review.

#### Lessons from running agent workflows in production

The team identified several practical takeaways. First, protect engineers' attention: raw Codex investigation output can be a wall of text, so the workflow links related incidents, groups them by metrics teams care about, and ranks estimated impact. Second, evaluate agent changes rigorously. The team backtests investigation skills against past production incidents and scores timing, impact size, affected requests, and how well agents use evidence to rank causes. They also test handoff summaries by checking whether an engineer unfamiliar with the system can understand what happened.

Third, turn recurring failures into self-repair. A broken profiler or missing permission can block investigations in ways that are hard to spot in final reports. The team runs a daily agent to scan recent runs for tool failures and access errors. When the same error blocks multiple investigations, the Performance Factory creates a task to fix it - and if the cause is clear, an agent proposes the fix for engineer review.

#### Why this matters for IT and development teams

The Performance Factory demonstrates a pattern that service teams can apply beyond performance engineering: define explicit task states, let agents advance work when ready, use separate reviewers to challenge conclusions, and bring engineers in only when judgment is required. The same machinery - configurable workflows, a dispatch loop, cloud execution, and event subscriptions - can run different workflows with different instructions. For teams building their own agentic systems, the progression from manual work to reusable skills to automated workflows offers a practical adoption path. Professionals looking to apply these patterns can explore [AI Agent Courses](https://completeaitraining.com/tag/automation/) and [AI for Developers Courses](https://completeaitraining.com/tag/it-and-development/) that cover agent workflow design and cloud-based automation.

---

This is the Markdown form of https://completeaitraining.com/news/openai-automates-performance-engineering-with-codex-powered/. The whole site is mapped at https://completeaitraining.com/llms.txt.
