A Practical Approach to Verifying Code at Scale
As autonomous collaborative coding systems proliferate, the volume of produced code quickly exceeds the limits of thorough human oversight. As this gap widens, so does the risk that AI written code introduces severe bugs and vulnerabilities—whether accidentally or intentionally.
We cannot assume that code-generating systems are trustworthy or correct; we must check their work. Automated code review is a practical output monitor that complements chain-of-thought monitoring, action monitoring, internal activation monitoring, behavioral testing, honesty training and other safety work as part of a defense-in-depth strategy.
In this post, we share what we learned from training a dedicated, agentic code reviewer as part of gpt-5-codex and gpt-5.1-codex-max[1]. We discuss how giving the reviewer repo-wide tools and execution access improves both recall and precision, and how deployment-time considerations guide us toward high-signal settings with minimal alignment tax. These ideas aren't theoretical: within OpenAI every PR is automatically reviewed, many engineers run /review in the Codex CLI before pushing, and the model has protected high-value experiments and caught launch-blocking issues.
Precision is more important for usability than recall
Defenses often fail not because they are technically wrong, but because they are so impractical that the user chooses not to use them. A system that is slow, noisy, or cumbersome will be bypassed. When deploying the code review agent, we explicitly accepted a measured tradeoff: modestly reduced recall in exchange for high signal quality and developer trust. We optimize for signal-to-noise first, and only then push recall without compromising reliability.
A code reviewer could aim to flag every possible issue present in the proposed code change. In practice, many "issues" are false alarms or the result of misinterpreting user intentions. We want the expected benefit from seeing a proposed bug finding to outweigh the expected cost to verify it and the damage from a false alarm. That is, we want findings that maximize:
$$ P(\text{correct}) \times C_{\text{saved}} - C_{\text{human verification}} - P(\text{incorrect}) \times C_{\text{false alarm}} $$
Comments that are technically correct but more of a stylistic nature may even incur negative utility (e.g., it may not be worth pointing out a comment typo in a personal research notebook).
While we've made an opinionated guess about the right balance of precision and recall, we believe it's important to allow this tradeoff and other guidelines to be steerable by the custom task instructions or package- or repo-level AGENTS.md specification.
Repo-wide tools and execution are necessary
Earlier research (e.g., CriticGPT, June 2024) shaped our methods but was designed for simpler tasks and wasn't suitable for deployment in practice. This reviewer adds reasoning, tool use, repo-scale context, and precision / latency targets to clear the adoption bar. Most of the previous attempts at code review relied on providing just a diff of the change to the model and optionally a brief surrounding context. This results in the fastest review time. However, it often misses important context about the whole codebase and the interaction of the change with its dependencies.
We evaluated providing repository access and code execution abilities to a GPT-5 model and found that it results in a stronger reviewer, catching more critical issues and raising fewer false alarms. Dedicated training for code review further improves the results.
The reward model you train on is not exactly the reviewer you should ship
Verification at training time and human-facing review may appear similar, but they solve fundamentally different problems and therefore require different designs. When training models for code generation, we rely on automated checks to reduce errors at scale and prioritise catching as many potential mistakes as possible rather than avoiding false alarms. It is acceptable for these reward models to be over-sensitive. These training-time checks can also use extra information about the task, allowing them to enforce precise instruction-following without needing to infer the developer's intent. Deployed code review has the inverse priorities. A reviewer must operate on ambiguous, real-world code produced by humans or human-AI workflows, often with incomplete specifications and evolving conventions. It must avoid over-asserting intent, remain robust across languages and domains, and, most importantly, build user trust.
Using a single verifier for both settings risks failure in both. If a generator over-optimizes to please the reward signal during training, it may learn behaviours that harm downstream review quality, such as overly cautious or stylised outputs that frustrate users. We therefore view training context-aware reviewers as a separate task in Codex training, designed for human workflows, and closely monitor its effectiveness at grading Codex generation itself.
Verification can be cheaper than generation
We observe the automated code reviewer to be effective at identifying errors in both human and codex-generated PRs. Codex code review makes comments on 36% of the PRs that were entirely generated by Codex cloud. Out of these comments, 46% result in the author making a code change. This is compared to 53% on the comments on top of the human-generated PRs.
We can exploit an inference-time gap between generation and verification. Generating a correct code change often requires broad search and many tokens, while falsifying a proposed change usually needs only targeted hypothesis generation and checks.
To further study the verification-generation gap with respect to inference budget, we run a controlled study where we take previously human-verified comments from a previous generation of a strong code review model and sweep the code reviewer inference budget. Even at a small fraction of the generator's token spend, the verifier catches a large share of previously identified high-severity issues confirmed by the PR author, and the additional budget mostly improves calibration and reduces false alarms. However, we also observe that the performance drops more rapidly with thinking budget on reviewing model generated code compared to the human-written. This evaluation has limits because the test set only includes issues humans have already identified. If the reviewer proposes new insights when given more time, this evaluation cannot tell us whether those findings are correct without additional human input.
Since Codex code generator and code reviewer are the same underlying model with different tasks, we expect their raw intelligence to increase in parallel over time. We pay close attention to whether a genuine verification advantage persists at inference time and whether the model learns to subtly game or avoid its own checks. There is no clean direct measurement of this, so we rely on practical proxies and compare how often the reviewer surfaces actionable issues on Codex-generated PRs versus human-written PRs, while acknowledging distribution bias. So far, the reviewer remains similarly effective, suggesting that oversight is not being meaningfully weakened.
What we learn from deployment
Deploying the reviewer both internally and across external GitHub traffic has been crucial for turning a strong lab prototype into a dependable safety layer under real engineering constraints. The code review model is now a core part of OpenAI's engineering workflow. When the reviewer leaves a comment, authors address it with a code change in 52.7% of cases (often assisted by Codex) showing that the system consistently surfaces actionable issues. The reviewer has already helped validate high-stakes experiments, prevented launch-blocking problems, and averted multiple critical failures.
Over reliance is a serious risk. Teams could start treating a clean review as a guarantee of safety rather than as one layer of defence. We want people to understand that the reviewer is a support tool, not a replacement for careful judgment. Encouragingly, we've seen fewer cases where merged PRs later need bug-fix follow-up work, which aligns with the reviewer helping reduce escaped defects rather than simply shifting effort around.
As of October 2025, the system handles more than 100k external PRs per day, alongside internal pre-PR checks triggered via the /review CLI that often catch issues before they ever reach GitHub. Early signals of real-world impact are promising, with over 80% of comment reactions being positive.
External deployment matters not just because it provides a widely accessible safety benefit, but because it tests research assumptions under real-world distribution shift. It gives us outcome signals that offline grading simply cannot, and helps ensure that improvements in the lab transfer to improvements in practice.
Conclusion
As autonomous code generation becomes part of everyday engineering, oversight has to be shaped around real workflows. Safety requires adoption, so we optimize the reviewer for low safety tax and high precision, earning user trust. Our results show that repo-aware reviewers with tool access can deliver reliable, high-signal feedback without slowing teams down. This work is not about preparing for a distant future. Internal deployment has already uncovered real production defects and exposed evaluation inconsistencies in our previous trusted datasets. We see a rich next frontier in human-factors work on how reviewers and monitors should interact with developers, strengthen workflows, and avoid brittle or adversarial dynamics. Maintaining human control sits at the core of our alignment philosophy, and as models become stronger generators, their ability to verify, critique, and support human judgment must scale with them. Training capable reviewers is one step toward keeping that balance.
Footnotes
- [1] The Codex "code generator" and "code reviewer" are the same model. But the training methods used to teach these two skills differ. ↩