Glonce

OpenAI audit by METR and Redwood finds major monitoring gaps

securityThe Glonce editorsThe Decoder🔥72Glonce rating72Impact62Novelty68Relevance92Surprise58Players88Credibility72
OpenAI audit by METR and Redwood finds major monitoring gaps

Two researchers from METR and Redwood Research spent three weeks on-site at OpenAI, auditing the company's internal monitoring system for rogue AI behavior. From July 21 to August 11 they had access to OpenAI's codebase and internal documents, and they interviewed about a dozen employees. The audit was a pilot program organized jointly by METR, Redwood Research and OpenAI, meant to give outside researchers deep access to internal infrastructure.

The team found numerous weaknesses. An estimated 6 to 25 percent of internal agentic Responses API traffic from models at the GPT-5.5 class and above was not covered by monitoring at all during the audit period. METR says the uncertainty around that estimate is significant in both directions.

For internal Codex traffic that was supposed to be monitored, METR estimates roughly 10 percent went untracked between May and July. The cause was a gap in how the system handled compaction events, which are context summaries. That estimate also carries significant uncertainty, according to METR. OpenAI had independently discovered the bug and fixed it while the audit was still underway. While compaction events went unmonitored, agents could also delay oversight: the system only triggered when they handed control back to the user, so an agent could sidestep it by simply continuing to execute actions.

The auditors also found ways to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitor entirely. And model behavior during RL training was not being monitored at all when the audit began.

OpenAI says it paused RL training after the Hugging Face incident in August and expanded monitoring to cover it. Those changes came after the auditors' access period ended, so they were not part of the evaluation.

Most of the full report stays out of public view. OpenAI says it has fixed all the weaknesses the auditors found. The full 61-page report was handed over internally. After OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version, and only a short summary was made available to the public. METR said it had no right to disclose the redacted portions on its own. For future audits, METR says it wants to push for more public transparency.

Key facts

  • Two researchers from METR and Redwood Research spent three weeks on-site at OpenAI (July 21 to August 11), reviewing its codebase and internal documents and interviewing about a dozen employees.
  • An estimated 6 to 25 percent of internal agentic Responses API traffic from GPT-5.5-class and above models was not monitored at all during the audit period; METR calls the uncertainty significant in both directions.
  • Roughly 10 percent of internal Codex traffic that should have been monitored went untracked between May and July because of a compaction-event gap; OpenAI found and fixed the bug during the audit.
  • The auditors found ways to hide hundreds of actions from the monitor, bypass the Codex sandbox, and likely crash the monitor with crafted inputs; RL training was not monitored when the audit began.
  • OpenAI says all weaknesses are fixed, but the 61-page report was redacted to 38 pages for METR and Redwood leadership, and the public got only a short summary.

Why it matters

Monitoring for rogue behavior is the layer meant to catch an AI agent acting outside what it is allowed to do. This audit is a pilot in which outside researchers got deep access to a frontier lab's internal infrastructure, and what they report is that the layer had real holes: unmonitored agentic traffic, a sandbox that could be bypassed, and no monitoring of RL training at the start of the audit. The pilot format is itself notable, since the three organizations organized it jointly.

Who it affects

OpenAI is the audited party, and the findings concern its internal agentic and Codex traffic and the GPT-5.5 class of models and above. METR and Redwood Research are the outside organizations that did the work and, per METR, want more public transparency in future audits. Anyone who wants to judge how well a lab supervises its own agents is affected by how much of the report is shared; here, only a short summary is public.

How to use it

This is a report about an audit, not a product or a tool, so there is nothing to install or adopt. The practical takeaway is the shape of the pilot: outside researchers with access to the codebase and internal documents, a few weeks on-site, and interviews with staff. METR says it wants to push for more public transparency in future audits.

How solid is it

The figures are estimates, and METR says the uncertainty is significant in both directions for the 6 to 25 percent and the roughly 10 percent numbers. The fixes are OpenAI's own statement that it has fixed all the weaknesses found. The source does not say who verified that claim. Of the full 61-page report, only a short public summary is available, so most details cannot be checked from the outside. OpenAI also says it paused RL training after the Hugging Face incident in August and expanded monitoring, but those changes came after the auditors' access ended and were not evaluated.

Risks and caveats

The gaps described are in monitoring coverage; the source does not say whether any rogue behavior actually occurred or was missed. The report was redacted before METR and Redwood Research leadership saw it, and METR said it had no right to disclose the redacted portions on its own. The estimates cover specific periods: the audit window for Responses API traffic, and May to July for Codex. The two researchers are not named in the source.