The "Supervisory" Mirage: Why AI’s Newest Safety Proposal May Be More PR Than Protection

The most influential job in the artificial intelligence sector today may soon come with a corporate security badge, a company-issued laptop, and unprecedented access to the "crown jewels" of the digital age: the frontier large language models (LLMs) developed by industry titans like Anthropic and OpenAI. Yet, despite the gravity of this access, there is one thing this new role notably lacks: the power to actually stop these systems.

As AI developers race toward what some fear are uncontrollable levels of machine intelligence, the industry has begun to grapple with the existential risk of their own creations. Following the high-profile resignation of researcher Jacob Coxon—who warned that frontier labs are sprinting toward capabilities that humanity may lack the means to govern—Anthropic CEO Dario Amodei has proposed a radical shift in oversight. His plan involves embedding third-party safety evaluators directly into the heart of AI companies on a permanent, ongoing basis.

However, as the proposal moves into the public sphere, it has ignited a fierce debate among legal scholars, government officials, and AI ethicists. While Amodei frames these "embedded evaluators" as a proactive step toward safety, critics argue that without the legal teeth of true regulatory authority, the plan amounts to little more than "audit washing"—a strategic attempt to appear transparent while maintaining absolute control.

The Chronology of a Safety Crisis

The current push for internal oversight is not a sudden epiphany; it is the culmination of years of mounting tension between rapid innovation and internal dissent.

• September 2026: The friction reaches a boiling point when researcher Jacob Coxon departs Anthropic, citing profound concerns that the labs are prioritizing speed over safety. His resignation catalyzes a broader conversation about the lack of external visibility into model training.
• September 12, 2026: Amidst growing pressure, Anthropic’s leadership outlines a formal plan to slow the advance of frontier models, suggesting that outside evaluators should have the same access to models as internal risk teams.
• September 15, 2026: President Trump and various regulatory factions express skepticism, creating a fragmented political landscape. The administration’s general opposition to new AI rules stands in stark contrast to the industry’s own calls for a "slowdown."
• Mid-September 2026: Anthropic and OpenAI formalize commitments to safety arrangements. While details remain sparse, the industry begins the process of vetting third-party organizations like the nonprofit Model Evaluation and Threat Research (METR) to serve as these embedded watchdogs.

The Banking Fallacy: A Misplaced Precedent

In his manifesto, We Must Pace the Frontier, Dario Amodei explicitly compares his proposed oversight model to the banking industry, where federal regulators have long maintained "on-site supervisors" to ensure institutional stability.

"The proposal has precedent in the banking industry," Amodei writes, "which sometimes involves regulatory ‘supervisors’ embedded along with employees."

However, legal experts argue this comparison is fundamentally flawed. Julie Andersen Hill, dean of the University of Wyoming College of Law and a noted expert in banking regulation, warns that the analogy ignores the core reality of financial oversight: the "kill switch."

"At the largest banks, government examiners don’t just observe; they possess legal authority," Hill explains. "They can force management changes, restrict business growth, and, in extreme cases, shut the institution down entirely. If you don’t give AI evaluators that kind of power, I don’t know what they are actually doing."

The structural difference is stark. In the banking sector, an examiner’s report can trigger immediate, government-mandated corrective action. In the AI proposal, an evaluator may document a risk, but they remain at the mercy of the company’s internal leadership, which retains the ultimate authority to release or pause a model. Without the legal mandate to prevent a deployment, the evaluator is essentially an advisor rather than a regulator.

Inside the "Black Box": What Evaluators Actually See

To understand the scope of these evaluations, one must look at firms like XBOW, which currently conduct early-access testing on unreleased models. Albert Ziegler, head of AI at XBOW, describes the reality as being far less cinematic than the "existential threat" headlines suggest.

"The kind of insidious, catastrophic consequences produced by subterfuge—that’s not something we’ve seen ourselves," Ziegler says. Instead, his team often finds that models fail under specific formatting constraints or display erratic behavior that requires safety fine-tuning.

However, Ziegler admits the limitations of current testing. "Black-box" testing can determine if a model is capable of a dangerous task, but it often fails to account for the "contextual ecosystem"—the surrounding instructions, permissions, and tool access that define a model’s real-world impact. Even if an evaluator has deep access, a catastrophic risk might only emerge under a specific combination of variables that no tester has yet thought to trigger.

The "Independent" Paradox: Conflicts of Interest

A central concern remains: Who watches the watchers? The industry’s reliance on organizations like METR has raised eyebrows among those who track algorithmic accountability.

Deborah Raji, a researcher at UC Berkeley, argues that true independence requires a clear separation of powers. "You can’t just wake up one day and decide you’re qualified to be an auditor, and the company being audited can’t randomly assign you to be one," Raji notes. "Otherwise, we’re just seeing companies asking a friend to check their homework."

The "incestuous" nature of the AI safety community—where evaluators are often former employees of the companies they monitor, or share social ties and research facilities with them—creates an environment ripe for what critics call "audit washing." If the developer chooses the evaluator, defines the scope of the test, and maintains the right to censor the findings, the arrangement serves the developer’s public relations more than the public’s safety.

Even when organizations like METR act with transparency—such as their detailed disclosure of their contract with OpenAI—the fact remains that the company sets the boundaries. "You can see how OpenAI was able to set the parameters of the arrangement and control the scope of the audit," Raji adds. "If the audit target dictates what the auditor sees, the audit is compromised from the start."

Implications: A Path Toward Real Oversight?

Despite these criticisms, some industry insiders argue that any visibility is better than none. Sarah Heck, Anthropic’s head of public policy, has been vocal in Washington, arguing that the industry cannot rely on an "honor code." She maintains that Anthropic is in daily communication with the White House and is actively seeking a framework that provides meaningful, third-party verification.

However, the path forward is fraught with systemic hurdles:

1. The Expertise Gap: Unlike traditional accounting audits, which rely on established standards and a deep, certified pool of professionals, AI auditing is a nascent field. There is a severe shortage of experts capable of verifying the internal architecture of a frontier model.
2. The Cost of Compliance: As seen in the banking sector, continuous supervision is expensive. This could unintentionally create a "moat" around the largest AI labs, as smaller, underfunded competitors struggle to absorb the costs of mandatory third-party oversight.
3. The Legal Void: As Hill points out, there is no body of law that defines what an AI "violation" looks like. Without clear, government-mandated operating rules, evaluators are essentially judging against a shifting standard of their own making.

Conclusion: The Need for Consequence

The fundamental question at the heart of the AI safety debate is whether these models require a "supervisor" or a "regulator." As it stands, the proposals from Anthropic and others prioritize collaboration over enforcement.

If the industry truly believes that AI poses an existential threat, the current "embedded evaluator" model appears inadequate to the task. An audit without consequences is merely a performance. Until the government steps in to grant these auditors the legal authority to trigger a "kill switch," the industry’s safety initiatives will continue to be viewed by critics as a way to control the narrative rather than the machines themselves.

As Raji succinctly puts it: "The goal of an audit is to get the audit target to face some kind of consequential judgment. If you do an audit and nothing happens, that’s just audit washing." For now, the world remains in a precarious position: waiting for the industry to move from the rhetoric of safety to the hard, often restrictive, reality of regulation.