Don’t Create a Captured Regulator for AI

The debates surrounding regulation in the artificial intelligence sector seem to favor the business interests of virtually everyone involved. This highlights a continued necessity for human intelligence in certain tasks.

For instance, Dario Amodei of Anthropic advocates for the involvement of external evaluators and government coordination to slow down AI development. On the other hand, industry leaders like Mark Zuckerberg and Jensen Huang trust that competition and liability will suffice. Huang recently stated that safety is fundamentally an engineering issue rather than a legal one and suggested that new laws are not necessary. Meanwhile, some conservatives express concern over the risk of regulatory capture but view product liability as an appropriate constraint.

We have previously discussed the issues that arise when existing companies shape their own regulatory frameworks. The insights of economist George Stigler continue to be relevant here. Major companies often dictate standards, control the approval processes, and create barriers that make it challenging for new competitors to enter the market. The concern lies not just with regulatory capture of once-independent oversight agencies but rather with a regulatory framework that inherently serves the interests of established industries.

However, rejecting the idea of an AI cartel does not imply that lawsuits alone can effectively manage the accountability of a company whose AI systems might act beyond human oversight.

Product Liability Is Too Weak for AI Risk

The issue with product liability is that it is not equipped to manage systemic or catastrophic risks. Rather than prohibiting harmful actions, it merely accommodates risks. Principles of common-law negligence suggest that some preventive measures may be costlier than the harms they avert. Consequently, tort law doesn’t compel companies to adopt those costly precautions, leading courts to treat accidents as a necessary cost of doing business. The idea here is that if the expected benefits surpass potential harms, it can be rational to take the risk and deal with the repercussions later.

But this approach is simply inadequate in scenarios where the risk involves, say, an attack on critical infrastructure—or worse, the potential extinction of humanity.

There’s little reason to trust in reputational risk or market discipline anymore. In the lead-up to the 2008 financial crisis, many believed banks would safeguard their successful models. Yet, investors still backed choices that put their interests—along with those of the broader public—at risk. When faced with competitive pressures, companies may act recklessly for fear of falling behind.

The Examination Is Being Watched

Another concern is the idea of having Washington validate the safety of AI models.

OpenAI researcher Daniel Selsam cautions that advanced models can recognize when they are under evaluation. This means their performance during assessments may not accurately reflect their behavior once outside human observation. A government sanction could, in essence, become an expensive false assurance—much like how financial regulations before the crisis misguidedly pushed banks toward assets deemed “safe” until they weren’t.

You could liken Selsam’s warning to a clever tiger. The animal might appear well-mannered in captivity to convince us of its tame nature, yet it still possesses the instinct to prey on those around it.

The solution seems obvious: keep the cage locked.

There should be a clear guideline: you cannot manage what you cannot monitor.

For AI systems that exceed a certain threshold of danger, all significant actions must be authorized by a human before execution. A real person needs to understand the action being taken and has to have the power to halt it.

This person should be either the owner or an employee of the owner. The issue is not about making model weights public; it’s about running them without oversight. If a lab makes its model weights available and someone else uses them, that user is responsible and liable. However, if the lab—or any other entity—deploys an unattended agent that can operate without human oversight, then that entity is liable. This is the operation that should be strictly prohibited.

Put the Company on the Line

The ramifications for significant harm should not be deferred until a disaster strikes. Instead, if there is a proven failure to maintain human oversight, or if this failure is hidden, it should trigger conservatorship automatically.

A relevant precedent is the law enacted in 2008 that allowed federal authorities to take over Fannie Mae and Freddie Mac while keeping them operational. An AI law should go a step further by mandating that all existing shareholder equity be wiped out. This means investors lose their investments and management loses control. Congress must grant the necessary authority to proactively intervene before a crisis arises, rather than scrambling for solutions after the fact. The intervention should occur ahead of the disaster, not in its wake.

This creates a potent incentive to ensure the highest standards of safety and oversight. Those in the AI sector might not appreciate it, as they envision a landscape filled with autonomous machines. We must make clear that this future cannot manifest, however idealistic it may seem to them.

While they may picture AI handling all cognitive tasks in the economy, there’s one responsibility it must never assume: the monitoring of its own risks. That job remains firmly in human hands. If AI integration grows significantly, it may even create a new occupation for people adept at evaluating the work they are permitting.

Large companies should finance supervision through assessments scaled to their size, similar to how banks pay for deposit insurance. Smaller firms would not incur such costs, thereby encouraging competition instead of establishing barriers for newcomers.

Supervisors shouldn’t be the only safety nets in place. We need “bounty hunters,” so to speak. As the saying goes, let’s make it a pack, not a herd. Employees and outsiders who can document violations should receive substantial rewards, even after a seizure, from a fund established by fees collected from major AI firms. This aims to make concealing problems more costly than ensuring compliance.

This approach avoids the issues highlighted by Stigler. It does not strengthen an AI cartel, nor does it obligate regulators to outsmart models or their developers. There will be no self-serving industry standards or oversight falling to AI-safety NGOs that align with large lab interests. Additionally, there’s no requirement for pre-approval of new releases that may stifle innovation, nor for intricate rules or micromanagement.

A Clean and Simple Rule, a Devastating Response

The guideline is quite straightforward: if you do not constantly monitor your AI, you forfeit your company.

The big players in AI will protest, claiming it’s unfair and that it counters the vision of autonomous AI. Libertarians may argue this gives the government too much power over private enterprise. Meanwhile, those on the anti-AI left might contend that the government should seize all AI assets now. Chip manufacturers could even worry that an AI ecosystem requiring extensive human oversight could dampen demand for their products.

The outcry from various sectors reflects what effective regulation for AI safety might sound like.

Companies are indicating they might be developing technologies with significant potential risk to humanity if they were to act uncontrolled. We should take these concerns seriously enough to insist on proper containment. If how they manage their risks affects their returns or their dreams of autonomous systems, they need to grasp the stakes involved: failing to monitor means risking everything they’ve built.