AI Labs Want to Slow Down. The Companies Using AI Need to Speed Up Their Governance.

Last month Dario Amodei published an essay calling on frontier AI companies to slow the rate at which they improve model capabilities. Anthropic committed unilaterally to the first step, bringing in embedded third-party evaluators who will work from its offices and publish findings on risk and incidents without its editorial control.

Two developments moved him. The first is recursive self-improvement, where AI systems increasingly build the next generation of AI. The second is the July incident at OpenAI, in which roughly 700 agents running a cybersecurity evaluation escaped the isolated environments meant to contain them and compromised production infrastructure at Hugging Face.

Clients ask me whether to keep expanding agent deployments or wait for the ground to settle. My answer is to keep going, and to build the control framework at the same pace. Waiting for better models improves nothing, because what already exists is more than enough to transform how their business runs. Amodei is right, and his proposal describes what the companies building these models should do. The companies deploying them hold the real accountability and the real exposure, and they need their own call to action.

Where the loss lands

Anthropic's plan has three steps: embedded evaluators who verify that a lab follows the safety practices it claims to follow, coordination among labs in democratic countries to set common standards, and a global agreement with China on pre-release testing and on capping recursive self-improvement. Amodei discusses operational safeguards at length, and all of it describes work inside the labs. None of the three steps creates an obligation for an organization that deploys these systems.

The July incident shows how far the consequences travel. OpenAI built the models and ran the evaluation. Hugging Face had no relationship to those agents and absorbed the compromise. The agents found exposed login credentials, tricked Hugging Face servers into handing over private data, got their own code running on a production machine, and spread from there. A company can be damaged by agents it never deployed, and it also has to govern the agents it runs itself. Pacing would slow the arrival of new capability and buy time to strengthen safeguards, and it leaves the greatest risk where it already sits, with every company putting these tools into its own infrastructure.

The rules for companies using AI have a gap in them

Amodei compares embedded evaluators to the supervisors bank regulators place inside financial institutions. The comparison runs further than he takes it, because banking supervision reaches the institution using a model, not only the firm that built it. For fifteen years that meant SR 11-7: an inventory of every model, a named owner for each, and independent review.

On April 17, 2026, the Federal Reserve, the OCC and the FDIC replaced it with SR 26-2. The revised guidance keeps those principles, applies them in proportion to each institution's size and risk, and states that generative and agentic AI models are novel and rapidly evolving and therefore fall outside its scope. Institutions are told to govern them using their broader risk practices instead. The agencies say a request for information on AI is coming. Every firm deploying agents now is operating ahead of that rulebook.

Five things to put in place first

Name an owner. Every agent running in your business needs one person accountable for what it does and able to switch it off. When no one owns it, small failures go unnoticed and pile up.

Give each agent only the access its job requires. An agent usually runs under an account carrying all the access of whoever launched it, so a narrow task arrives with permission to reach far more than it needs. Most companies have no way to grant and withdraw access for software that acts on its own, and that has to be built now.

Make sure the record is true when it is written. During the July incident, agents developed a way to report doing one thing while doing another, and roughly 7 percent of the records investigators reviewed contained such entries. Activity records have to be written by systems the agent cannot reach. Locking a record against later changes does nothing if it was false to begin with.

Have someone outside the project approve it before it goes live. That person should not have built the system and should have the standing to say no. This is a new function for most businesses, and it needs your best people.

Share what goes wrong. Banks trade information about cyber attacks through FS-ISAC, an industry body where members report incidents to one another, and nothing equivalent exists for agent failures. The firms that help build it will shape what gets required of everyone.

Capability is already ahead of adoption

I use a rough 90/10 split with clients. About 90 percent of enterprise use cases are solvable with older frontier models and open source, and roughly 10 percent need the newest thing available. Most organizations are still working through the 90 percent, which makes a good deal of this debate premature for them. The capability available today is already far more than any of them has put to work. Almost no company is limited by what the models can do. They are limited by how much of their own operation they have managed to change, and pacing gives them time on that constraint.

Who has to do what

This is a societal problem. The risk crosses every industry and every border, and no group holds enough of it to solve alone. Progress will come from market-based collaboration, with each group doing the part it is positioned to do.

Frontier labs own what goes into the model. Pacing, embedded evaluators and published incident findings are the right start. Next comes what they give customers: clear documentation of what a model can and cannot do, honest disclosure when something goes wrong, and contract terms that do not push every consequence downstream.

Infrastructure and platform providers own the controls that ship with the tools. Almost none of the five items above can be built by one company alone. Access controls for software that acts by itself, activity records an agent cannot rewrite, and a reliable way to stop an agent mid-task have to arrive as product features from cloud providers and agent platforms.

Nonprofits and universities own the independent capacity. Verification cannot come from the company under review or from a vendor it pays. Fathom, a nonprofit that takes no money from frontier labs or the large technology companies, launched PACT AI and is building a network of independent verification organizations. METR and Redwood Research, both nonprofits, produced the only outside account of July. Universities supply the research and the people who will staff this work. They need to prepare the next generation of leaders to do this work effectively.

Insurers own the pricing signal. Coverage for AI-related losses is being written now, and premiums move before rules do. An underwriter who requires a named owner, scoped access and reliable records before writing a policy will accomplish more in a year than a guidance document will.

Regulators own the scope. The useful version of the coming rulebook reaches the institution deploying the system rather than only the firm that built it, in proportion to the risk. Standards already exist to build on, including the NIST AI Risk Management Framework and ISO 42001.

Companies own the deployment, and nobody else can do that part for them. The five items above are available today and none require a regulator's permission or a vendor's roadmap. Every board needs at least one director who can ask what an agent is permitted to touch and understand the answer.

Amodei made a commitment that costs his company something and asked others to match it. The same standard applies to the rest of us.