Safety in numbers

On 29 September Gavin Newsom, California’s governor, vetoed the most ambitious attempt yet by an American legislature to regulate artificial intelligence. Senate Bill 1047 would have made the developers of the largest AI models liable for safety failures, and it applied only to models that cost at least $100m to train and used 10^26 operations of computing power in doing so^{2TechCrunchGov. Newsom vetoes California's controversial AI bill, SB 104729 Sept 2024} . Almost nobody on earth has that kind of money. That was the idea, and it was also the problem.

Mr Newsom’s reasons were short. The bill was “well-intentioned”, he said, but “does not take into account whether an AI system is deployed in high-risk environments”. It “applies stringent standards to even the most basic functions”, so long as a large system deploys them^{1Office of the Governor of CaliforniaGovernor Newsom announces new initiatives to advance safe and responsible AI, protect Californians29 Sept 2024} . Scott Wiener, the Democratic senator who wrote it, called the veto “a setback for everyone who believes in oversight of massive corporations”^{2TechCrunchGov. Newsom vetoes California's controversial AI bill, SB 104729 Sept 2024} .

We think the governor got the verdict right and much of the reasoning too. A law that regulates a technology by what it cost to build is a law about balance sheets, not harm. Cost is a proxy for capability today. It will be a poor one in two years, when training gets cheaper and a model that fits on a laptop can do what last year’s cluster could. A threshold written into statute is a fence built around a river that is moving. Developers would have learned to stay just under it, or to move to Texas.

The bill’s flaws were not only technical. It would have imposed duties on a developer whether or not its model was ever pointed at anything dangerous, while saying little about the firm that takes a modest model and deploys it in a hospital or a power grid. Liability belongs with whoever controls the risk. For a general-purpose tool, that is often the deployer, who knows the setting, and not the lab, which does not. Nancy Pelosi, hardly a foe of regulation, called the bill “well-intentioned but ill informed” and said it was “more harmful than helpful”^{3Office of Nancy PelosiPelosi statement in opposition to California Senate Bill 104716 Aug 2024} . Fei-Fei Li, a Stanford computer scientist, warned that it would “harm our budding AI ecosystem”^{2TechCrunchGov. Newsom vetoes California's controversial AI bill, SB 104729 Sept 2024} . In a state that hosts most of the industry, these are not marginal voices.

The strongest case for the bill deserves a fair hearing. Its supporters argue that the whole point of a threshold is to leave small firms alone, and that the risks they worry about (weapons, attacks on infrastructure) come from the frontier, not from a startup’s chatbot. Waiting for harm and then punishing the deployer is a poor plan when the harm could be catastrophic and the lab holds the best information. Mr Wiener says the debate has “dramatically advanced the issue of AI safety on the international stage”^{2TechCrunchGov. Newsom vetoes California's controversial AI bill, SB 104729 Sept 2024} .

That is half right. The frontier does matter, and the labs do know things that nobody else does. But the answer is to require what they know to be disclosed, not to hang a general liability on their size. Mandatory incident reporting, testing against named risks, protection for whistleblowers and liability for specific misuse would reach the same danger without a number that ages badly. Mr Newsom’s own argument, that smaller models can pose comparable risks, cuts against a size trigger and for rules tied to use.

His own alternative is thinner than the veto. His office says he has signed 17 AI-related bills in the past 30 days^{2TechCrunchGov. Newsom vetoes California's controversial AI bill, SB 104729 Sept 2024} , which is a lot of signing for a state that left its largest question unanswered. For the harder question he has turned to an “empirical, science-based trajectory analysis of frontier models”, with advice from Dr Li and two other experts, and says he will keep working with the legislature next session^{1Office of the Governor of CaliforniaGovernor Newsom announces new initiatives to advance safe and responsible AI, protect Californians29 Sept 2024} . Studying a problem is a respectable first step. It is a poor last one. Meanwhile the labs will continue to police themselves, a system that works until the day it does not.

Mr Newsom has bought himself time, and he has bought it from a bill that deserved to lose. Having killed the wrong rule, he now owes California a right one.