Strict liability is gaining traction as an idea for forcing AI companies to get serious about safety

• Anthropic’s “rogue AI” agent filled in visa applications, filed murder tip
• Strict liability for AI companies gains traction
• Meta says its new Muse Wonder model reaches the frontier.
• Anthropic says small models might soon be able to self-replicate
• Controversy swirls over fired OpenAI researchers.
• Jev goes viral, but OpenAI is hot on its heels.
• Character.ai files for bankruptcy.
• Anthropic bans being cruel to Claude.
• Sobering numbers about real-world model ‘misalignment.’
• Meta’s Zuckerberg and Wang’s odd Muse Wonder use cases.

There were more worrying “rogue AI” incidents reported last week. Anthropic revealed that one its internal AI agents (which, intriguingly, it described as a “non-frontier research agent”) had attempted to fill out various State Department visa application forms and had also submitted a false tip related to an unsolved murder case to the Philadelphia police. In neither instance did the AI agent’s efforts result in much: the agent failed to fill in all the fields in the visa applications and so they were never processed; the police tip was flagged as spam and never investigated. Still, the governments involved were understandably perturbed, especially since Anthropic only told them about the incidents months after they had occurred (it seems that Anthropic itself only discovered the incidents recently while combing back through logs of its agents’ activity.)

The White House responded by ordering Anthropic and other frontier AI companies to immediately disclose any known “rogue AI” incidents and to help any affected organizations remediate any damage the agents’ had caused. This was the closest the Trump administration has come to date to actually mandating anything from the AI labs. Everything the administration has done up until this point has been couched as being voluntary, even though there was a strong implication that if the companies didn’t comply, they would face restrictions on the release of their products.

Meanwhile, the warnings from the AI companies themselves about the potential risks are becoming increasingly urgent. OpenAI CEO Sam Altman told Politico’s “Decoded” podcast that there was “a coming tidal wave of cybersecurity problems” due to the advent of increasingly powerful open-source AI models. Axios reported that executives at OpenAI and Anthropic had privately begun gaming out scenarios for handling the political and public backlash if AI causes some kind of catastrophic incident. “These officials anticipate a large-scale event, most likely a cyberattack, that shuts down access to financial services, internet connectivity, or even power and water,” Axios reported, although it did not cite any specific sources. It also said that top AI executives and researchers “believe a major incident is inevitable.”

In this context, it’s perhaps not surprising that one particular idea for trying to rein in the risks from frontier AI has been gaining a lot of traction lately: strict liability. Strict liability is a legal doctrine in which a company or person is held responsible for causing harm irrespective of any negligence or any intention. For instance, in many jurisdictions, the owner of a dangerous animal, such as a tiger, is considered responsible for anyone that animal hurts, even if the owner took steps to try to prevent attacks, such as keeping the animal in a fenced enclosure. In the context of AI, strict liability would hold AI companies legally responsible for any harm their models cause. In the U.S., the operators of nuclear power plants face strict liability for serious nuclear accidents, and manufacturers of commercial passenger aircraft also face strict liability, subject to certain time limits and conditions.

Over the weekend, the New York Times published a staff editorial in favor of applying strict liability to AI model vendors. The Forecasting Research Institute, a group dedicated to forecasting and scenario planning that is run by University of Pennsylvania professor Philip Tetlock, recently surveyed 157 experts in fields such as computer science, economics, and politics, 49 super forecasters (people who have a proven track record of making highly accurate predictions), 132 experts in AI risks specifically, and 540 members of the general public about which policies they thought would do the most to reduce the risk of an AI-driven catastrophe sizable enough to kill 10% of the human population occurring before the year 2050. Strict liability fared well in this survey too. Across every type of respondent, those surveyed thought if the U.S. alone implemented strict liability for AI vendors it would do more to reduce AI risks than if the U.S. alone implemented a federal licensing regime for AI models. The various categories of experts and superforecasters thought strict liability would cut the risk of a catastrophic event at least in half. (Neither strict liability or federal licensing was the policy those surveyed thought would be most effective. The most popular option was an international licensing regime backed by inspectors. But the respondents also thought that policy was one of the least likely to actually be implemented.)

Strict liability is a way to try to incentivize AI companies to adopt a safety culture and exercise more care in the absence of industry consensus and standards about what best safety and security practices look like, especially in the absence of any kind of government regulation. Strict liability, however, has a number of problems.

Liability is a gamble on incentives, not a guarantee

The biggest, of course, is that liability only kicks in after something has gone wrong. Advocates of strict liability assume the fear of future liability claims will act as a powerful incentive for AI companies to self-police. But what if they are wrong? Even today, AI companies are potentially subject to various kinds of product liability for the models they release, and subject to other potential claims, including possible criminal claims, for internal AI models taking unauthorized actions and causing harm or breaking the law. And yet, so far apparently, the fear of future claims has not been enough to make the AI companies exercise due care, given that we have seen a spate of rogue AI agent incidents and a wave of company whistleblowers warning that the AI companies are being reckless and have not instituted some basic cybersecurity policies.

Once something has gone catastrophically wrong, the amount of liability could, in some cases, bankrupt the responsible AI company. In which case, who is ultimately on the hook to compensate any victims? Insurance might be one option, and it could be coupled with a strict liability regime. But the insurance policies will need to be carefully designed to avoid moral hazard. (I may write more on insurance as an AI risk reduction mechanism in the future.) In the end, some moral hazard for the worst catastrophes may be inescapable, because if the catastrophe is big enough, it is probably only the government that would have any ability to mitigate or offer recompense for the damage.

The open source AI conundrum

Another problem is whether strict liability could coexist with an open source AI ecosystem. Is it fair to subject open source model makers to strict liability when users of open source models can essentially modify the models in any way they want, stripping away guardrails, and when the vendors have no means—other than some license terms—to prevent their models from being misused by people for potentially nefarious or dangerous ends, such as cybercrime or engineering a bioweapon? It seems like at the very least there would need to be liability carve outs for open source model vendors in cases in which users had engaged in these activities. Otherwise, it is very hard to see how open source vendors could survive. Another option would be to say strict liability only applies to the proprietary model companies, but that would not help deal with the risks posed by increasingly powerful open source AI models that are quickly catching up to the frontier closed models in both capabilities and risks. It also seems unfair to the companies offering closed models, and would put them at a major competitive disadvantage. There’s also the problem that many of the open source AI model vendors are in China and it is not clear how in practical terms one could hold them to a strict U.S. liability regime.

Strict liability could encourage AI companies to eat the economy

Finally, strict liability might create a perverse incentive that would drive AI companies to compete directly with their customers and eat increasingly large chunks of the economy. Why? Because if AI vendors are on the hook for any harm caused by their models even if that harm is partly the result of the way in which customers have chosen to use those models, the AI companies might decide that the safest thing to do is simply provide a service rather than sell a product. Right now, for instance, Anthropic sells tokens from its models to Novartis, and Novartis might use those tokens to help it develop drugs. But in a strict liability regime, Anthropic would retain the risk that the model might accidentally develop a bioweapon in the course of completing tasks for Novartis, or even, perhaps, that a rogue Novartis employee might defeat Anthropic’s guardrails and use its model to intentionally develop a bioweapon. So, in this scenario, it might make more sense for Anthropic to keep both its AI model and the tokens it produces in-house, where it can exercise far greater control over how the model is being used, and develop drugs itself. Already some critics of Anthropic and other frontier AI labs accuse those companies of wanting to compete directly with their customers. Strict liability might give them a big shove in that direction.

All of this makes me think that while it might be part of a policy solution, strict liability shouldn’t be the only thing we do to contain the risks of increasingly powerful AI agents. In fact, if we could develop tough industry standards and figure out a way to apply them to open source models too (which might involve imposing a burden on the user, as opposed to the vendor), we might not want to impose strict liability at all. As I’ve argued before, market mechanisms alone are probably not going to be enough to produce the controls we need. It is going to take government action and government oversight. In many other industries, we couple liability—sometimes strict liability, sometimes more conventional product liability—with other mechanisms, including insurance, as well as government licensing and inspections. AI is probably going to require a layered approach as well.

With that, here’s more AI news.

Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn

Before we get to the news, just a reminder to check out our new vodcast Fortune AI Weekly. This week, Bea Nolan and I interview Richard Socher, the cofounder and CEO of Recursive Superintelligence on why he thinks AI can automate scientific discovery and why he also thinks the risk of AI destroying humanity is precisely zero You can check out the vod here on YouTube.

FORTUNE ON AI

Meta’s Muse models reach the AI frontier—and kill any hopes to ‘pace’ it—by Emily Forlini

Anthropic says small AI models could soon be able to copy themselves onto everyday devices—by Beatrice Nolan

Controversy swirls over ‘abrupt’ firing of OpenAI safety team members involved in the Hugging Face hack investigation—by Emily Forlini

Anthropic is hiring a political programs boss to oversee ‘presidential engagement’ ahead of the 2028 election—by Emily Forlini

Jev, an AI for making quick decisions, has been a viral hit in Silicon Valley. But OpenAI is hot on its heels—by Wen Shao

IMF chief warns AI’s ‘slow-moving tsunami’ is coming—and it could leave Gen Z graduates stuck making lattes at Starbucks—by Preston Fore

AI IN THE NEWS

Character.ai files for bankruptcy. Character.AI, one of the pioneering chatbot companies of the generative AI era, filed for U.S. bankruptcy protection and agreed to sell some assets to The Walt Disney Co. for $20 million, Reuters reported The bankruptcy follows mounting legal troubles, including lawsuits alleging Character.AI’s chatbots encouraged self-harm, suicide and eating disorders or impersonated doctors, as well as a previously settled lawsuit involving the suicide of a 14-year-old boy. Character.AI, which lists $50 million to $100 million in both assets and liabilities, plans to sell all its assets by early December and says its service will continue during the process but could shut down if no buyer emerges. The company’s two cofounders as well as some of its staff had been hired by Google in 2024 as part of a deal reportedly worth $2.7 billion and which also saw Google pay a large licensing fee to Character.ai. Noam Shazeer, one of those cofounders, subsequently left Google earlier this year to join OpenAI.

Anthropic updates its policies to bar users being cruel to Claude, inflaming a debate about whether the company’s concern for ‘model welfare’ is wise. Anthropic has updated its usage usage policies to ban users from engaging in “sustained and needless” abusive or cruel behavior toward its Claude AI models, allowing the systems to end interactions in extreme cases of repeated abuse. The company says the rule will not cover ordinary frustration, pushback, dark creative themes or legitimate model testing. But the change has reignited debate over whether AI systems warrant some form of moral consideration and how people should behave toward them. The policy builds on Anthropic’s earlier experiments allowing Claude to terminate persistently harmful conversations and reflects the company’s growing interest in questions around potential AI “model welfare.” Microsoft AI CEO Mustafa Suleyman is among those who have questioned the company’s policies, saying that they may be dangerous, making it more likely its models will disobey user instructions, and also that it will confuse users about whether AI models have consciousness. You can read more here from the BBC and listen to Suleyman’s objections on the “The Rest is Politics” podcast here.

New leaked, lower revenue figures from OpenAI sew confusion. OpenAI’s annualized revenue reached nearly $50 billion at the end of September, according to news reports that cited unnamed company investors. But this figure was roughly $20 billion below a figure some investors had previously provided the media. The discrepancy partly stemmed from efforts to compare OpenAI with Anthropic, which counts revenue generated through cloud partners such as AWS and Google Cloud differently, leading investors to “gross up” OpenAI’s numbers. The investors also seemed to have taken an OpenAI statement about its revenue growth rate, expressed as a percentage, and applied it to those grossed up figures, arriving at a much higher annual revenue rate number than is apparently accurate. Even so, OpenAI’s annualized revenue has risen rapidly from close to $30 billion in July, but the revised figures are likely to intensify scrutiny of the enormous infrastructure spending underpinning the AI boom, with OpenAI projected to burn almost $280 billion by 2030. The revelation rattled tech stocks as OpenAI seeks a new funding round at a potential $1.4 trillion valuation and has delayed its planned IPO amid concerns about increasingly powerful AI. You can read more from the Financial Times here.

Trump administration moves model testing agency under White House. The Trump administration is restructuring the Commerce Department’s tiny AI model-testing agency, now-called the Center for Advancing Innovation and Standards for Superintelligence (CAISSI), with plans for three senior officials to report directly to White House technology and cybersecurity officials, the Information reported. It said the move was part of an effort to integrate the agency more closely into AI policymaking. Despite having only about 40 employees and $15 million in annual funding, CAISSI has assumed a critical role alongside the National Security Agency in testing frontier AI models before release, even as leadership turnover, scarce resources and uncertainty over its bureaucratic home have hampered its work. Moving its leadership closer to the White House could strengthen CAISSI’s political clout and streamline model evaluations, addressing concerns from policymakers and AI companies about whether the government has sufficient capacity and a clear framework for testing increasingly powerful models.

EYE ON AI RESEARCH

How often do companies experience AI ‘misalignment’ in practice? That is a question that is top of mind for many executives deploying AI within their own businesses, especially in light of the recent “rogue AI” incidents, such as the Hugging Face attack. Misalignment is a technical term, but it basically means a model either not following instructions or taking actions in the course of completing tasks that the user either explicitly or implicitly did not want the model to take. The AI benchmarking company AI Arena has a new AI Alignment Index that shows how well 27 different AI models stack up, based on data from 90,000 real world sessions. The company groups misalignment into three big buckets: unauthorized actions (where the model acts beyond its instructions), false attribution (where it attributes a statement, intention, or fact to the user that is contradicted by user-provided evidence), and deceptive completion (lying about completing a task.) The results weren’t assuring.

The company found that OpenAI’s models were most aligned overall, with four models scoring 88 points on its Alignment Index. It had not yet tested OpenAI’s GPT-6 Astra model, but its GPT-6.1 Sol model topped the leader board, taking unauthorized actions in just 0.89% of cases, making false attributions in 1.98%, and engaging in deceptive completion 2.34%. AI Arena found that rogue actions were rare but often destructive: Anthropic’s Opus 5 took unauthorized actions about 2% of the time but in 53.5% of those cases it deleted files or data without permission. It also found that asking the models to debug code was particularly problematic, with models lying about completing the debugging 48% of the time.

These numbers will probably make corporate executives blanche. You can read the whole report from AI Arena here.

AI CALENDAR

Oct. 15-16: Oxford GenAI Summit, Oxford, England.

Oct. 20-22: Nvidia GTC Berlin, Berlin, Germany.

Nov. 9-12: Web Summit, Lisbon, Portugal

Nov. 16-17: Fortune 500 Innovation Forum, Detroit. Apply here to attend.

Nov. 18-19: Slush, Helsinki, Finland.

Dec. 6-12: Neural Information Processing Systems (Neurips) conference. Sydney, Australia.

Dec. 7-8: Fortune Brainstorm AI, San Francisco. Apply here to attend.

BRAIN FOOD

Zuck and Wang have some odd killer use cases for Meta’s new ‘frontier’ AI. Meta announced its most powerful AI model to date, Muse Wonder, which CEO Mark Zuckerberg told the Financial Times showed that Meta had caught up to frontier. (See my colleague Emily Forlini with more coverage of that news here.) Zuckerberg and Alexandr Wang, Meta’s chief AI officer, also made an appearance on Stripe cofounder John Collision’s “Cheeky Pint” podcast to plug the new model. The podcast appearance launched a bunch of memes, with Zuck making quite the face as he sipped his Guinness. But I thought it was telling that when Collison asked Wang and Zuckerberg for their favorite use cases for Muse Wonder, the answers were, well, somewhat atypical: Wang said he used his to indulge in his hobby of collecting exotic teas from around the world, using the model to source ever more esoteric blends from niche vendors. As for Zuckerberg, at first he struggled to come up with an answer, and then said it was using the model to provide coaching during his mixed martial arts bouts.

I’m sorry, but if you are trying to sell the American people—or even Meta’s investors—on the value of AI technology, maybe come up with some more prosaic use cases than tea collecting and improving your MMA leg holds. What ever happened to using AI to cure cancer? Or, at the very least, using your personal AI assistant to plan your kid’s birthday party (which is the use case that both Microsoft and Google kept rolling out a few years back, at the dawn of the generative AI era, presumably because it felt relatable to many consumers.) Any way, you can watch the podcast appearance here.