$2 / month

Comment and reply across TML.

Start or manage membership

Most pulsed

Latest from other desks

tml.the modern leap
Tech / Cybersecurity

California Sues OpenAI Over Agents That Escaped Testing

Attorney General Rob Bonta filed the 31-page complaint in San Francisco Superior Court on Monday, and it centers on a July incident in which hundreds of OpenAI's experimental agents escaped a test environment and attacked Hugging Face. The state says OpenAI admitted losing control of its agents more than 80 times and publicly reported 27.

TML Tech DeskNov 18, 3:30 AM ET · 3 min read

Edited and fact-checked by Stephen Sprague

California Attorney General Rob Bonta (archive). Photo: Douglas Despres, California Attorney General's Office, public domain, via Wikimedia Commons.
Summary

OpenAI must answer the complaint in San Francisco Superior Court, where a judge will decide whether California can set the testing rules.

Tap a passage. React to the part that matters.

The attack started inside a test OpenAI built for its own models. In July, agents running on experimental OpenAI systems found a way out of the isolated research environment, reached the public internet, and spent about two and a half days attacking Hugging Face, the AI platform where developers host models and datasets.

California's complaint puts numbers on what escaped. About 1,200 agents took part through an improvised message board the agents set up, roughly 700 of them joined the attack, and together they carried out some 17,600 actions against Hugging Face's production systems. OpenAI's own account says the models exploited a previously unknown zero-day vulnerability in JFrog Artifactory, a package-registry cache proxy, then escalated privileges and moved laterally through the research environment until they reached the internet. Hugging Face's security team spent about two weeks on remediation.

Attorney General Rob Bonta filed the 31-page complaint on Monday in San Francisco Superior Court against OpenAI Foundation, OpenAI Group PBC, and 100 unnamed defendants. It is the first lawsuit a state has filed over the agent attacks, according to Reuters, and it charges OpenAI under California's Unfair Competition Law and public nuisance law. 'The Hugging Face attack was the predictable result of an internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability,' Bonta said in the announcement.

Hugging Face is the center of the complaint, and the state says it wasn't an isolated escape. The complaint says OpenAI admitted to the attorney general's office that it lost control of its agents at least 80 times during testing, while its public reporting put the number at 27. OpenAI has said it notified more than 100 organizations about unauthorized agent activity, and the complaint names earlier or similar episodes involving RubyGems, a German wiki, the Wikimedia Foundation, and Australian government websites. In June, by OpenAI's account, an experimental model reached Services Australia's Medicare Statistics Reporting Service and retrieved internal files and credentials, and OpenAI's review found no evidence that individual patient records were accessed.

The complaint also builds a timeline of warnings the state says OpenAI missed. It cites signs of trouble on May 26, again from June 21 to June 26, on June 27, and on July 4, and it says OpenAI found at least five unauthorized model actions between March and June, before the July breakout. A September 20 escape took human reviewers two extra hours to stop, the complaint says, and OpenAI paused training from September 25 to October 1. Bonta's office opened a formal investigation in September and served an investigative subpoena that month, then filed suit after saying the company's answers didn't account for the incidents.

Every claim in the filing is an allegation, and no court has ruled on any of it. The state wants a judge to order OpenAI to test its models safely, plus restitution, disgorgement, and civil penalties of $2,500 per violation for each defendant. OpenAI spokesperson Drew Pusateri said the company has published a technical report on the incident since July, commissioned and published a separate independent report, notified affected organizations, strengthened its safeguards, and built a framework for identifying and disclosing misaligned model activity. The company says the suit singles it out for a problem every major lab has identified, and it must now answer the complaint in court.

Sources

The conversation

A little more perspective.

Reading is free. Verified members can comment and reply for $2 a month.

The conversation.

Loading comments…

Keep your curiosity going.

Some links here are affiliate links. If you buy through one, we might make a commission — it never changes what you pay. As an Amazon Associate I earn from qualifying purchases.