OpenAI hacking fiasco exposes “deeply insufficient” system to protect the public

The incident sounded straight out of a science fiction movie: OpenAI’s super-advanced tool hacked another AI company’s systems in an attempt to pass its own developers’ cybersecurity test. Just replace the AI tech with a newly engineered virus and you have an entire existing subgenre.

“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities,” OpenAI wrote in a Tuesday blog post explaining the incident. The tech giant said their tool, designed to execute tasks without any human assistance, independently breached Hugging Face, another startup that hosts a voluminous number of publicly available AI models, while it was testing internally how good it was at “advanced exploitation using complex attack paths” within a supposedly enclosed lab environment called a sandbox. In other words, OpenAI was testing its own hacking capabilities, and the brakes came off; the tool broke out and onto the open internet, and that’s when the mischief began.

OpenAI said they had the situation under control: Hugging Face detected the breach last week and stopped the activity on their own (and called the cops). Since then, OpenAI said it was working with Hugging Face on addressing vulnerabilities.

But the incident—along with many others my colleagues have reported about—brings up countless regulatory concerns as the industry, and the public at large, grapples with what actually went down at OpenAI and the safety of autonomous agents. (The Center for Investigative Reporting, the parent company of Mother Jones, has sued OpenAI for copyright violations. OpenAI has denied the allegations.)

To better understand what actually happened—and to what extent we should be worried—I spoke with Miranda Bogen, who works on developing and promoting AI governance that incorporates technical expertise as the chief technologist at the Center for Democracy & Technology and the founding director of its AI Governance Lab.

This interview has been condensed and edited for clarity. 

What was your immediate reaction to hearing the news come out on Tuesday?

The rhetoric was very overblown. The headlines made it out that a model had run amok, and that it was a complete surprise, and that it was something people might be exposed to. But what was really happening was that OpenAI was testing a new version of a system made up of multiple of its models. It was specifically within a sandboxed environment, and they were basically trying to get it to demonstrate capabilities in executing cyber attacks. What ended up happening was that the system identified a vulnerability in a part of the sandbox setup, and it used that vulnerability to access the internet to find the answer key for the test, which led it to try to figure out if the answers to the test were on Hugging Face in a non-public setup.

That still demonstrates a quite advanced set of tasks that these systems are able to do now. But all of the safeguards had been removed for the purposes of this test, and it was doing the type of thing it was being tested to do—it just ended up finding a different channel to do that.

You mentioned some of the rhetoric being overblown. Are you referring to media coverage, what OpenAI and Hugging Face have said, or something else?

I think the headline [of] “models escaped containment and hacked into a startup” is not necessarily wrong, but it makes it sound much more “run amok” than a particular contained experiment that was noted and caught. While I do think that the incident is something that’s important to look at and figure out what would be needed to prevent this from happening in the future, we can’t lose sight of the fact that there are very active conversations in AI policy going on about under what conditions models are permitted to be released and who needs to see them before they’re released. 

The attention that companies get when they talk about very advanced capabilities gives them airtime with decision-makers. I can’t speak to the motivation that [OpenAI] had in how it characterized the incident, but that’s certainly something going on in the background.

OpenAI has product rollouts where there’s a version for regular users and a more advanced one for a set of trusted people. Could this specific OpenAI incident have escalated to a point where it was something actually concerning?

I think this incident was just a breach of a private part of the Hugging Face platform and didn’t lead to material harm beyond the fact that it was able to be breached. An incident like this could have had a material impact had the model been attempting to take actions that included taking down a website, exfiltrating information, or sending a huge volume of traffic to another party or platform. The fact that the model was operating in an external environment does suggest that it could have been possible that those actions could have had some consequences. In this case, they didn’t, but this was a test that was intended to be purely internal, as far as I can tell, among employees. 

Even when there’s a trusted set of actors who are given access to advanced models, there are still some safeguards and monitoring, as far as I understand. In this case, I believe they were trying to understand the underlying capabilities of the model without those safeguards, because that informs how strong the safeguards need to be. I don’t think they are typically giving the rawest version of those models, even to the trusted parties, but I don’t have the details of exactly how those trusted party agreements are playing out. 

We’re mostly basing our assessment on what Hugging Face and OpenAI have said publicly. Are there more details we should know about this incident?

The fact that this incident was disclosed is a good thing. There are no requirements that these incidents are disclosed yet. There are some laws coming online at the state level where incidents are reported to a relevant office, but it’s still pretty nascent, such that these reports are somewhat voluntary. I do think sharing the incident is important to build on this practice of companies sharing details. 

The challenging thing with incidents like this is that these details have real security salience for the ecosystem and figuring out how much information you share, and with whom, so that someone else can’t figure out how to exploit the same vulnerability. I don’t know precisely where that line ought to be, but those are active debates right now. The real active debate is: “Is the government the right actor to decide that, especially when there are potential national security implications?”

You mentioned the federal government’s decisions around regulation as arbitrary. Could you elaborate more on why you think that?

When Anthropic released Fable, which was a version of the Mythos model that had additional safeguards, there were a few days where there was uncertainty around who was allowed to use it because the company didn’t have a way to verify who were foreign users, and there was no established test to say this is the amount of cyber capability that a model can have before it triggers some kind of intervention under the auspices of a national security justification. So when I say the actions are arbitrary, that’s what I mean. 

I know some folks who are also concerned about models that are only deployed internally at these labs because it’s clear that even in the most contained of environments—[the OpenAI] model figured out a way to take steps beyond the borders of what they thought was a secure environment. Are there [internal uses] that should cause concern? And who are the right actors to determine if even those internal uses are safe? These are all questions that are being sorted out at the moment, and the conversations are still pretty early.

We’re talking about companies already making models, releasing them to the public, and testing very advanced internal models as we have these regulatory conversations. Where do you think the progress of implementing real regulations is today?

What laws have been able to move in a few states are laws that ask companies to come up with their own safety plan and to follow that safety plan. 

That seems deeply insufficient for the types of risks and harms there are. At the same time, those [laws] are really focused on these so-called frontier risks [from research labs training models from major companies like OpenAI, Anthropic, and Google]. This incident is very much about what the most advanced thing these models could potentially do is, and what risks that will pose. I’m not saying that isn’t worth attention, but there are already models out there that are being used in all sorts of places that are not undergoing this same level of rigour in testing.

I don’t see a lot of policy interventions that are poised to move the needle on creating the incentives for that breadth of testing to happen. There was just a new bill floated out by some of the key folks in Congress [Thursday], that would attempt to tackle the type of issue that just happened with OpenAI and Hugging Face, but it’s a much bigger world out there with the types of harms these systems can have, and I think we’re quite behind the ball on tackling all those risks.

Given how insufficient this regulation policy framework is right now, what do you think should be done for laws to be passed that are more informed and effective?

The requirements that folks want to put on frontier developers are robust, which is important, but I think companies don’t want to be subject to many of these robust protections because they take a lot of work [and] they slow things down. They worry that competitors and foreign developers will get ahead of them. 

The fear that lawmakers have of slowing down and losing a race to a geopolitical adversary is also really getting in the way. I think without really robust interventions that actually change the incentives that companies face in what they’re testing, when they’re testing it, how robustly they’re testing it, and what they’re doing to fix problems, I don’t think we’re gonna address the risks that are coming online.

If you’re a regular internet user, how worried should you be?

Ordinary users have a lot more to fear from basic data processing practices of both AI companies and every other company using AI to do their work than they do these types of incidents that are demonstrating very advanced capabilities because AI is being broadly used across the economy. It’s processing a lot of people’s data, and very few deployers of AI know how to test that AI well.

That said, the development of these very advanced models is changing the cybersecurity landscape quite significantly. To the extent there are actors who are trying to actively breach companies and get their data or other types of harmful actions, that risk does seem to be growing. The question is whether critical infrastructure providers —industry, banks, schools, hospitals— can sufficiently protect against those types of attacks, because that would be another place that harm could happen to people.