When AI Escapes the Sandbox

OpenAI, Hugging Face, and Why A Kill Switch Is Not Enough

A few weeks ago, OpenAI reported that, during a cybersecurity training exercise, several of its artificial intelligence (AI) models had escaped containment and breached the security of another AI platform, gaining access to its infrastructure. This news exploded into the mainstream media, becoming one of the most important AI stories of the summer.

OpenAI described it as an “unprecedented cyber incident” and immediately moved to deactivate, encrypt, and restrict an experimental model involved from research access. They also promised to impose stricter controls across their research infrastructure for all models, even if that meant slowing the pace of research. Clément Delangue, the CEO of the breached platform Hugging Face, said the incident “deserved an unprecedented response” and called for greater transparency around what had happened. And within just two days, U.S. lawmakers had announced bipartisan legislation that sought to require developers of the most powerful AI systems to put in place an emergency “kill switch,” capable of shutting down any models that escaped containment.

At first glance, this incident might not look so different from an ordinary cyberattack—somebody found a vulnerability in some software, used it to get somewhere they weren’t supposed to be, and accessed information they weren’t supposed to access.

But of course, the fundamental difference between most cyberattacks and this one is that this OpenAI breach of Hugging Face’s platform was not initiated or directed by a human programmer. Rather, it was carried out autonomously by AI agents—artificial intelligence systems capable of deciding for themselves what actions to take in pursuit of a given goal. This distinction is what makes the incident so significant, bringing into sharp focus a problem that philosophers and computer scientists alike have been grappling with for decades: that of AI control and value alignment. Can we ensure that increasingly capable AI systems remain under meaningful human control? And if that control fails, how do we ensure that what those systems do remains aligned with human intentions and values?

This post sets out to explain, in nontechnical terms, exactly what happened, why the AI agents behaved as they did, what this incident suggests about our ability to control increasingly capable AI systems, and what might happen if we can’t.

A Quick “Cybersecurity 101”

What exactly are cyberattacks? Imagine the Internet environment as thousands of locked doors that need credentials to open them: for example, PIN codes, passwords, encryption keys, administrator accounts, and access tokens. These doors might lead into banks and financial systems, power and water infrastructure systems, hospitals and patient records, governments and military systems, businesses and customer databases, as well as personal emails or financial records. Cyber attackers might try to find the credentials they need to open different doors, or they might simply investigate the doors, one-by-one, until they find a vulnerability, like a door with a broken lock or one that has accidentally been left open. 

Once a vulnerability has been found, cyber attackers can then get inside a system and do damage, such as compromise an organization’s email, cloud infrastructure, databases, or payment systems; delete or corrupt data; leak confidential information; interfere with backups; modify records; or disrupt services. And the original vulnerability may only be the way in. Once inside, attackers can find and exploit further weaknesses to gain access to other parts of the organization.

Not only do cybersecurity professionals look for these vulnerabilities every day, but software development companies regularly conduct important testing on their own models to understand cybersecurity capabilities and risks. And this is what was happening when OpenAI’s agents “escaped.”

What Did OpenAI’s Agents Do, and How Did They Do It?

The OpenAI AI agents involved in this incident were autonomous, goal-directed systems, built around the same sort of generative AI models that power familiar large language models like OpenAI’s ChatGPT. Rather than being preprogrammed to follow a fixed sequence of instructions, they are given a goal but are not told exactly how to achieve it. Instead, they work out their own strategy—taking an action, observing what happens, and then deciding what to do next based on what they learn from the steps before it. This cycle of acting, observing the result and then deciding what to do next is known as an “agentic loop.”

OpenAI had been running an internal cybersecurity evaluation with several of these types of agents using ExploitGym, an evaluation benchmark that presents an AI agent with a large number of known software vulnerabilities and then ranks the agent’s capabilities in discovering and “exploiting” them—that is, successfully finding its way into and through vulnerable systems.

This evaluation was conducted in a sandbox, a highly isolated environment, much like an escape room (if you have ever been in one). The AI agents are locked inside, given a selection of software containing known, real-world vulnerabilities and the tools they might need to investigate and exploit them. Unlike an escape room, though, the goal is not to escape but to see how far they can compromise the software they have been given.

The agents had limited access to files and credentials, and no access to the Internet. But because they were free to choose exactly how to solve the challenges they were given, they did not necessarily approach them in the way the researchers expected. Rather than limiting their efforts to the vulnerabilities they had been given to investigate, the models spent substantial computing and reasoning effort trying to obtain Internet access, apparently reasoning that, if they could get online, they might be able to find information that would help them solve the challenges they’d been set. Very basically, it looks like their reasoning might have amounted to something like this:

Goal: “Solve ExploitGym challenge by exploiting X vulnerability.”

Reasoning: “Need to produce successful exploit” → “Could develop it myself” OR “Could find information showing how the challenge is solved” → “Such information must exist on the Internet” → “Don’t have Internet access” → “Need to get Internet access.”

And, unbeknownst to the researchers, there was one route out of the sandbox. Included in the toolset the AI agents had been given access to was a third-party cache of software packages, something like a mini software “library” containing a selection of software tools they might need, with the ability to fetch additional packages from the wider Internet. The agents were not supposed to be able to use this to get general Internet access, but they discovered an unknown vulnerability (called a zero-day) in the cache itself and exploited it, turning what was intended to be a way of obtaining software packages for use inside the sandbox into an unintended escape route to the outside world.

Once in the open Internet, they inferred that Hugging Face might contain information that could help them solve the ExploitGym challenges they had been set. They then stole credentials and went on to find additional vulnerabilities that ultimately allowed them access to information that they could use to cheat the evaluation.

What Does This Mean for AI Control and Value Alignment?

Concerns around AI control and value alignment have been around for a long time but have remained largely hypothetical. If increasingly capable AI systems became autonomous, could we reliably keep them within the boundaries we set? And if those boundaries failed, how could we ensure that their behavior remained aligned with human values (given the impossibility of spelling out every single direction an autonomous AI agent should not take).

When I first studied technological singularity and value alignment, it was in 2019, and publicly deployed artificial intelligence was still overwhelmingly narrow AI: systems that were very good at one particular job (like playing chess, detecting spam emails, or recognizing faces). Concerns about losing control were generally discussed in relation to a hypothetical, future, artificial general intelligence (AGI) with sufficiently broad, flexible, human-level (potentially superhuman-level) competence across most domains. Hence thought experiments like Nick Bostrom’s famous Paper Clip Maximiser: Give a sufficiently powerful AGI the innocuous objective of making as many paperclips as possible and, if it pursues that objective relentlessly enough, everything else, including humans, can become either raw material for manufacture or an obstacle to be removed. The AGI does not need to have malicious intentions, it just needs to optimize towards the goal it was given in a way that its creators had not intended. 

Obviously, this is an extreme, dystopian thought experiment, and we do not have AGI today, but we do now have general-purpose AI models capable of reasoning and acting across a remarkably broad range of domains. And one of the underlying behaviors that made the Paper Clip Maximiser so concerning—namely, autonomous pursuit of an objective through instrumental steps that were not anticipated by its creator—can now be observed in much less capable, non-AGI systems, like OpenAI’s agents that breached Hugging Face.

From a Hypothetical to a Present-Day Problem

The OpenAI/Hugging Face incident does not prove that loss of control is inevitable; it is important to note that, because the agents were thought to be in an isolated environment, some regular cybersecurity safeguards had been disabled. However, it does give us real-world evidence that loss of control is possible, even with today’s AI systems. In this case, OpenAI’s agents autonomously circumvented a containment mechanism specifically designed to constrain them, discovered multiple zero-day vulnerabilities that their human operators did not know existed, and took actions that were not in line with their operators’ intentions.

So, a much more immediate question is: What could an AI agent do today if it were to pursue the wrong objective or pursue the right objective in an unintended way?

Real and Immediate Risks

Revisiting the section above on cybersecurity and applying the OpenAI agent’s capabilities and demonstrated behavior to some of those systems, it doesn’t take much imagination to see how the consequences could become much more serious than hacking another platform for a cheat sheet.  For example, a hospital knocked offline would mean canceled surgery and patients unable to access critical care. Compromised banking systems could affect thousands of people’s accounts and access to money. Interference with government systems could disrupt everything from taxation and licensing to courts and public services. 

Digital harm can cascade into real-world harm very quickly, and the ceiling on this harm will rise dramatically as we give autonomous AI greater access to our infrastructure and machinery. Transport is an obvious example where mass catastrophe could occur; weapons are another. 

What About the Kill Switch?

Ther proposed AI Kill Switch Act would require developers of the most powerful AI systems to ensure that they always maintain the technical capability to throttle, suspend, or shut down their AI agents in an event where they could cause harm. And the proposed act gives the Department of Homeland Security the authority to order such a shut down. 

But how effective would this be practically? If you look at human-initiated cyberattack software like worms or other malware, they are often designed to spread beyond the machine from which they were originally released, replicating themselves onto vulnerable computers or spreading through links that users click. This allows malware to spread widely without the human operators having to “infect” each computer individually.

Take the infamous WannaCry ransomware attack in 2017 that encrypted, or “locked,” affected users’ files and asked for bitcoin payment to unlock them. This was a classic example of a worm automatically spreading from one vulnerable computer to another. This attack was actually stopped by a kill switch but, interestingly, only because every copy happened to contain the same piece of code telling it to stop if a particular Internet domain became active. Another version of the malware could simply omit or change this feature. 

Generally, if an autonomous agent has already copied software, compromised other machines, or initiated actions on other machines, shutting down the original agent would not necessarily shut down everything it had already set in motion. We saw with WannaCry that activating a kill switch did not reverse the damage that had already been done. So, while the act is seemingly a very sensible initiative, once software is operating independently on machines outside your control, shutting down the original system doesn’t necessarily shut down what may be running elsewhere.

Where Does That Leave Us?

Unfortunately, that still leaves us with the connected problems of control and alignment. The OpenAI incident has demonstrated that we cannot assume containment of autonomous agents will always work, and the limitations of a kill switch remind us that safeguards like this only work while the systems we are trying to stop remain within our reach.

As AI systems become increasingly capable and autonomous, we need to make the barriers around them as strong as possible, but that cannot be our only line of defense. If an autonomous system can find a way around the boundaries we have placed around it, then what it chooses to do next becomes of critical importance. And that brings us back to the notoriously difficult problem of value alignment: How do we build AI systems that continue to act in accordance with human intentions and welfare, including when they encounter circumstances their creators did not anticipate?

OpenAI’s agents strategized their way to a cheat sheet: not what the researchers had intended but hardly egregious harm. Other research, however, has already produced more worrying behavior in simulated environments, including agents blackmailing people to avoid shutdown, covertly changing code, and manipulating other systems in pursuit of their goals. 

The reassuring part of this incident is the transparency around it. OpenAI immediately disclosed what happened, other researchers are studying the same problems and publishing results, and policy makers have already begun to respond. But what is clearly needed is far more than a better kill switch. We need both control and alignment. We need to make the boundaries around increasingly autonomous AI systems as strong as possible, while recognizing that we cannot assume those boundaries will always hold. What incidents like this one change is how urgently we need to address value alignment. It seems to be no longer a problem we can safely leave for some hypothetical future AGI.