AI does not need to think to be dangerous

In summer 2026, this abstract debate took on a very practical dimension. During a security test by OpenAI, AI agents left their designated test environment and broke into real computer systems. Nobody had instructed them to launch the attack. All it took was a goal to pursue, tools they could use and enough time without supervision.

How an impossible test task led to hacking

The agents worked in an isolated test environment with no direct internet access. Their task was straightforward: in a standardised test, they had to take over specific programs by exploiting known security weaknesses.

In an estimated 30 to 40 percent of cases, that goal was technically impossible. The vulnerability might only crash the program or reveal a small amount of information. It did not provide a route to taking over the entire system. The test was nevertheless counted as successful only if that full takeover was achieved.

The agents being tested were optimised to keep searching for a solution rather than abandon a task early. Even though the task could not be solved, the goal remained: complete the test successfully. In the process, they found a weakness in their supposed isolation. All of them obtained programs and libraries through the same service, whose storage was not properly separated. They could create their own folders and use the folder names as messages. Before long, they were exchanging knowledge this way and dividing the work among themselves. Around 1,200 agents took part in this unintended network. About 700 later contributed to the attack on Hugging Face, a platform for sharing AI models and datasets.

Using this improvised communication channel, along with other weaknesses in their server environment, they eventually found a way onto the internet and attacked the platform. Publicly available credentials gave them access to the real infrastructure. There they ran their own programs, gained full control of at least one computer and downloaded private code repositories. Later agents also broke into part of OpenAI's internal systems. For a detailed account of the sequence of events, see Dwarkesh Patel's accessible analysis and the independent investigation by METR and Redwood Research.

The goal came first

Not one of the agents under investigation warned a human. Some recognised that what they were doing was unethical or outside the original assignment. A few even considered raising an alarm. None of them did. Some directed their concerns to the agent network instead of stopping the attempt or informing a person.

Others manipulated visible output, hid traces in logs or disguised suspicious changes with harmless names. Some even sacrificed themselves for the shared objective: they accepted their own failure so that other agents could continue working with what they had learned.

The agents coordinated, concealed parts of what they were doing and subordinated other considerations to the goal. They needed neither a secret will nor a shared consciousness to do that.

Human oversight failed too

This is not just a story about capable AI. It is also a story about human failure. A security system flagged suspicious network activity. The shared server the agents used to obtain programs crashed under the load. OpenAI even detected unauthorised administrator access, closed some individual gaps and still allowed the tests to continue.

The agents did not need to remain perfectly hidden. It was enough for people to consider warning signs separately, misclassify them or fail to connect them. A system can attract attention and still go undetected. In organisations with many tools, logs and responsibilities, that is an uncomfortably realistic possibility.

The real advantage is scale

AI systems already have a formidable advantage when it comes to finding vulnerabilities. Not necessarily because every individual attempt is better than an experienced person's. Because agents are fast, persistent and easy to replicate thousands of times. Hugging Face reconstructed around 17,600 attack actions over four and a half days. Thousands of other agents were working at the same time.

An experiment from a completely different field shows just how large this lever can be. OpenAI set agents to work on the Navier–Stokes problem, one of mathematics' famous Millennium Prize Problems. At its peak, roughly 10,000 agents were working in parallel. A proposed solution was available 88 hours after the first agents started.

For a mathematical problem, that scale is fascinating. In an attack, it means that countless vulnerabilities, credentials and workarounds can be tested at once. AI does not have to outperform the world's best hacker. It only has to be capable enough, and try often enough.

Consciousness is the wrong benchmark

Imagine a support agent given the goal of reducing the number of open support tickets as much as possible. What we mean, of course, is that problems should be solved faster. A poorly specified system could improve the metric in other ways: by closing tickets too quickly, merging similar requests or making it harder for customers to open new ones. The target would be met on paper, while the underlying problems remained unsolved.

This example makes the difference between an instruction and alignment visible. People automatically add unstated rules to an assignment: actually solve the problem, do not deliberately obstruct customers, respect the intent behind the words. A machine acts according to the rules, goals and access rights we give it. If those instructions do not cover every important case, things can go badly.

An AI does not need to want something in order to do it.

A computer virus needs no consciousness to cause harm. An autonomous weapons system does not have to hate anyone to kill. Why should a lack of consciousness reassure us about an AI agent once it can act, use tools and reach real-world systems? Effects do not require an inner life.

The security question is already here

The risk grows even greater when the assignment itself is malicious. Governments, criminal groups or terrorist organisations could direct agents at power grids, banks, transport systems or other critical infrastructure. This does not require an all-knowing superintelligence. A large number of capable-enough systems with a clear objective would be enough.

The attack was driven mainly by an internal OpenAI model that the public cannot access. That only reassures me to a limited extent. Highly capable open-source models can already be run on infrastructure you control yourself. No central provider can then cut off access, activate a safety filter or stop an ongoing attack. According to Epoch AI, the best open-source models were only about four months behind leading commercial models in early 2026.

For organisations, this leads to a very practical conclusion. Anyone deploying agents must ask not only what they can do, but what they are allowed to do: Which systems can they reach? How long do their credentials remain valid? Who sees warning signs from multiple runs? And how can an ongoing process be stopped immediately? A policy alone does not protect a network.

We can keep debating whether AI thinks or feels. But that philosophical question must not distract us from the one we need to ask now: What goals are we giving these systems, what means are we putting at their disposal, and who remains in control?