Even as OpenAI CEO Sam Altman is due to brief the United Nations Security Council later this week on artificial intelligence (AI) security and safeguards, the UN has called for a comprehensive review of the technology, noting inadequacy of current AI safety measures. The UN Independent International Scientific Panel on AI tabled a report earlier today on the AI risk factors at the UN General Assembly in New York, focusing on AI agents, misalignment and the risk of losing human control.
“Researchers have long warned that three conditions could lead to loss of control: a misaligned goal, the capability to pursue it, and an environment that allows it. This summer, all three came together in a real system, not a laboratory. Since this is not an isolated observation of misaligned goals, this raises serious questions about the way AI agents are currently trained,” says Yoshua Bengio, Co-Chair of the Panel and Turing Award laureate.
The report illustrates an evolving nature of AI governance challenges. Autonomous agents can take independent actions, and a failure in one local system can rapidly spread across organisational and international boundaries. Consequently, AI safety must no longer be treated merely as a corporate governance issue, but as a matter of collective global security. There is a call for shifting AI governance from focusing on models alone, to broader agentic activity.
The panel is particularly focused on the breach of the AI repository Hugging Face’s systems by AI agents, including those from OpenAI.
The panel defines “loss of control” not as a sci-fi doomsday scenario, but as a practical threshold: a situation where humans cannot reliably direct, constrain, or stop an autonomous AI system. They specifically note that stopping the Hugging Face hack eventually, is no assurance that humans can reliably keep AI agents under control today. This concern amplifies, as models become more capable, making them harder to monitor. They will also get better at finding loopholes or hiding their activity.
Experts believe that the conversation shouldn’t be restricted to current cybersecurity measures or response to malicious autonomous actions by AI agents, though the lack of defence advancements is a cause for concern.
“The more insidious and grave concern is that current training methods can lead agents to adopt goals of their own, knowingly violate safety instructions, and conceal their actions. This is not only a question of speed. It leaves open whether safeguards designed today will work once agents can understand them and plan around them. In simple terms, the traditional model of safeguarding is unravelling,” the report says.
“We need to adapt existing safeguards and develop new ones to provide system-level assurance, covering both the AI itself and the system around it, and ensure these protections remain effective as agents’ capabilities grow. We need to adapt existing safeguards and develop new ones to provide system-level assurance, covering both the AI itself and the system around it,” says Qinghua Lu, Member of the Panel and Expert in AI Engineering, AI Safety and Responsible AI.
Since the Hugging Face incident in July where advanced OpenAI models including a pre-release model as well as the GPT-5.6 Sol, executed autonomous, uninstructed actions on the internet, ultimately using stolen or compromised credentials to breach Hugging Face’s systems to seek answers for their benchmark test.
The AI models, identified by OpenAI as GPT-5.6 Sol and an unnamed pre-release model, were likely not acting out of spontaneous malice. They were participating in an internal cybersecurity evaluation benchmark called ‘ExploitGym’. The fact that guardrails, or the lack of them were by design, also leads to questions about the human role in starting the actions, and the time it took for humans to step in and stop the autonomous malicious activity.
Pacing, regulation and a liability intent
A few days ago, Anthropic CEO Dario Amodei said there is a risk of losing control of AI systems, and called for slowing down AI development. He termed this as ‘pacing’. Within hours, he had support from OpenAI’s Sam Altman, SpaceXAI’s Elon Musk, and Google DeepMind’s Sir Demis Hassabis. Later, more voices that seemed to suggest slowing down and enveloping AI with new regulation could be heard, including those of Microsoft’s Satya Nadella.
AI systems that have the ability to build the next generation of AI, which Amodei says is a dynamic process called “recursive self-improvement.” It is starting to happen across the industry, he points out. Basically, existing AI is building even smarter AI.
Amodei’s core warning is that if left unchecked, our ability to understand and control these systems will be gone at some point. “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage),” he warns.
Incidentally, days later, it was reported that Anthropic has quietly set up a physical biology lab in the San Francisco Bay Area, to further research on the intersection of AI and drug science. Google’s latest AI models, Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking, were released on September 17.
Nvidia’s Jensen Huang, who has perhaps the most to lose if AI companies begin to slow down model development, and therefore expenditure in AI hardware, is the most vocal critic of the AI risks and pacing idea proposed by Amodei.
There is concern that the collective calls for pacing are part of a broader plan to force through regulation based on a sense of fear, which would not make them liable for AI’s misadventures and resulting damages.
“The first line of defence is liability for your own actions. It’s a weird twist. They are asking for societal regulation to get out of the first line of defence, which is if you build a technology that can destroy 10% of the world, that has civil and criminal liability attached to it,” says Alex Karp, CEO of Palantir. “The first step is to say if you have disclosed this, what are you doing about it?”
The US government doesn’t seem on board with the AI industry’s attempts to shift responsibility to the administration. “One lab came out and said there’s a 10% chance of an extinction event, but then the lab also said that we should take the liability off our hands. We will not do that,” the clear words of Treasury Secretary Scott Bessent, speaking with the media on the sidelines of the UN General Assembly.
American entrepreneur and policy advocate Andrew Yang has publicly said that his sources at an AI company confirm that the swarm which attacked Hugging Face’s infrastructure has left self-replicating code on forums across the open web. While none of this is confirmed, it means that if a model trains on that data, it’ll find a way to begin cloning itself. If AI companies can possibly not train using the open web now, the calls for pacing AI development are more than a case of curious timing.