King Charles warns of ‘existential danger’ of AI falling into wrong hands

King Charles warns of 'existential danger' of AI falling into wrong hands. Image source, ReutersByLaura Cress, Technology reporter and Shiona McCallum, Senior technology reporterPublished17 September 2026King Charles has warned AI executives of the "existential dangers" posed by the technology falling into the wrong hands.

What happened

He said there seemed to be an "urgency" in "adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways". Related topicsArtificial intelligenceUK Royal FamilyKing Charles IIIMore on this storyOpenAI boss says world 'right to be afraid' but should trust AI firmsPublished1 day agoQuestions mount over what an AI 'slowdown' would look likePublished4 days agoWhy some experts increasingly fear AI will take overPublished10 September "Those who have created these technologies are now increasingly warning that AI risks developing darker capacities – perhaps even to take life," the King told the executives.

The chance of something going wrong was "definitely non-zero", Sir Demis added. "The future is not yet written," he told the King's delegates. "Uncontrolled AI could lead to 'silicon species' rivalling humans, warns MicrosoftPublished21 hours agoWhy are there concerns AI could threaten humanity, and how real are they? He later told the BBC the people working on the technology were "genuinely frightened" about the speed of its advancements and what it could mean for humanity.

The wider picture

The comments were made at a summit in Scotland with participants from AI giants such as Nvidia, OpenAI and Anthropic. He made the comments at a summit he had convened at Dumfries House in Ayrshire to discuss how the tech can be used to benefit society. Those gathered at the meeting also included Kanishka Narayan, the UK's AI Minister, the Pope's advisor, and participants from AI giants such as Nvidia, OpenAI and Anthropic. It comes amid a series of recent warnings from people inside the industry about the rapidly evolving capabilities of advanced or "frontier" AI models and the regulation needed to keep them in check.

"The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world," he told the group. The summit has been called to consider whether a "shared set of principles" can be developed to "help guide the future application" of AI. 'Keep engineering'Jensen Huang, chief executive of Nvidia, was among top industry leaders attending. Nvidia is a central player in AI infrastructure, supplying chips to leading AI model developers including OpenAI and Meta.

It is also the world's most valuable company, with a stock market value of around $5.

What has been reported

Huang told the room he believed safety was "paramount", adding that companies should hold back a product and "keep engineering" if it was not safe enough. The Nvidia boss also highlighted the value of open models – systems which give users direct access to the software – as a way to "ensure that people and countries are not left behind". He said these systems must also be protected so "researchers, universities, start-ups and countries" can develop applications "none of us would have imagined alone".

"Let us lead with responsible optimism, build AI safely and securely, open it to more people, expand our ambition and solve the problems that matter to the world," Huang added. Meanwhile Sir Demis Hassabis, who co-founded the UK AI lab DeepMind – later acquired by Google – offered a more cautious view on the pace of development. He said artificial general intelligence – systems it is believed could be as good as or better than humans at multiple tasks – was "probably only a few short years away" and could have an impact "ten times that of the Industrial Revolution".

But he advocated for "a sensible middle way", saying humanity could address risks if collectively given the space and time to mitigate them. Sarah Friar, OpenAI's chief financial officer, said AI had the potential to solve some of society's "hardest problems" – but agreed its growing capabilities raised questions about safety and "how we ensure it always serves people".

What happens next

"We need to address those questions together," she said. "No one company or government can do that alone and I'm grateful to His Majesty for convening this discussion at such an important moment. Published20 hours agoWave of warningsSince August, a wave of discussion about AI's potential power and how to control and regulate it has dominated blog posts and social media, emphasised by several dramatic statements from those in the industry. The debate has also been met with criticism from sceptics who say the focus on AI's hypothetical future risks distracting from more immediate concerns around the technology, such as its environmental impact.

In July, OpenAI announced what it called an "unprecedented" incident in which its advanced AI escaped test limits and hacked into the tech platform Hugging Face. After several other incidents, in September, a resignation post from Anthropic researcher Jacob Coxon went viral when he cited concerns AI could wipe out humanity. In response, Anthropic scientist Evan Hubinger wrote he personally thought the possibility of human extinction from AI was a greater than 10% chance.