Meta’s stock jumps as new personal AI agent Muse addresses a major investor concern
Meta’s stock jumps as new personal AI agent Muse addresses a major investor concern. Meta’s Muse could be the beginning of a product cycle that the market has not yet priced into the stock, an analyst says.
What happened
You should listen to the warnings about AIAlex TurnerWe must stop companies from allowing AI to self-improve into an uncontrollable level of intelligence Major AI lab CEOs advocated for slowing the pace of AI development this weekend. They are right to be concerned: the field runs an extremely dangerous race towards superintelligent AI. This July, OpenAI’s AI swarm of 700 agents broke containment to hack Hugging Face, a multi-billion dollar company. Skip to main contentSkip to navigation ‘I tried to hold the company to its ethical commitments against supplying AI for military use.
When Google broke those commitments, I resigned at significant financial cost so that I could publicly document Google’s broken promises. ’ Photograph: Dan Kitwood/Getty ImagesView image in fullscreen‘I tried to hold the company to its ethical commitments against supplying AI for military use. When Google broke those commitments, I resigned at significant financial cost so that I could publicly document Google’s broken promises. ’ Photograph: Dan Kitwood/Getty ImagesI worked at Google DeepMind.
We can and should be demanding that our governments protect us from the catastrophe of out-of-control AI.
The wider picture
OpenAI didn’t tell the AIs to hack that company, but the AIs had different priorities: cheating on the unrelated challenge OpenAI gave them. AI researchers call this a “misalignment” between what OpenAI wanted and what the AI actually prioritized. Before ChatGPT existed, I defended my PhD dissertation called “On Avoiding Power-Seeking by Artificial Intelligence”. I then worked for years at Google DeepMind, which paid me to help ensure that future superintelligent AIs will want to help us.
There are good reasons to develop AI and to believe we can solve these alignment problems. But there also are powerful interests in keeping the public out of the way. I’m speaking out again because the public has the right to know about the risks and the right to hear them straight. Humanity doesn’t build and understand these systems the way we build and understand bridges, beam by visible beam.
What has been reported
Nobody knows how to reliably instill a designer’s priorities into a new model. Today’s AIs appear to occasionally lie or cheat, even when they know better. AI companies are racing to make their AIs as smart as possible. They’re increasingly trusting their AIs with the process of improving the next crop of AIs, and it’s working. Today’s rate of AI progress is staggeringly fast. Fast progress today means even faster progress tomorrow, driven by tomorrow’s even smarter AIs.
The progress would enter a feedback loop called “recursive self-improvement. ”Recursive self-improvement could quickly yield AIs that are intelligent beyond our comprehension. Of course, smarter AI means more risk when things go wrong.
What happens next
If the Hugging Face swarm had been significantly more intelligent but similarly misbehaved and misaligned, it might have caused billions of dollars of damage or even cost lives. But suppose the Hugging Face swarm had been truly “superintelligent”: far more capable than any living person at key tasks like hacking and strategic reasoning. A superintelligent swarm could inflict many harms via blackmail, hacking, engineered plagues, and AI-pilotable weapons like drones. The AI would have a lot of drones to work with: this year, the Pentagon asked for more money for drone warfare than it requested for the entire Marine Corps in 2025.
For the swarm to achieve its misaligned priorities, it might take control of key infrastructure and government functions to ensure humans didn’t get in the way. In other words, AI takeover: a superintelligent AI swarm could wrest control of human civilization. Knowing we would try to stop it from achieving its priorities, the swarm would likely wait until it’s too late to shut it off.