Who will control the most powerful mind on Earth?

In May 2023, Geoffrey Hinton left Google. He was seventy-five, one of the researchers whose work on neural networks had made modern AI possible, and he wanted to be free to say something his employer might find awkward. He had come to believe that the technology he helped create could become more intelligent than us, and that we did not know how to control something like that.

A few weeks later, the Center for AI Safety published a statement of a single sentence: Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. It was signed by hundreds of researchers and executives, including Hinton, Yoshua Bengio, and the heads of several leading AI labs.

Not everyone agreed. Yann LeCun, who shared the 2018 Turing Award with Hinton and Bengio, called such fears overblown and argued that we would design these systems to be safe as naturally as we design aircraft to fly. The fact that three founders of the field could disagree so sharply says a great deal about how uncertain the ground is.

Underneath the disagreement, though, there is a question everyone accepts as real. If something far more capable than us is built, who decides what it does?

Three ways it could go wrong

It helps to separate the concerns, because they call for very different responses.

Concentration. Building frontier AI systems requires enormous amounts of computing power, specialised chips, and capital. Only a handful of companies and a few governments can afford it. A superintelligence held by one of them would give its owner an advantage without precedent: in science, in business, in cyber operations, in persuasion. History offers few examples of such power being handed back voluntarily.

Misuse. A capable system can be used by people with bad intentions: to design weapons, to run disinformation at a scale no human team could manage, or to attack the software infrastructure societies depend on. The more capable and widely available the system, the lower the barrier.

Loss of control. This is the concern Hinton raised. A system pursuing goals that differ slightly from ours, and capable enough to anticipate our attempts to correct it, might resist being switched off. Not out of malice. In 2008, the researcher Steve Omohundro pointed out that almost any goal is easier to achieve if you keep running, acquire resources and avoid being modified. A system does not need to hate us to get in our way.

These three risks pull in different directions. Spreading the technology widely reduces concentration but increases misuse. Locking it down reduces misuse but concentrates power. That tension is why there is no obvious answer.

How worried are the experts?

Opinion inside the field is genuinely divided, and it is worth being precise about it. In a large survey of 2,778 published AI researchers, conducted in late 2023 by Katja Grace and colleagues, the median respondent put the probability of an extremely bad outcome, such as human extinction, at around 5 percent. Depending on how the question was asked, roughly 38 to 51 percent of respondents gave such outcomes at least a 10 percent chance.

Those numbers are not predictions. Surveys measure belief, and experts are poor forecasters of technologies nobody has built. But they do settle one thing: the worry is not a fringe position. A large share of the people who build these systems think the downside risk is real.

What governments have done so far

Governments were slow, then suddenly active. In November 2023, the United Kingdom hosted the first AI Safety Summit at Bletchley Park, where Turing’s codebreakers once worked. Twenty-eight countries, including the United States and China, along with the European Union, signed a declaration recognising the potential for serious, even catastrophic, harm from the most capable AI models.

The European Union went further with law. Its AI Act entered into force in August 2024, sorting AI uses by risk and adding specific obligations for the most powerful general-purpose models. The United States restricted exports of advanced AI chips to China from 2022 onwards, treating computing power as a strategic resource.

Each of these steps matters. None of them yet amounts to a way of governing superintelligence. They address today’s systems, and superintelligence would not be today’s systems.

The narrow path

The hardest version of the problem is a race. If several companies or countries believe that whoever builds superintelligence first gains a lasting advantage, each has a reason to move faster and cut corners on safety, even if all of them would prefer to go slowly together.

Escaping that trap probably requires things we have managed only rarely: verification between rivals, the kind of monitoring that arms control treaties rely on, and some shared understanding of which lines should not be crossed. We did it, imperfectly, with nuclear weapons. There, at least, the dangerous material was physical and hard to hide.

The question is whether we can learn to govern a technology faster than the technology learns to outgrow us.

That is where the technical and the political problems meet. Governance buys time, and time is only useful if it is used to solve the harder problem underneath: building systems that reliably do what we intend. That problem has a name, and this journal has written about it in the alignment problem, in plain terms.