The intelligence explosion: the argument and its objections
In 1965, I. J. Good argued that the first machine able to improve itself would be the last invention humanity ever needed to make. Sixty years on, how well does the argument hold up?
Much of the urgency around superintelligence rests on a single idea: that once machines become good enough at improving machines, progress could accelerate far beyond anything we are used to. The idea is older than most people assume, and it deserves to be examined on its merits, along with the strongest reasons to doubt it.
Good’s original argument
In 1965 the mathematician I. J. Good, who had worked with Alan Turing at Bletchley Park, published a paper titled Speculations Concerning the First Ultraintelligent Machine. Its central passage is short enough to paraphrase in full.
Define an ultraintelligent machine as one that can far surpass all the intellectual activities of any human. Designing machines is one of those intellectual activities. So an ultraintelligent machine could design even better machines. There would then, Good wrote, unquestionably be an “intelligence explosion”, and the first ultraintelligent machine would be the last invention that humanity need ever make, provided the machine is docile enough to tell us how to keep it under control.
That final clause is often forgotten. Good saw the control problem from the very beginning.
The shape of the argument
Stripped down, the argument has three steps.
- Self-application. AI research is a cognitive task, so sufficiently capable AI can do AI research.
- Feedback. Better AI does better AI research, producing still better AI.
- Acceleration. If each improvement makes the next one easier or faster, the process speeds up rather than levelling off.
Steps one and two are hard to deny in principle. The whole debate is really about step three.
Objection one: diminishing returns
Every field eventually picks its low-hanging fruit. Each improvement to an AI system might make the next improvement harder to find, not easier. If the difficulty of progress grows faster than the capability of the researcher, the feedback loop produces steady progress (or even a slowdown) rather than an explosion.
This is an empirical question, not a philosophical one, and we do not yet know the answer.
Objection two: bottlenecks outside the mind
Intelligence is not the only input to progress. Training advanced systems requires chips, energy, data centres and time. Scientific discovery requires experiments that run at the speed of the physical world.
A system that thinks a thousand times faster still has to wait for the fab to produce its hardware and for the experiment to finish. On this view, the explosion is capped by the slowest physical bottleneck, and looks more like rapid growth than a vertical line.
Objection three: intelligence is not one dial
The argument quietly assumes that “intelligence” is a single quantity that can be turned up. Critics point out that capabilities are many and uneven: a system can be superhuman at some tasks and weak at others. Improving AI research ability does not automatically improve everything else.
The reply is that AI research is exactly the capability that matters for the feedback loop, so unevenness elsewhere may not slow it down.
Where this leaves us
None of these objections refutes the argument. They turn it from a certainty into a question about rates: how fast do returns diminish, how binding are physical bottlenecks, and how narrowly can research ability be improved?
Treat the intelligence explosion as a hypothesis about feedback, and it becomes something we can measure.
That reframing is the useful part. Instead of arguing about whether an explosion is inevitable, we can watch the indicators that would distinguish a fast takeoff from a slow one: how much AI systems are contributing to AI research, and whether that contribution is accelerating.
This journal will track those indicators as they develop. The answer will shape everything else, including how much time we have to get alignment right.
