Superintelligence is impossible

The essay says the risk of superintelligent AI has been overblown and contrasts two ways of thinking about intelligence. One treats intelligence as a single dial that can keep increasing; the other treats it as adaptation to a niche, where gains in one area bring tradeoffs in others.

It argues that the No Free Lunch principle fits the second view better. On that view, there is no system that can excel at all tasks at once, so true superintelligence would be impossible, while narrower systems and human-equivalent general AI can still emerge.

Reading notes

#

• The text says that the risk of superintelligent AI has been exaggerated and that it is worth looking at this threat more skeptically.
• It presents two ways of understanding intelligence: one as a single scale, close to a “deist” view, and another as evolutionary adaptation to specific niches.
• It revisits the definition of superintelligence as something that would make humans seem like children or ants and that would escape human control.
• It uses the universal intelligence definition of Legg and Hutter to show that, in this reading, intelligence would be a broad measure of performance on many problems.
• It contrasts this with the evolutionary view, in which different intelligences coexist and each one fits a limited set of problems.
• It takes the No Free Lunch theorem as support for the idea that there is no model or algorithm that is best in all situations.
• It states that the same reasoning applies to organisms and to intelligent systems: gains in one context tend to imply losses in others.
• It concludes that there can be general AI at the human level and agents that work well in several tasks, but not a superintelligence with perfect performance at everything.