Gary Marcus is known for his skeptical views on artificial intelligence.
The tech industry is now beginning to realize the harsh truth that generative artificial intelligence (AI) models are reaching technological limits.
As some experts have long predicted, the improvements that were once easily achieved by scaling up large language models—that is, adding more parameters, training data, and computing power—are now slowing down, if at all.
Gary Marcus, a cognitive scientist and AI skeptic, warns that when everyone realizes these limitations, the entire industry could collapse.
“The economic situation is likely to be bleak,” Marcus writes in his Substack. “The inflated valuation of companies like OpenAI and Microsoft is largely based on the idea that large language models will reach artificial general intelligence as they continue to scale.”
“As I’ve always warned,” he adds, “this is just a fantasy.”
The first signs of this came when The Information reported this week that OpenAI researchers had found that its upcoming flagship model, codenamed Orion, showed fewer improvements over its predecessor, GPT-4, than GPT-4 showed over GPT-3.
In areas like programming — one of the main reasons these AI models are so popular — there may not even be any improvements.
This is borne out by other industry sources. Ilya Sutskever, founder of the startup Safe Superintelligence and co-founder and former chief scientific officer of OpenAI, told Reuters that improvements through scaling AI models have peaked.
In other words, the dogma that “bigger is better” for AI models, which has underpinned the industry’s massive growth, may no longer be true.
This is not the end of AI. “But,” Marcus writes, “the economic calculations will probably never add up: additional training is expensive, and the more you scale, the higher the price.”
According to Reuters, training large models can cost tens of millions of dollars, require hundreds of AI chips, and take months to complete. Tech companies have also exhausted the freely available training data, having practically “scraped” the entire freely available web.
“Large language models will become a convenience; price wars will keep revenues low. Given the high cost of chips, profits will be hard to come by,” Marcus predicts. “When everyone figures this out, the financial bubble could burst quickly.”
There may be a way out of this economic impasse, though. According to reports from The Information and Reuters, researchers at OpenAI are developing methods to overcome the scaling problem by training models to “think” or “reason” in a human-like way, capabilities that are already evident in their o1 model.
One way to achieve this is through a technique called “compute-at-test,” in which an AI model considers multiple options for complex problems and chooses the most promising one, rather than making a direct inference.
Whether this approach will pave the way for significant improvements in AI in the long term remains to be seen. For now, however, the AI industry continues to struggle with profitability, and since economic markets are rarely patient, an “AI winter” could set in if these improvements don’t materialize quickly enough.