Aussie AI

AI Hitting the Wall?

  • Last Updated 28 February, 2025
  • by David Spuler, Ph.D.

Update Jan 2025: DeepSeek Ends the Wall Debate

The wall debate is over, but it's not OpenAI that provided the sledgehammer. The recent release of DeepSeek R1 by a Chinese startup has largely ended the wall debate. Their model beat OpenAI's o1 model on multiple benchmarks (but not all), and did so using only single-step inference (whereas o1 is multi-step inference). Hence, they used training advances to train a much smarter model than other single-step models.

Bye-bye, wall!!

The Nov 2024 Wall Debate

Recently, there's been a two-fold indication that AI progress is "plateauing" or "hitting a wall." The two main indicators are:

  • Inference-based reasoning ("test time compute")
  • Underwhelming progress in new models

The GPT "o1" model released in September 2024 wasn't a bigger, more heavily-trained model with trillions more weights. Instead, it's a model that improves intelligence by doing multiple steps of inference, rather than one smarter step in an uber-trained model. This algorithm for "multi-step reasoning" is known as "chain-of-thought" and uses repeated calls to process queries, before merging them together into the one final response.

Why does this change to multi-step inference for reasoning support the "wall" theory? Well, inference is a slow process when it runs, and "o1" is therefore slow for users — the line of logic goes that OpenAI wouldn't tolerate using this slow method if they could do it with one request to a bigger model. It almost seems like a kind of workaround.

Hence, wall.

Secondly, there are also rumors that the big players are having difficulty training much better next-gen models. In particular, there are indicators that the GPT-5 release is having trouble gaining capabilities compared to GPT-4. Instead of launching GPT-5 soon, we got "o1" with its multiple steps.

Obviously, training trillion-parameter models is a specialist field, and it's evolving fast, with literally billions of dollars in funding being applied there. But open source models seem to be keeping up with the leading commercial vendors (albeit, after a lag), which tends to indicate that there's only incremental progress in reasoning capabilities, and the commercial vendors don't have a huge "secret sauce" algorithmic advantage in training. Some of the constraints include:

  • Shortage of new high-quality training data (text).
  • Complexity of software algorithms to train ever-bigger LLMs.
  • Sheer volume of training data needed for multimodal LLMs (audio, images, and video).
  • Capital cost of GPUs to crunch all that.
  • Apparent lack of a new algorithmic advance in one-shot reasoning.
  • Fundamental limitations of the way that LLMs and Transformers work.

On the other hand, there's a lot of research happening in training and in making LLMs better at reasoning in general. Some of the newer areas include:

  • Newer GPU hardware for training (e.g., Blackwell).
  • Faster software training algorithms (optimizing both computations and inter-GPU network traffic).
  • Resiliency improvements to training (both software and hardware).
  • Synthetic training data and derivative data.
  • Multi-step reasoning algorithms are smarter (if slower).
  • Long context processing seems to be a solved problem now.
  • Inference optimization research (makes each step of multi-step reasoning faster).
  • Next-gen architectures beyond LLMs (e.g., SSMs, Mamba, Hyena, and hybrid versions).

Is there a wall? OpenAI CEO Sam Altman posted on X that "there is no wall." And there are certainly signs that many of the bigger players are still gearing up to use NVIDIA Blackwell GPUs for even bigger training runs. And there have been two multi-billion dollar fund raises in just the last month. So, the plateau may only be a temporary thing.

Research on the AI Progress Wall

Articles and papers on recent AI progress:

More AI Research

Read more about: