Aussie AI

Scaling Laws in Generative AI

  • Last Updated 21 March, 2025
  • by David Spuler, Ph.D.

What are AI Scaling Laws?

Scaling laws are the contention that AI models will become smarter by scaling the model size in terms of paramter count, and/or the total number of input tokens used in model training. Recent reductions in effects from greater training have thrown some of these scaling laws in doubt, giving rise to a new scaling law called the "inference scaling law," which says that scaling the amount of inference computations can also increase model intelligence.

What are Inference Scaling Laws?

Inference scaling laws are the contention that smarter LLMs can be created by using additional inference computations, such as repeated LLM queries at runtime, rather than by more extensive training. The success of the OpenAI "o1" model has supported this trend, as it is based on a multi-step inference algorithm called "Chain-of-Thought."

Research on Inference Scaling Laws

Research papers on the scaling laws in regard to multi-step inference:

What is Test Time Compute?

Test time compute is using additional computation at the LLM inference stage, rather than in pre-training or fine-tuning. The model weights stay constant during inference, but certain algorithms can improve reasoning through advanced prompting strategies and multi-step inference algorithms.

Research papers on test time compute:

Research on Scaling Laws

Research on the traditional scaling laws of model size and training data:

More AI Research

Read more about: