Aussie AI

Probabilistic Optimizations

  • Last Updated 14 October, 2024
  • by David Spuler, Ph.D.

I'm going to say it straight up: all neural networks are probabilistic algorithms. The entire concept of a neural network is a probabilistic approximation of reality based on what it's seen during training. In that sense, an AI engine is one big probabilistic algorithm, and optimizing it with another low-level probabilistic algorithm is really creating a doubly-probabilistic algorithm. After all, conceptually speaking, the result of an AI engine's output, called "logits", are probabilities of the likelihood of outputting a particular word. The core functionality of an AI model is to train it so it knows what words are "likely", and then it becomes good enough to "guestimate" what words should be output based on the probabilities that it calculates.

Similar to probabilistic algorithms are "stochastic algorithms", where an amount of randomness is injected to give results. For example, in the core theory of Computer Science, the "Monte Carlo method" is a well-known algorithm for measuring the area under a curve using randomly generated points. Some of these concepts can also be applied to AI models. These stochastic algorithms are technically a subset of probabilistic algorithms, since there are probabilistic methods that do not require random numbers, but still rely on probabilities for correct execution (e.g. neural networks). An example of a stochastic algorithm for neural networks is "stochastic quantization", where intentional randomness attempts to improve a quantized model.

The main use of probabilistic algorithms in AI inference is the randomness injected into decoding algorithms with the top-k, top-p, and temperature hyper-parameters. The randomness in these methods ensure that the AI engine doesn't output the exact same response when it receives the same input, so it can show creativity and originality in its responses to prompts. Note that the goal of randomness in decoding algorithms is variety of responses, and does not improve the speed of inference. However, the final stage of decoding is rarely a bottleneck, so the randomness doesn't degrade latency either.

Randomness of stochastic algorithms is not always beneficial. It can be useful to turn off the randomness in these decoding algorithms when regression testing an AI engine, which is a situation where having the exact same responses can be useful.

Probabilistic Optimizations

Probabilistic optimizations are methods of speeding up inference based on the relative probabilities of events. There are both high-level and lower-level ways that neural networks can use probabilistic algorithms. There are two main types of probabilistic algorithms:

  • Common case first. These are optimizations where a likely case is handled first, using fast code, but the unlikely case is also handled as a backup, with slower code. In such cases, the probabilistic algorithm does not affect the result of the neural network, only its speed.
  • Lossy or error-accepting algorithms. These are probabilistic algorithms where the unlikely case is simply ignored, and an error in such cases is accepted as part of the logic. Such algorithms may be appropriate for neural networks because they have an inherent resilience to errors.

Arguably, many of the inference optimization techniques and almost all model compression optimizations are lossy probabilistic optimizations, where a known level of error is tolerated. They take the probabilistic AI engine, add some known extra error sources to it (but making it faster), and the result is a faster, but slightly less accurate model. Examples of such "probabilistic optimizations" that offer trade-offs at a high-level are techniques such as:

  • Quantization (accepts errors resulting from the loss of precision).
  • Early exits (assumes that later layers won't change results, tolerating cases where this is erroneous).
  • Magnitude pruning (assumes that low-value weights have little impact, accepting the resulting errors).
  • Big-little models (common case first method; see ensemble algorithms)
  • Approximate algorithms (various types)

Some examples of the lower-level specific uses of probabilistic algorithms in neural networks include:

Probabilistic Algorithm Research Papers

General research papers on specific probabilistic algorithms (including stochastic algorithms) in neural networks:

More AI Research

Read more about: