What is Normalizationand?

What is Normalization?

Normalization is the process of controlling the generated numbers in AI inference. We want the numbers to be like probabilities, so that the weights modify those probabilities. Although probabilities should be in the range 0..1 (inclusive), we can allow computations to create bigger or smaller numbers within computations (e.g. inside a component), provided that we scale them back whenever we need them to be real probabilities. Hence, the numbers are “normalized” in Transformer computations at various points between components and between layers.

There are several types of normalization:

• Basic min-max scaled normalization
• Z-score normalization
• Root Mean Square normalization (RMSNorm)
• Batch normalization (BatchNorm)
• Layer normalization (LayerNorm)

Note that Softmax is a type of normalization of a vector of numbers that's treated specially in Transformers. Softmax is examined in detail in Chapter 25.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++