Root Mean Square Normalization

Root Mean Square Normalization

The Root Mean Square (RMS) is a well-known measure of accuracy in statistical regressions. It is computed as the square root of the average of the squares of every element.

    float sum_squares = aussie_vector_sum_squared(v, n); // Sum of squares
    float avg_squares = sum_squares / n;  // Average of the squares
    float rms = sqrtf(avg_squares);  // RMS factor

When the RMS computation is applied to a vector, there is a relationship with the vector magnitude. The RMS value is the vector's magnitude divided by the square root of N. Hence, the RMS is significantly smaller than vector magnitude.

Here is a basic unoptimized C++ version of RMSNorm. Note that it uses an epsilon factor to avoid division-by-zero and subnormal fraction problems.

    void aussie_vector_rms_normalize_basic(float v[], int n)  // Basic RMSNorm
    {
        const float epsilon = 0.00005; // Smoothing term -- 1^e-5 (0.00005)
        float sum_squares = aussie_vector_sum_squared(v, n);  // Sum of squares
        float avg_squares = sum_squares / n;         // Average of the squares...
        float denom = sqrtf(avg_squares + epsilon);  // RMS factor
        aussie_vector_divide_scalar(v, n, denom);  // Divide by the RMS scale factor
    }

RMSNorm uses the RMS factor to scale the elements of the logits vector. Every element is divided by the RMS factor, which should make all the elements smaller if most are large. However, if enough of the vector elements are small fractions, then the RMS can be a fraction less than 1, and its division will increase the size of all the vector elements.

The RMSNorm does not ensure all elements are scaled into the range [0..1], nor does it re-center the elements to any particular mean, and it also does not ensure they add up to 1. This can make it faster to calculate than some other normalizations. On the other hand, all those multiplications for the squares can slow it down.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++