Basic Min-Max Scaled Normalization

Basic Min-Max Scaled Normalization

Min-max normalization is the simplest incarnation of normalization, scaling numbers so that they are in the range [0..1], inclusive. This transformation changes the vector so that the smallest value (minimum) is 0, whereas the largest value (maximum) becomes 1. If all the numbers are positive, this effectively moves them back to zero, and then scales them with a multiplier so that the biggest is 1. If any of the numbers are negative, this scaling also works with a minimum value that is negative, effectively moving any negatives over to the positive side (i.e. with the most negative value starting at zero), and then scaling the magnitudes into the lower part of the range [0..1] inclusive, with any positives scaled closer to 1. In code, the simplest method is based on the minimum and maximum values:

    range = xmax - xmin;
    x[i] = ( x[i] - xmin ) / range;

Using this formula, the minimum will become 0, and the maximum will become 1. All of the other numbers will be inside the range [0..1]. As a practical matter, note that there is a divide-by-zero problem if the minimum and maximum are the same value.

Here is the C++ code for a naive unoptimized version:

    void aussie_vector_normalize_min_max_basic(float v[], int n)
    {
        float fmin = aussie_vector_min(v, n);   // Minimum
        float fmax = aussie_vector_max(v, n);   // Maximum
        float frange = fmax - fmin;
        if (frange == 0.0f) {
            yassert(frange != 0.0f);
            return;  // fail
        }
        for (int i = 0; i < n; i++) {
            v[i] = (v[i] - fmin) / frange;
        }
    }

Reciprocal Multiplication: The division can be optimized to a multiplication by its reciprocal, as follows:

    void aussie_vector_normalize_min_max_reciprocal(float v[], int n)
    {
        float fmin = aussie_vector_min(v, n);   // Minimum
        float fmax = aussie_vector_max(v, n);   // Maximum
        float frange = fmax - fmin;
        if (frange == 0.0f) {
            yassert(frange != 0.0f);
            return;  // fail
        }
        float frecip = 1.0f / frange;
        for (int i = 0; i < n; i++) {
            v[i] = (v[i] - fmin) * frecip;  // Reciprocal
        }
    }

Pointer Arithmetic. The loop can be optimized with pointer arithmetic to remove the index variable. The new version looks like:

    void aussie_vector_normalize_min_max_pointer_arith(float v[], int n)
    {
        float fmin = aussie_vector_min(v, n);   // Minimum
        float fmax = aussie_vector_max(v, n);   // Maximum
        float frange = fmax - fmin;
        if (frange == 0.0f) {
            yassert(frange != 0.0f);
            return;  // fail
        }
        float frecip = 1.0f / frange;
        float* vend = &v[n];
        for (; v != vend; v++) {
            *v = (*v - fmin) * frecip;  // Reciprocal multiplication
        }
    }

Loop Fusion: We have two loops in a row computing min and max, which can obviously be merged into one scan of the vector. This also removes the overhead of one function call. Here is the fused-loop version of computing minimum and maximum of a vector:

    float aussie_vector_min_max_fused(float v[], int n, float& vmax)  
    {
        // Mininum returned, maximum in parameter
        vmax = v[0];
        float vmin = v[0];
        for (int i = 1 /*not 0*/; i < n; i++) {
            if (v[i] < vmin) vmin = v[i];
            if (v[i] > vmax) vmax = v[i];
        }
        return vmin;
    }

    void aussie_vector_normalize_min_max_fusion(float v[], int n)
    {
        float fmax = 0.0f;   // Maximum
        float fmin = aussie_vector_min_max_fused(v, n, fmax);
        float frange = fmax - fmin;
        if (frange == 0.0f) {
            yassert(frange != 0.0f);
            return;  // fail
        }
        float frecip = 1.0f / frange;
        float* vend = &v[n];
        for (; v != vend; v++) {
            *v = (*v - fmin) * frecip;  // Reciprocal
        }
    }

Benchmarking Results of MinMax Normalization. The MinMax normalization method is simple and relatively fast, when compared to Z-score normalization or BatchNorm (see further below), which do more arithmetic operations. Running the various versions of MinMax gave the following benchmark results:

    MinMax benchmarks (N=2048, ITER=100000)
    MinMax basic: 999 ticks (1.00 seconds)
    MinMax reciprocal: 842 ticks (0.84 seconds)
    MinMax ptr arith: 710 ticks (0.71 seconds)
    MinMax loop fusion: 653 ticks (0.65 seconds)

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++