Introduction to Logarithmic Models

Introduction to Logarithmic Models

Logarithms use high-school mathematics to change multiplications into additions. And this is an interesting idea, given all the shade that's been thrown on the expensive multiplication operators in neural networks, even with all this hardware acceleration going on.

A few people have thought of this already, and the amount of literature is beyond extensive. There was much theory in the 1970s and 1980s, and some real hardware implementations in the 1990s and 2000s. There's also been a lot of more recent research on this area in the 2010s and 2020s.

Logarithmic Number System The way that logarithmic models work in the log-domain is to use the Logarithmic Number System (LNS). The LNS is a numeric representation that uses logarithms, but isn't the standard mathematical logarithmic arithmetic. It has been considered for use with neural networks as far back as 1997. In the LNS, multiplication changes to a fast additive operation, but addition itself becomes expensive. Thus, computation of vector dot products or multiply-add operations in machine learning are problematic, with various theoretical attempts to overcome the difficulties (with addition operators) researched in the literature.

LNS is not the only unusual theoretical number system available. In addition to the simple floating-point and fixed-point representations, LNS should be compared to other complicated number systems considered for machine learning, including the Residue Number System (RNS), Posit number system, and Dyadic numbers (see Chapter 55 on advanced mathematics).

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++