Advanced Number Systmes Introduction

Advanced Number Systmes Introduction

The methods that computers use to represent numbers are mostly arbitrary. They are based on standardization that occurred years ago, and re-exploring these bit layouts has already yielded faster AI methods (e.g. bfloat16 from Google Research). There is much theoretical research on advanced mathematics and this has recently been applied to optimizing inference algorithms and training. The main areas of research focus include:

• Numeric bit representations
• Advanced number systems

Numeric bit representations refer to the bit layouts for integers or floating-point numbers. Integer representations are not changing much, but there's still a choice between one's complement and two's complement to explore. Floating-pointer bit representations, such as modifications to the IEEE 754 floating-point standard, remain an ongoing area of advancements.

Various alternative number systems have been used in computer software and hardware. They have different trade-offs in terms of precision and arithmetic complexity. Several of these have been examined in terms of model inference algorithms, mainly to reduce the number of multiplications. The main ones include:

• Dyadic numbers
• Residue Number System (RNS)
• Posit Number System (PNS)
• Logarithmic Number System (LNS)

Dyadic numbers, RNS, and PNS have all been applied to AI engines (see below). LNS is an additive method with an extensive amount of theory (see Chapter 52)

If you enjoy this kind of stuff, or if you need a way to get your infant to go to sleep without driving around the block a hundred times, there's also some obscure ones:

• Tropical algebra (related to Max-Plus networks).
• Minimax algebra (related to tropical algebra)
• Double-Base Number System (DBNS)
• Multiple-Base Number System (MBNS)
• Multi-Dimensional Logarithmic Number System (MDLNS)
• Semi-Logarithmic Number System (SLNS)

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++