Aussie AI

Dyadic Numbers

  • Book Excerpt from "Generative AI in C++"
  • by David Spuler, Ph.D.

Dyadic Numbers

One type of numeric representation that has received some specific attention in relation to neural networks is dyadic numbers (or “dyadic rationals”). These are rational numbers represented as a division with an integer numerator and a power-of-two denominator. Hence, the number system actually represents them as a pair of two numbers. Although doubling the number of weights doesn't sound like much of an optimization at first glance, the top number allows integer arithmetic to be used, and the bottom number is an integer power-of-two, which allows bitshifting. It may also be possible to use a scaled dyadic engine that pushes the numerator into higher-order integers, and defers the bitshift of the denominator to a less-frequent operation.

Research papers on dyadic numbers:

  1. Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, Kurt Keutzer, 2021, HAWQ-V3: Dyadic Neural Network Quantization, Proceedings of the 38th International Conference on Machine Learning, PMLR 139:11875-11886, 2021, https://arxiv.org/abs/2011.10680
  2. Nilsson, J. 2009, On numbers badly approximable by dyadic rationals, Isr. J. Math. 171, 93–110 (2009), https://doi.org/10.1007/s11856-009-0042-9
  3. C. Avalos-Ramos, J. A. Felix-Algandar, J. A. Nieto, 2020, Dyadic Rationals and Surreal Number Theory, IOSR Journal of Mathematics (IOSR-JM), e-ISSN: 2278-5728, p-ISSN: 2319-765X. Volume 16, Issue 5 Ser. IV (Sep– Oct 2020), p.35-43, www.iosrjournals.org, PDF: https://www.academia.edu/download/64798757/E1605043543.pdf

For more research papers on dyadic numbers, see also https://www.aussieai.com/research/advanced-ai-mathematics#dyadic. Note: see also dyadic quantization in Chapter 44.

 

• Next:

• Up: Table of Contents

• Buy: Generative AI in C++: Coding Transformers and LLMs

Generative AI in C++ The new AI programming book by Aussie AI co-founders:
  • AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++