Aussie AI

Other Addition Networks

  • Book Excerpt from "Generative AI in C++"
  • by David Spuler, Ph.D.

Other Addition Networks

These are research papers that use addition to attain zero-multiplication, but not the specific techniques above.

Research papers on other types of additive neural networks:

  1. Baluja S, Marwood D, Covell M, Johnston N., 2018, No multiplication? no floating-point? no problem! training networks for efficient inference, arXiv preprint arXiv:1809.09244, http://arxiv.org/abs/1809.09244 (This paper is mainly about low-bit integer quantization to avoid multiplication.)
  2. Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. 2019, Ghostnet: More features from cheap operations, arXiv preprint arXiv:1911.11907, 2019, https://arxiv.org/abs/1911.11907 (Applies linear operations to create extra “ghost features”, rather than a simple additive neural network.)
  3. O. Yildiz. 2017, Training methodology for a multiplication free implementable operator based neural networks, Master’s thesis, Middle East Technical University, 2017. URL https://hdl.handle.net/11511/26664
  4. O. Yildiz, 2017, Training Methodology for a Multiplication Free Implementable Operator Based Neural Networks, M.S. - Master of Science, Middle East Technical University, 2017. https://open.metu.edu.tr/handle/11511/26664 PDF: http://etd.lib.metu.edu.tr/upload/12621234/index.pdf
  5. Atli Kosson, Martin Jaggi, 2023, Hardware-Efficient Transformer Training via Piecewise Affine Operations, May 2023, https://arxiv.org/abs/2305.17190
  6. J. Johnson. 2018, Rethinking floating-point for deep learning, arXiv preprint arXiv:1811.01721, 2018, https://arxiv.org/abs/1811.01721 (“log float multiply-add” in hardware)
  7. Mark Horowitz, 2014, 1.1 computing’s energy problem (and what we can do about it), In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10–14. IEEE, 2014, https://doi.org/10.1109/ISSCC.2014.6757323
  8. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018, Mobilenetv2: Inverted residuals and linear bottlenecks, In CVPR, pages 4510–4520, 2018, https://arxiv.org/abs/1801.04381
  9. Z. Du, K. Palem, A. Lingamneni, O. Temam, Y. Chen, and C. Wu, 2014, Leveraging the error resilience of machine-learning applications for designing highly energy efficient accelerators, in 2014 19th Asia and South Pacific Design Automation Conference (ASP-DAC), 2014, pp. 201–206. https://ieeexplore.ieee.org/document/6742890
  10. Jingyong Cai, Masashi Takemoto,Yuming Qiu and Hironori Nakajo, 2021, Trigonometric Inference Providing Learning in Deep Neural Networks, Appl. Sci. 2021, 11(15), 6704; https://doi.org/10.3390/app11156704, https://www.mdpi.com/2076-3417/11/15/6704, PDF: https://www.mdpi.com/2076-3417/11/15/6704/pdf
  11. Afrasiyabi A, Badawi D, Nasir B, Yildi O, Vural FTY, Çetin AE. 2018, Non-euclidean vector product for neural networks, In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2018, pp. 6862–6866. https://ieeexplore.ieee.org/document/8461709, PDF: https://par.nsf.gov/servlets/purl/10067379

For more research papers on other addition-based neural networks, see https://www.aussieai.com/research/zero-multiplication#addmisc.

 

• Next:

• Up: Table of Contents

• Buy: Generative AI in C++: Coding Transformers and LLMs

Generative AI in C++ The new AI programming book by Aussie AI co-founders:
  • AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++