Aussie AI

Log-Sum-Exp Networks

  • Book Excerpt from "Generative AI in C++"
  • by David Spuler, Ph.D.

Log-Sum-Exp Networks

Log-sum-exp (LSE) networks involve the formula that is a triple sequence on a vector of numbers: take the logarithm of a sum of exponentials. This is a theoretically interesting area, but not a mainstream neural network architecture.

Haven't we seen the log-sum-exp pattern elsewhere? Yes, several other areas of research are related to log-sum-exp theory. Because logarithmic number system (LNS) addition involves computing exponentials of log-domain values (i.e. antilogarithms), adding them, and then re-converting them to log-domain, this is also emulating a “log of a sum of exponentials” calculation. Hence, log-sum-exp theory relates to approximating LNS addition (for a zero-multiplication logarithmic model). Also, the “sum of exponentials” is the same as the calculation required for the denominator of Softmax calculations, so log-sum-exp theory is also indirectly related to Softmax approximation. Finally, since the use of the maximum function is one way to approximate log-sum-exp (and also LNS addition), the theory of “max-plus networks” based on “tropical algebra” is indirectly related to log-sum-exp networks.

Research papers on Log-Sum-Exp networks:

  1. GC Calafiore, C Possieri, 2020, Efficient model-free Q-factor approximation in value space via log-sum-exp neural networks, 2020 European Control Conference, https://ieeexplore.ieee.org/abstract/document/9143765/, PDF: https://core.ac.uk/download/pdf/327178231.pdf
  2. G. Calafiore, S. Gaubert, and C. Possieri, 2019, Log-sum-exp neural networks and posynomial models for convex and log-log-convex data, IEEE Transactions on Neural Networks and Learning Systems, 2019. https://ieeexplore.ieee.org/abstract/document/8715799/, https://arxiv.org/pdf/1806.07850
  3. GC Calafiore, S Gaubert, 2020, A universal approximation result for difference of log-sum-exp neural networks, IEEE Transactions on Neural Networks and Learning Systems (Volume 31, Issue 12, December 2020), https://ieeexplore.ieee.org/abstract/document/9032340/, PDF: https://arxiv.org/pdf/1905.08503

For more research papers on the Log-Sum-Exp algebra, see also https://www.aussieai.com/research/advanced-ai-mathematics#logsumexp.

 

• Next:

• Up: Table of Contents

• Buy: Generative AI in C++: Coding Transformers and LLMs

Generative AI in C++ The new AI programming book by Aussie AI co-founders:
  • AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++