Addition Optimization

Addition is not the main bottleneck when compared to multiplication, but there are various ways to improve addition, or to use addition in optimization of neural networks.

Addition has a role in optimization techniques such as:

• Adder networks (and other types of multiplication-free networks)
• Add-as-integer approximate multiplication
• Logarithmic models (because logarithms convert multiplications to additions)
• Binary quantization or ternary quantization (only requires additions and/or subtractions, or neither if bitwise operators used)
• Approximate addition algorithms
• Max-Plus networks (using addition and maximum operations)
• Log-sum-exp (LSE) networks

Addition operations are a secondary bottleneck in a vanilla Transformer architecture, but they can be reduced for extra speedup. Some ways to only remove the addition operations from Transformers, without changing any multiplications, include:

• Bias vector pruning (i.e., don't add any bias vectors in FFNs)
• Residual connection pruning (i.e., remove skip connections)

Addition Optimization

Research papers on optimizing the arithmetic addition operation:

• O. Yildiz. 2017. Training methodology for a multiplication free implementable operator based neural networks. Master’s thesis, Middle East Technical University. URL https://hdl.handle.net/11511/26664. >>>>>>> .r1599
• David Spuler, March 2024, Chapter 53. Arithmetic Optimization Research, Generative AI in C++: Coding Transformers and LLMs, https://www.amazon.com/dp/B0CXJKCWX9
• Yunsheng Li, Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Lu Yuan, Zicheng Liu, Lei Zhang, Nuno Vasconcelos, MicroNet: Improving Image Recognition with Extremely Low FLOPs, 2021, https://ieeexplore.ieee.org/abstract/document/9857393 PDF: https://openaccess.thecvf.com/content/ICCV2021/papers/Li_MicroNet_Improving_Image_Recognition_With_Extremely_Low_FLOPs_ICCV_2021_paper.pdf
• Sue Ann Campbell, Delay independent stability for additive neural networks, January 2001, Differential Equations and Dynamical Systems 9(3), https://www.researchgate.net/publication/268246122_Delay_independent_stability_for_additive_neural_networks PDF: https://www.math.uwaterloo.ca/~sacampbe/preprints/deds01.pdf
• Xu Y, Xu C, Chen X, Zhang W, Xu C, Wang Y. Kernel based progressive distillation for adder neural networks. 2020 arXiv preprint arXiv:2009.13044 http://arxiv.org/abs/2009.13044
• Vincenzo Liguori, 9 Jun 2024, Procrastination Is All You Need: Exponent Indexed Accumulators for Floating Point, Posits and Logarithmic Numbers, https://arxiv.org/abs/2406.05866

Approximate Addition Algorithms

Early papers examined the use of approximate addition. This is probably only of interest to hardware designers now.

• V. Gupta, D. Mohapatra, S.P. Park, A. Raghunathan, “IMPACT: IMPrecise adders for low-power approximate computing”, International Symposium on Low Power Electronics and Design (ISLPED), pp. 409–414, 2011, https://dl.acm.org/doi/10.5555/2016802.2016898
• V. Gupta, D. Mohapatra, A. Raghunathan, K. Roy, “Low-Power Digital Signal Processing Using Approximate Adders”, IEEE Transaction on CAD of Integrated Circuits and Systems 32(1): 124-137, 2013, https://dl.acm.org/doi/10.1109/TCAD.2012.2217962
• M. Shafique, W. Ahmad, R. Hafiz, J. Henkel, “A Low Latency Generic Accuracy Configurable Adder”, IEEE/ACM Design Automation Conference (DAC), 2015, https://ieeexplore.ieee.org/abstract/document/7167270
• R. Ye, T. Wang, F. Yuan, R. Kumar, Q. Xu, “On reconfiguration-oriented approximate adder design and its application”, International Conference on Computer-Aided Design (ICCAD), pp.48-54, 2013, PDF: https://www.cse.cuhk.edu.hk/~qxu/ye-iccad13.pdf
• J. Miao, K. He, A. Gerstlauer, M. Orshansky, “Modeling and synthesis of quality-energy optimal approximate adders”, International Conference on Computer Aided Design (ICCAD), pp. 728-735, 2012, https://ieeexplore.ieee.org/document/6386754
• A. B. Kahng, S. Kang, “Accuracy-configurable adder for approximate arithmetic designs”, IEEE/ACM Design Automation Conference (DAC), pp.820-825, 2012, https://ieeexplore.ieee.org/document/6241600
• S. Mazahir, O. Hasan, R. Hafiz, M. Shafique, J. Henkel, “An Area-Efficient Consolidated Configurable Error Correction for Approximate Hardware Accelerators”, ACM/EDAC/IEEE 53rd Design Automation Conference (DAC), 2016, https://ieeexplore.ieee.org/document/7544339
• N. Zhu, W.-L. Goh, K.-S. Yeo, “An enhanced low-power high-speed Adder for Error-Tolerant application”, 12th International Symposium on Integrated Circuits (ISIC), 2009, https://ieeexplore.ieee.org/document/5403865
• Ryu, H. Kim, W. Yi and J.-J. Kim, "BitBlade: Area and energy-efficient precision-scalable neural network accelerator with bitwise summation", Proc. 56th Annu. Design Autom. Conf., pp. 1-6, Jun. 2019. https://ieeexplore.ieee.org/document/8807054
• Ao Ren, Ji Li, Zhe Li, Caiwen Ding, Xuehai Qian, Qinru Qiu, Bo Yuan, Yanzhi Wang, 2017, "SC-DCNN: Highly-scalable deep convolutional neural network using stochastic computing", ACM SIGPLAN Notices, vol. 52, no. 4, pp. 405-418, 2017. https://arxiv.org/abs/1611.05939 (Stochastic method with multiplication and addition approximations via AND gates and multiplexers.)
• Salar Shakibhamedan, Amin Aminifar, Nima TaheriNejad, Axel Jantsch, 2024, EASE: Energy Optimization through Adaptation — A Review of Runtime Energy-Aware Approximate Deep Learning Algorithms, https://eclectx.org/Publications/2024_M13.pdf (Survey paper on techniques for adaptive inference with a focus on approximations of inference, including loop performance, stochastic algorithms, approximate arithmetic, quantization, pruning and low-rank.)

More AI Research

Read more about:

• Multiplication Optimization
• Division Optimization
• Zero-Multiplication Models
• Advanced AI Mathematics
• Matrix Algebra
• Logarithmic Models
• Approximate Computing
• Inference Optimizations
• « Research Home