Addition Optimization

Last Updated 11 August, 2026

by David Spuler, Ph.D.

What is Addition Optimization?

Addition optimization is a compute improvement to basic addition, which is typically done in hardware nowadays. Addition is not the main bottleneck when compared to multiplication, but there are various ways to improve addition, or to use addition in optimization of neural networks.

Addition has a role in optimization techniques such as:

Adder networks (and other types of multiplication-free networks)
Add-as-integer approximate multiplication
Logarithmic models (because logarithms convert multiplications to additions)
Binary quantization or ternary quantization (only requires additions and/or subtractions, or neither if bitwise operators used)
Approximate addition algorithms
Max-Plus networks (using addition and maximum operations)
Log-sum-exp (LSE) networks

Addition operations are a secondary bottleneck in a vanilla Transformer architecture, but they can be reduced for extra speedup. Some ways to only remove the addition operations from Transformers, without changing any multiplications, include:

Bias vector pruning (i.e., don't add any bias vectors in FFNs)
Residual connection pruning (i.e., remove skip connections)

Addition Optimization Techniques: Book Excerpts and Blog Articles

Free online book excerpts with full text chapters online and free PDF downloads, and the Aussie AI blog, including related articles:

Zero Multiplication Inference Algorithms: Multiplication causes a lot of trouble. It's slower than addition or bitshifting, and AI models need to calculate the times tables lots of times (literally billions). That adds up to a lot of CPU and GPU time spent doing the same thing. So why not do zero multiplication instead?... more about Zero Multiplication Inference Algorithms »
Shift-add models are zero-multiplication LLM optimizations that avoid multiplication operations in favor of bitshifting and addition. Power-of-two quantization is an example of such an architecture. However, none of these models have become mainstream, although they remain and interesting optimization research area.... more about Shift-add models »
Logarithmic quantization is a rarely-used type of quantization where each weight is a bitshift count, which is the logarithm of the original weight. This method is also known as "bitshift quantization" and "power-of-two quantization". It is not widely used because there are accuracy issues in the representational capacity of sparse power-of-two values, and modern CPU and GPU instruction sets have made the performance gain from changing multiplication to bitshift instructions much less significant than it was in the past... more about Logarithmic quantization »
Approximate addition is the theoretical analysis of algorithms to perform addition faster. However, this isn't really a modern concern, since both CPUs and GPUs perform addition as a single instruction at ... more about Approximate addition »
Add-as-integer models use an arithmetic optimization where floating-point multiplication is changed to integer addition. This is known as "add-as-integer" arithmetic and was ... more about add-as-integer »
Addernet models are an optimization whereby multiplications are replaced by integer addition. This is obviously faster, but is less accurate, since ... more about Addernet models »
Additive attention is an LLM optimization method whereby the attention module uses addition rather than multiplication. The advantage is speed, but ... more about additive attention »
David Spuler, March 2024, Chapter 53. Arithmetic Optimization Research, in book "Generative AI in C++", https://www.aussieai.com/book/ch53-arithmetic-optimization
David Spuler, March 2024, Generative AI in C++: Coding Transformers and LLMs, https://www.aussieai.com/book/toc PDF: https://www.aussieai.com/pdf/BOOK-Generative-AI-CPP-Spuler-2024.pdf

Addition Optimization

Research papers on optimizing the arithmetic addition operation:

O. Yildiz. 2017. Training methodology for a multiplication free implementable operator based neural networks. Master’s thesis, Middle East Technical University. URL https://hdl.handle.net/11511/26664. >>>>>>> .r1599
David Spuler, March 2024, Chapter 53. Arithmetic Optimization Research, Generative AI in C++: Coding Transformers and LLMs, https://www.amazon.com/dp/B0CXJKCWX9
Yunsheng Li, Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Lu Yuan, Zicheng Liu, Lei Zhang, Nuno Vasconcelos, MicroNet: Improving Image Recognition with Extremely Low FLOPs, 2021, https://ieeexplore.ieee.org/abstract/document/9857393 PDF: https://openaccess.thecvf.com/content/ICCV2021/papers/Li_MicroNet_Improving_Image_Recognition_With_Extremely_Low_FLOPs_ICCV_2021_paper.pdf
Sue Ann Campbell, Delay independent stability for additive neural networks, January 2001, Differential Equations and Dynamical Systems 9(3), https://www.researchgate.net/publication/268246122_Delay_independent_stability_for_additive_neural_networks PDF: https://www.math.uwaterloo.ca/~sacampbe/preprints/deds01.pdf
Xu Y, Xu C, Chen X, Zhang W, Xu C, Wang Y. Kernel based progressive distillation for adder neural networks. 2020 arXiv preprint arXiv:2009.13044 http://arxiv.org/abs/2009.13044
Vincenzo Liguori, 9 Jun 2024, Procrastination Is All You Need: Exponent Indexed Accumulators for Floating Point, Posits and Logarithmic Numbers, https://arxiv.org/abs/2406.05866
Xi Zhang, Huihui Liu, Junrui Xi, Menglu Chen, Tao Zhu, 30 Mar 2025, High-Dimensional Evolutionary Algorithm Based Design of Semi-Adder, https://arxiv.org/abs/2503.23322
Zhenhua Xu, Qichen Liu, Zhebo Wang, Wenpeng Xing, Dezhang Kong, Mohan Li, Meng Han, 25 Aug 2025, Fingerprint Vector: Enabling Scalable and Efficient Model Fingerprint Transfer via Vector Addition, https://arxiv.org/abs/2409.08846
Yao Yan, 9 Sep 2025, Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs, https://arxiv.org/abs/2506.07824
Rickard Br\"annvall and Andrei Stoian, 1 Oct 2025, The Inhibitor: ReLU and Addition-Based Attention for Efficient Transformers under Fully Homomorphic Encryption on the Torus, https://arxiv.org/abs/2310.02041
Qinyuan Ye, Robin Jia, Xiang Ren, 27 Sep 2025, Function Induction and Task Generalization: An Interpretability Study with Off-by-One Addition, https://arxiv.org/abs/2507.09875
Xinyan Hu, Kayo Yin, Michael I. Jordan, Jacob Steinhardt, Lijie Chen, 9 Oct 2025, Understanding In-context Learning of Addition via Activation Subspaces, https://arxiv.org/abs/2505.05145
David Spuler, March 2024, Chapter 53. Arithmetic Optimization Research, in book "Generative AI in C++", https://www.aussieai.com/book/ch53-arithmetic-optimization
Ghabeli H, Sabbagh Molahosseini A, Emrani Zarandi AA, Sousa L, 2021, Variable Latency Carry Speculative Adders with Input-based Dynamic Configuration. Comput Electr Eng 93:107247. https://doi.org/10.1016/j.compeleceng.2021.107247
David Spuler, March 2024, Generative AI in C++: Coding Transformers and LLMs, https://www.aussieai.com/book/toc PDF: https://www.aussieai.com/pdf/BOOK-Generative-AI-CPP-Spuler-2024.pdf

Approximate Addition Algorithms

Early papers examined the use of approximate addition. This is probably only of interest to hardware designers now.

V. Gupta, D. Mohapatra, S.P. Park, A. Raghunathan, “IMPACT: IMPrecise adders for low-power approximate computing”, International Symposium on Low Power Electronics and Design (ISLPED), pp. 409–414, 2011, https://dl.acm.org/doi/10.5555/2016802.2016898
V. Gupta, D. Mohapatra, A. Raghunathan, K. Roy, “Low-Power Digital Signal Processing Using Approximate Adders”, IEEE Transaction on CAD of Integrated Circuits and Systems 32(1): 124-137, 2013, https://dl.acm.org/doi/10.1109/TCAD.2012.2217962
M. Shafique, W. Ahmad, R. Hafiz, J. Henkel, “A Low Latency Generic Accuracy Configurable Adder”, IEEE/ACM Design Automation Conference (DAC), 2015, https://ieeexplore.ieee.org/abstract/document/7167270
R. Ye, T. Wang, F. Yuan, R. Kumar, Q. Xu, “On reconfiguration-oriented approximate adder design and its application”, International Conference on Computer-Aided Design (ICCAD), pp.48-54, 2013, PDF: https://www.cse.cuhk.edu.hk/~qxu/ye-iccad13.pdf
J. Miao, K. He, A. Gerstlauer, M. Orshansky, “Modeling and synthesis of quality-energy optimal approximate adders”, International Conference on Computer Aided Design (ICCAD), pp. 728-735, 2012, https://ieeexplore.ieee.org/document/6386754
A. B. Kahng, S. Kang, “Accuracy-configurable adder for approximate arithmetic designs”, IEEE/ACM Design Automation Conference (DAC), pp.820-825, 2012, https://ieeexplore.ieee.org/document/6241600
S. Mazahir, O. Hasan, R. Hafiz, M. Shafique, J. Henkel, “An Area-Efficient Consolidated Configurable Error Correction for Approximate Hardware Accelerators”, ACM/EDAC/IEEE 53rd Design Automation Conference (DAC), 2016, https://ieeexplore.ieee.org/document/7544339
N. Zhu, W.-L. Goh, K.-S. Yeo, “An enhanced low-power high-speed Adder for Error-Tolerant application”, 12th International Symposium on Integrated Circuits (ISIC), 2009, https://ieeexplore.ieee.org/document/5403865
Ryu, H. Kim, W. Yi and J.-J. Kim, "BitBlade: Area and energy-efficient precision-scalable neural network accelerator with bitwise summation", Proc. 56th Annu. Design Autom. Conf., pp. 1-6, Jun. 2019. https://ieeexplore.ieee.org/document/8807054
Ao Ren, Ji Li, Zhe Li, Caiwen Ding, Xuehai Qian, Qinru Qiu, Bo Yuan, Yanzhi Wang, 2017, "SC-DCNN: Highly-scalable deep convolutional neural network using stochastic computing", ACM SIGPLAN Notices, vol. 52, no. 4, pp. 405-418, 2017. https://arxiv.org/abs/1611.05939 (Stochastic method with multiplication and addition approximations via AND gates and multiplexers.)
Salar Shakibhamedan, Amin Aminifar, Nima TaheriNejad, Axel Jantsch, 2024, EASE: Energy Optimization through Adaptation — A Review of Runtime Energy-Aware Approximate Deep Learning Algorithms, https://eclectx.org/Publications/2024_M13.pdf (Survey paper on techniques for adaptive inference with a focus on approximations of inference, including loop performance, stochastic algorithms, approximate arithmetic, quantization, pruning and low-rank.)