Softmax Overflow and Underflow

Softmax Overflow and Underflow

Note that a simplified version of Softmax has been used in the code examples of this chapter for simplicity of explanation. Not only is this computation still very slow (even if we precompute all those calls to expf), it's also prone to overflow and underflow. The real computation of Softmax needs to be further optimized algebraically and scaled to avoid these problems.

This scaled computation can then be optimized using many of the same methods as for the naive Softmax version, as above. Further optimizations may include the use of calls to hardware acceleration APIs, pre-computed lookup tables as approximations for the expf function, and converting the loops to pointer arithmetic.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++