Aussie AI

Alternatives to Attention

  • Book Excerpt from "Generative AI in C++"
  • by David Spuler, Ph.D.

Alternatives to Attention

Although attention has performed very well, there are still various attempts to replace it with something even better. Usually, the goal is a simpler type of attention algorithm with fewer computations involved. There are newer ideas, and also older ideas, where some of the research papers also try reverting back to earlier methods that existed before the current generation of attention algorithms. There is a lot of overlap with the research area of “non-autoregressive decoding algorithms” and “parallel decoding” (see Chapter 26) where the aim is also to avoid the quadratic cost of attention on long context sequences.

 

• Next:

• Up: Table of Contents

• Buy: Generative AI in C++: Coding Transformers and LLMs

Generative AI in C++ The new AI programming book by Aussie AI co-founders:
  • AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++