AI Tensor Computations

AI Tensor Computations

Everything's a tensor in AI. Tensors are technically the superset of all of the computational structures, and the number of dimensions is called the “rank” or “dimension” or “axes” of a tensor. Matrices are rank-2 tensors, vectors are rank-1 tensors, and even scalars are rank-0 tensors.

Another way to think about tensors is in terms of nested loops. Scanning a vector requires one loop, and a matrix needs two nested loops. Tensor operations require three or more nested loops to process all their data.

Conceptually, there's a hierarchy of operations on model data in an AI engine:

• 3-D tensor operations break down into 2-D matrix multiplications.
• 2-D matrix multiplications break down into vector dot products.
• 1-D vector dot products break down to a single float number (a scalar).
• 0-D scalars are probabilities between 0 and 1 (known as “logits”).

The final output logits are probabilities indicating whether to write “squirrel” or “aardvark” if the prior word was “ferocious”. I really don't know the answer to that, but your AI engine sure does, and to about three decimal places.

Note that I said “conceptually” not “in practice”, because the model data doesn't really get processed in a nice hierarchy like that. The real C++ code that does tensor operations sends off random strips of data to the GPU in a pattern like a turkey in a breakdancing competition. It's fun to watch, but who knows what's going on.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++