C++ Coding Strategy for AI

C++ Coding Strategy for AI

What's it like coding AI in C++? A short answer is “your brain will explode” because everything's different and at a scale you won't believe. But a longer answer is that although some of the optimizations get quite mind-bending, there are also many areas where it's just normal C++ coding applied to a new area of focus.

Some of the overarching characteristics of coding AI engines in C++ include:

• No if statements! Seriously, what is going on here? There is no logic like: “if this token, then do that vector, else do this vector." All of the conditional logic is wrapped up in the weights, and rather than doing any testing, we just multiply all the weights by all the inputs. If you want conditional tests in the high-level control flow, that's only in the various “adaptive” optimization algorithms and research enhancements, because there's none in the vanilla Transformer model.
• Brute-force algorithm. Every single weight is used in every invocation of the model. The whole logic of AI inference propagates a vector of computations through a fixed sequence of N layers and a fixed number of components in each layer, without much branching logic at all. It is finite, without any unbounded looping, and the whole algorithm can be flattened and unrolled. Indeed, that's exactly what ML compilers do by creating a graph representation.
• Backend coding. The AI engine has no user interface other than accepting a text prompt as input, and outputting its response one word at a time. Image engines are a little different, but most of the execution is still batch backend work.
• Non-interactive. The way that an AI engine cranks through all the weights and layers is not interactive. From the outside, there isn't anything much to do but wait for the first token to come out from the decoder. On the inside, all of the coding is batch.
• Single platform. In a lot of AI projects, you can control the data center servers and GPU hardware, allowing you to optimize the C++ code for a single platform. This is also true if you're researching AI engines by running them locally on your own desktop.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++