Aussie AI

AI Tech Stack

  • Book Excerpt from "Generative AI in C++"
  • by David Spuler, Ph.D.

AI Tech Stack

The tech stack for an AI project is similar to a non-AI project, with a few extra components. There's also a much greater importance tied to the choice of underlying hardware (i.e. GPUs) than in many other types of projects. The tech stack looks something like this:

  • User interface (client)
  • Web server (e.g. Apache or Nginx)
  • Application server
  • Load balancer (e.g. Apache Kafka)
  • AI request manager
  • AI Inference Engine (and model)
  • Operating system (e.g. Linux vs Windows)
  • CPU hardware (e.g. Intel vs AMD)
  • GPU hardware (e.g. NVIDIA V100 vs A100)

Some of these layers are optional or could be merged into a single component. Also, if you're using a remote hosted AI engine, whether open source hosting or wrapping a commercial engine through their API, then the bottom layers are not always your responsibility.

AI engine choices. How much of your AI tech stack will you control? If you have full control over the hardware and software, it makes sense to make symbiotic choices that allow maximum optimization of the combined system. For example, if you've decided to run the system on a particular GPU version, then your AI engine can assume this hardware acceleration is available, and don't need to waste resources on ensuring your engine's C++ software runs on any other hardware platforms.

 

• Next:

• Up: Table of Contents

• Buy: Generative AI in C++: Coding Transformers and LLMs

Generative AI in C++ The new AI programming book by Aussie AI co-founders:
  • AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++