Aussie AI

Near-Term Technology Trends

  • Book Excerpt from "Generative AI in C++"
  • by David Spuler, Ph.D.

Near-Term Technology Trends

Over time some of the obstacles to natively-executing inference on phones will diminish:

  • Better phone CPUs with hardware acceleration are already here (e.g. Apple Neural Engine since iPhone X, Qualcomm Snapdragon), with more on the way. Future phones will be much more AI-capable.
  • GPU phones will surely be coming to a store near you very soon.
  • Phone storage sizes are also increasing with terabyte storage sizes common.
  • 5G network connectivity will reduce concerns about transmission sizes.
  • Data compression algorithms can lower transmission sizes, and also possibly storage sizes.
  • Quantized models and other inference optimizations can improve speed and reduce storage size, giving reduced CPU usage, faster response times, lower storage size, and reduced transmission size (but with accuracy loss).
  • Training and fine-tuning of models doesn't need to happen on a phone (phew!).

But... you really need a “big” model, not a “small” model, if you want the app to be great with lots of happy users. And getting a big model running efficiently on a phone may take a while to come to fruition.

 

• Next:

• Up: Table of Contents

• Buy: Generative AI in C++: Coding Transformers and LLMs

Generative AI in C++ The new AI programming book by Aussie AI co-founders:
  • AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++