Native Smartphone AI

Native Smartphone AI

Can an AI model run fast enough on your phone? I'm not talking about having your phone talk to some anonymous server in the cloud to do its AI. I'm wondering whether it's possible to run the actual C++ engine natively on the phone's CPU.

This is an area of research that is of personal interest to me. As goals go, it's quite an ambitious one: run a big AI model that's usually thirsty for GPUs, on a small platform without a GPU.

Much of the early research that is relevant to fast phone execution of models relates to another type of computer, which you might know as a “car.” The need for computer vision models for automated or assisted driving has similar requirements to running on a phone, such as low latency and small storage. The general term is an “embedded” system or “real-time” system.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++