Obstacles to Smartphone AI

Obstacles to Smartphone AI

If it were possible, we'd already have many native AI apps. Although there are already plenty of “AI apps” available to install on your phone, these are almost certainly all sending the requests over the network to an AI engine in the cloud. Running an AI model directly on your phone is problematic for several reasons:

• Too slow to run — response times are too long.
• Hardware acceleration — phones lack a GPU and have less CPU acceleration.
• Storage size — e.g. a “small” 3B model with 32-bit weights will need 12 Gigabytes of storage. With modern phones often over 512GB, storing even a 13B model in 52GB seems reasonable.
• Memory usage — an entire model is loaded into RAM for inference. The obstacle is more the time cost of accessing this memory than the storage size.
• Transmission size — install a huge model over your phone's 4G or WiFi connection.
• Battery depletion — computations max out the phone's CPU and chew cycles.
• Heat generation — water-cooled phones are not a thing.

For these reasons, it's still faster to send AI requests off to a bigger server with lots of GPUs that's running in the cloud, even though it's a roundtrip network message. Before you see any truly “native” generative AI models in your app store, research is required to overcome all of the above obstacles.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++