Aussie AI Blog

State-of-the-Art LLM Backends

  • 26th August, 2024
  • by David Spuler, Ph.D.

State-of-the-Art LLM Backends

It's somewhat difficult to determine the state-of-the-art in LLM serving backends, as used by the industry's top players. Much of this information is commercially sensitive and is no longer appearing in public research papers (maybe it's in patents!). Nevertheless, there are some public papers and articles about various issues. Let's look at a few of them.

Character.AI companionbots backend. As detailed in their blog post, Character.AI has a very high level of traffic to their models. Inference optimization techniques include:

  • INT8 quantization of weights and activations
  • KV cache quantization (also INT8)
  • MatMul INT8 kernels
  • INT8 training (QAT)
  • Hybrid attention with interleaved layers of local attention and global attention (with global attention for only approximately 1 in every 6 layers)
  • KV cache compression
  • Multi-Query Attention (MQA)
  • KV cache layer fusion
  • Session KV caching (for chat sessions)
  • Prefix KV caching
  • Sticky sessions (avoids copying session caches)

They cite a 13.5X reduction in cost versus use of commercial model hosting (i.e., by doing it themselves), and also a 33X reduction compared to when they started optimizing.

Apple Intelligence for On-Device Inference. In announcing their "Apple Intelligence" initiative in June, 2024, Apple released certain information about the platform, specifically in relation to on-device execution of LLMs on iPhones and Mac. The exact details are somewhat opaque, but some aspects include:

  • M-series CPUs with NPU capabilities
  • 3B base LLM (with 16-bit precision)
  • LoRA adapters for fine-tuning (with 16-bit parameters, sized "in the tens of millions")
  • Multi-LoRA inference
  • Grouped Query Attention (GQA)
  • Low-bit quantization for some parameters (mixed 2-bit and 4-bit quantizations)
  • Talaria (tool for analysis)
  • KV cache quantization (bit precision undisclosed)
  • KV cache optimizations for "KV cache update" (details undisclosed)

With these optimizations, Apple reported performance on an iPhone 15 Pro of time-to-first-token latency of 0.6 milliseconds per prompt token, and decoding phase performance of 30 tokens per second.

Together AI data center networking. In various papers and announcements, Together AI has offered various details of its backend platform. Some example software optimizations available include:

  • CUDA backend for NVIDIA GPUs
  • Flash Attention
  • Flash Decoding
  • Medusa decoding (multi-token prediction)

Together AI also described their GPU cluster management, including the networking aspects and validation steps. This involves techniques and components such as:

  • H100 GPUs
  • Infiniband networking
  • NVLink and NVSwitch
  • NCCL
  • HPCX
  • SLURM (workload management)
  • Remote Direct Memory Access (RDMA) (GPUDirect RDMA)
  • Telegraf (open source monitoring)

Important steps in the process include:

  • GPU validation
  • Network validation
  • Storage validation

References

Free AI and C++ Books

Generative AI programming books:

  1. The Sweetest Lesson: Your Brain Versus AI, November 2025: full text online, free PDF available
  2. RAG Optimization: Accurate and Efficient LLM Applications, June 2025: full text online, free PDF available
  3. Generative AI Applications: Planning, Design and Implementation, November 2024: full text online, free PDF available
  4. Generative AI in C++ (Spuler, March 2024): full text online, free PDF available, table of contents, bonus materials, reference lists, source code

CUDA C++ GPU Programming Books:

  1. CUDA C++ Optimization: Coding Faster GPU Kernels, July 2024: full text online, bonus materials, free PDF available
  2. CUDA C++ Debugging: Safer GPU Kernel Programming, July 2024: full text online, free PDF available

Modern C++ Programming Books

  1. C++ AVX Optimization: CPU SIMD Vectorization, 2025: full text online, free PDF available
  2. C++ Ultra-Low Latency: Multithreading and Low-Level Optimizations, 2025: full text online, free PDF available
  3. Advanced C++ Memory Techniques: Efficiency and Safety, 2025: full text online, free PDF available
  4. Efficient C++ Multithreading: Modern Concurrency Optimization, 2025: free PDF available
  5. Efficient Modern C++ Data Structures: Container and Algorithm Optimizations, 2025: free PDF available
  6. C++ Low Latency: Multithreading and Hotpath Optimizations, 2025: free PDF available
  7. Safe C++: Fixing Memory Safety Issues, Oct 2024: full text online, free PDF available

More AI Research Topics

Read more about: