Aussie AI

Inference Optimization Techniques

Last Updated 24th August, 2024

by David Spuler, Ph.D.

This is a list of neural network and Transformer optimizations, with a specific focus on speeding up Transformer inference. Resources include:

Hot New Research Areas

Areas of inference efficiency research that have been recently getting attention:

On-device inference (native phone and PC AI)
Generalized speculative decoding
Consensus decoding
Multi-LoRA inference
KV Cache Compression/Quantization
KV Cache Layer Fusion
Prefix KV Caching
Prefill optimizations (decoder-only engines)
KV cache recomputation with early exit
Deep prefill, shallow decoder architecture
Fixed-point quantization (integer)
Fixed-point arithmetic
Block floating-point arithmetic
FFN sublayer pruning

Hot Old Research Areas

Longstanding research areas that are still seeing a continual stream of papers:

Model Compression

Model compression (overview) (static model optimizations)
Pruning
Quantization
Knowledge Distillation (KD)
Parameter sharing
Low-rank matrices
Neural Architecture Search (NAS)

Pruning

Pruning overview
Unstructured pruning
- Magnitude pruning
- Movement pruning
Depthwise structural pruning (vertical):
- Depth pruning (overview)
- Layer pruning
- Early exit (dynamic layer pruning)
- Layer skipping
- Layer approximation
Widthwise structural pruning (horizontal):
- Width pruning (overview)
- Channel pruning
- Filter pruning
- Attention head pruning
- Slimmable networks (width pruning)
Lengthwise structural pruning (longitudinal/end-to-end):
Pruning by dimension:
Transformer-specific pruning (component removal):
- Shallow decoder architecture (layer pruning)
- Attention head pruning (width pruning)
- FFN pruning
- Normalization pruning
- Positional embeddings pruning
- Softmax pruning
Dynamic pruning
Hybrid pruning

Quantization

Quantization overview
Post-Training Quantization (PTQ)
Quantization-Aware Training (QAT)
Low-bit integer quantization:
Integer quantization
- 4-bit quantization (INT4)
- 5-bit quantization (INT5)
- 6-bit quantization (INT6)
- 7-bit quantization (INT7)
- 8-bit quantization (INT8)
- INT16 quantization
- INT32 quantization
Quantization hybrids:
- Integer-only arithmetic quantization
- QLoRa (Quantized Low-Rank)
Floating-point quantization
- FP16 quantization
- FP8 quantization
- FP4 quantization
Other types of quantization:
- Mixed precision quantization
- Fixed-point quantization (integer quantization)
- Logarithmic power-of-two quantization (bitshift quantization)
- Double bitshift power-of-two quantization
- Division quantization
- Cluster-based quantization (Weight clustering)
- Dyadic quantization
- Fake quantization
- Simulated quantization
- Stochastic quantization (probabilistic)

Distillation

Parameter Sharing

Attention Optimization

Transformer Component Optimizations

Transformer architectures (overview)
Layers:
Activations:
Normalization:
- Norm optimizations
- Approximate normalization
- Norm reordering (pre-norm/post-norm)
- Integer-only normalization
- Normalization alternatives/replacements
- Fused normalization (e.g. "fused LayerNorm" in kernel fusion)
Softmax:
- Softmax optimizations
- Approximate Softmax
- Softmax alternatives/replacements
- Integer-only Softmax
- Fused Softmax (kernel fusion)
Feed-Forward Network (FFN):
- FFN pruning
- FFN approximation
- FFN alternatives/replacements
- Integer-only FFN
- Bias optimizations
- Fused add-bias (see kernel fusion)
- Bias vector pruning
MatMul/GEMM operations:
- Faster matrix multiplication (e.g. Winograd, Strassen)
- Approximate matrix multiplication
- Transpose cache
- Fused multiply-add (FMA)
- Fused transpose
- Vector dot product optimization
Positional Encoding (PE):
- Positional encoding optimization
- Pruning positional encoding (removal)
- Positional encoding approximation
- Integer-only positional encoding
Decoding algorithms:
Other:
- Approximate top-k algorithms

Transformer General Optimizations

KV Caching Optimizations

KV caching
KV caching in early exit
KV cache compression
KV cache sparsity
KV cache token pruning
KV cache eviction policies
KV cache quantization
KV cache layer fusion
KV cache layer pruning
KV cache reuse
KV cache global (multi-query KV caching)
Prefix KV cache
Session KV cache (multi-turn KV caching)
Substring KV cache (Lengthwise-fused KV caching)

Non-Multiplication AI Models

Prefill Phase Optimizations

Prefill optimizations generally (overview)
Chunked prefill
Disaggregated prefill scheduling
Context cache (global KV caching)
Prefix KV cache

Computation Optimizations

General Coding Efficiency

Loop Optimizations

Loop optimizations (overview)
Loop fusion (merging loops)
Loop unrolling
Loop perforation
Loop reordering
Loop tiling
Loop reversal
Loop fission (splitting a loop)
Loop interleave
Loop interchange
Loop coalescing
Loop-invariant code motion ("hoisting")
Loop distribution
Pointer arithmetic
Loop peeling (unrolling first iterations)
Loop splitting Loop sentinel
Loop collapsing
Loop normalization
Loop strip mining (Loop sectioning)
Loop skewing
Loop spreading

Memory Utilization Optimizations

Numeric Representation Optimizations

Fixed point number system (FXP) optimizations
Floating point number system (FLP) optimizations
Foating point bitwise arithmetic
IEEE 754 floating point optimizations
Binary quantization
Ternary quantization

Advanced Number Systems

Posit number system (PNS)
Residue number system (RNS)
Logarithmic number system (LNS)
Dyadic numbers
Double-base number system (DBNS)
Dynamic number systems
Hybrid number systems
Tropical algebra (max-plus)
MiniMax algebra
Multi-dimensional logarithmic number system (MDLNS)
Multiple-Base Number System (MBNS)
Semi-Logarithmic Number System (SLNS)
Lattice algebra

Faster Arithmetic

Low-Rank Matrices

LoRa
Low-rank matrices
QLoRa (Quantized Low-Rank)
Matrix factorization/tensor decomposition

Advanced Matrices

Data Structures

Multi-AI Architectures

Model selection algorithms
Ensemble inference (multi-model AI engines)
Big-little architectures
Cascades
Cloud inference servers
Collaborative inference
Consensus decoding
Mixture of Experts (MoE)
Speculative decoding
Generalized speculative decoding
Swarm ensemble architectures
Committee ensemble architectures
Ensemble averaging