First-Order and Second-Order Pruning

First-Order and Second-Order Pruning

Magnitude pruning is the simplest type of unstructured pruning, which only considers the current value of the weight during training. It is called a “first-order” pruning method because it doesn't consider the prior values or how they are changing.

Movement pruning is a generalization of magnitude pruning, which considers the “rate of change” of the weight, as it changes with each training step. This is called a “second-order” pruning method because it is considering the “gradient” or “derivative” in the mathematical sense.

The slope of the curve helps decide whether to prune a weight. For example, if a weight has a magnitude below a threshold and its rate of change is decreasing so that it is moving towards zero, then it is pruned to zero. In this way, tiny weights that are still increasing in magnitude, whether positive or negative, are not yet pruned.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++