Movement Pruning

Movement Pruning

Another type of unstructured pruning is “movement pruning”. This is a pruning of weights during training, based on how they change during the training process. Weights that are “moving” towards zero as training progresses are removed (pruned), whereas weights “moving away from zero” are retained.

Movement pruning can also have meta-parameters for the sparsity targets or multiple thresholds for the magnitude of values to consider as candidates for pruning, and the rate-of-change thresholds for deciding whether to prune.

Note that this pruning criteria relates to both positive and negative weights (with opposite movement directions). It's also possible to combine a movement threshold and an absolute magnitude threshold, so that important weights with large absolute values are not subsequently removed if they change a tiny amount towards zero. Overall, this means movement pruning focuses on the changes to weights as they are trained, rather than on their absolute value at the end of training.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++