Aussie AI

Channel Pruning

  • Book Excerpt from "Generative AI in C++"
  • by David Spuler, Ph.D.

Channel Pruning

Channel pruning is a structured pruning method for CNNs. By reducing the number of channels in a feature map, the “width” of the network is reduced. This method does not apply to Transformer architectures, but is analogous to attention head pruning.

Various research papers examine the method of “channel pruning”, which is a type of width pruning. Research papers on channel pruning:

  1. Y. He, X. Zhang, and J. Sun, 2017, Channel pruning for accelerating very deep neural networks, in Computer Vision (ICCV), 2017 IEEE International Conference on. IEEE, 2017, pp. 1398–1406, https://arxiv.org/abs/1707.06168
  2. J. Ye, X. Lu, Z. Lin, and J. Z. Wang, 2018, Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers, arXiv preprint arXiv:1802.00124, 2018. https://arxiv.org/abs/1802.00124
  3. N. Lee, T. Ajanthan, and P. H. Torr, 2018, Snip: Single-shot network pruning based on connection sensitivity, arXiv preprint arXiv:1810.02340, 2018. https://arxiv.org/abs/1810.02340
  4. Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. 2017, Shufflenet: An extremely efficient convolutional neural network for mobile devices, CoRR, abs/1707.01083, 2017. https://arxiv.org/abs/1707.01083 (Uses “channel shuffle” which is similar to channel pruning.)
  5. W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. 2016, Learning structured sparsity in deep neural networks, In Advances in Neural Information Processing Systems, pages 2074–2082, 2016, https://arxiv.org/abs/1608.03665 Code: https://github.com/wenwei202/caffe/tree/scnn
  6. Min Wang, Baoyuan Liu, and Hassan Foroosh. 2016, Design of efficient convolutional layers using single intra-channel convolution, topological subdivisioning and spatial ”bottleneck” structure, CoRR, abs/1608.04337, 2016, https://arxiv.org/abs/1608.04337
  7. Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, Yonghong Tian, June 2020, Channel Pruning via Automatic Structure Search, arXiv preprint, https://arxiv.org/abs/2001.08565
  8. Jing Liu; Bohan Zhuang; Zhuangwei Zhuang; Yong Guo; Junzhou Hua, 2018, Discrimination-aware channel pruning for deep neural networks, IEEE Transactions on Pattern Analysis and Machine Intelligence (Volume 44, Issue 8, 01 August 2022), https://ieeexplore.ieee.org/abstract/document/9384353
  9. Xitong Gao, Yiren Zhao, Łukasz Dudziak, Robert Mullins, Cheng-zhong Xu, Jan 2019, Dynamic Channel Pruning: Feature Boosting and Suppression, https://arxiv.org/abs/1810.05331
  10. Hanyu Peng, Jiaxiang Wu, Shifeng Chen, Junzhou Huang, 2019, Collaborative channel pruning for deep networks, Proceedings of the 36th International Conference on Machine Learning, PMLR 97:5113-5122, 2019. http://proceedings.mlr.press/v97/peng19c/peng19c.pdf
  11. Jinyang Guo; Weichen Zhang; Wanli Ouyang; Dong Xu, 2021, Model compression using progressive channel pruning, IEEE Transactions on Circuits and Systems for Video Technology (Volume 31, Issue 3, March 2021), https://ieeexplore.ieee.org/document/9097925
  12. J Guo, W Ouyang, D Xu, 2020, Channel pruning guided by classification loss and feature importance, Proceedings of the AAAI Conference, Association for the Advancement of Artificial Intelligence, https://aaai.org/ojs/index.php/AAAI/article/view/6720/6574
  13. Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Tim Kwang-Ting Cheng, Jian Sun, Aug 2019, MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning, https://arxiv.org/abs/1903.10258, http://openaccess.thecvf.com/content_ICCV_2019/papers/Liu_MetaPruning_Meta_Learning_for_Automatic_Neural_Network_Channel_Pruning_ICCV_2019_paper.pdf
  14. Y Huang, N Liu, Z Che, Z Xu, C Shen, 2023, Yaomin Huang, Ning Liu, Zhengping Che, Zhiyuan Xu, Chaomin Shen, Yaxin Peng, Guixu Zhang, Xinmei Liu, Feifei Feng, Jian Tang, 2023, CP3: Channel Pruning Plug-In for Point-Based Networks, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), https://ieeexplore.ieee.org/document/10203696, https://arxiv.org/abs/2303.13097, http://openaccess.thecvf.com/content/CVPR2023/papers/Huang_CP3_Channel_Pruning_Plug-In_for_Point-Based_Networks_CVPR_2023_paper.pdf
  15. Y Liu, D Wu, W Zhou, K Fan, Z Zhou, 2023, EACP: An effective automatic channel pruning for neural networks, Neurocomputing, Volume 526, 14 March 2023, Pages 131-142, https://www.sciencedirect.com/science/article/pii/S0925231223000255
  16. Hancheng Ye, Bo Zhang, Tao Chen, Jiayuan Fan, Bin Wang, 2023, Performance-aware Approximation of Global Channel Pruning for Multitask CNNs, IEEE Transactions on Pattern Analysis and Machine Intelligence ( Volume: 45, Issue: 8, August 2023), https://ieeexplore.ieee.org/document/10083285, https://arxiv.org/abs/2303.11923, Code: http://www.github.com/HankYe/PAGCP.git
  17. Kang, M. and Han, B., 2020, Operation-aware soft channel pruning using differentiable masks, In ICML, 2020, https://arxiv.org/abs/2007.03938
  18. Manoj Alwani, Yang Wang, Vashisht Madhavan, Feb 2022, DECORE: Deep Compression with Reinforcement Learning, in CVPR, 2022, https://arxiv.org/abs/2106.06091
  19. Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Sehoon Kim, Michael W Mahoney, Kurt Keutzer, 2022, Hessian-Aware Pruning and Optimal Neural Implant, in WACV, 2022. https://arxiv.org/abs/2101.08940
  20. Weizhe Hua, Yuan Zhou, Christopher M De Sa, Zhiru Zhang, and G Edward Suh. 2019, Channel gating neural networks, NeurIPS, pages 1884–1894, 2019, https://arxiv.org/abs/1805.12549
  21. Zhenda Xie, Zheng Zhang, Xizhou Zhu, Gao Huang, and Stephen Lin. 2020. Spatially adaptive inference with stochastic feature sampling and interpolation, arXiv preprint arXiv:2003.08866, https://arxiv.org/abs/2003.08866
  22. Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. 2017, Runtime neural pruning, In NeurIPS, pages 2181–2191, 2017. PDF: https://dl.acm.org/doi/pdf/10.5555/3294771.3294979

For additional research papers on channel pruning, see https://www.aussieai.com/research/width-pruning#channel.

 

• Next:

• Up: Table of Contents

• Buy: Generative AI in C++: Coding Transformers and LLMs

Generative AI in C++ The new AI programming book by Aussie AI co-founders:
  • AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++