Aussie AI

Slimmable Neural Networks

  • Book Excerpt from "Generative AI in C++"
  • by David Spuler, Ph.D.

Slimmable Neural Networks

Slimmable neural networks are optimized with a type of dynamic width pruning. They can adjust their width dynamically at runtime, and are thus a type of adaptive inference. There are various research papers on these methods, but they have not achieved widespread usage in industry models.

Some of the papers on “slimmer” or “thin” models via width pruning include:

  1. J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang. 2018, Slimmable neural networks, In International Conference on Learning Representations, 2018. https://arxiv.org/abs/1812.08928, Code: https://github.com/JiahuiYu/slimmable_networks (The earliest paper that coined the term “slimmable networks”.)
  2. J. Yu and T. S. Huang. 2019, Universally slimmable networks and improved training techniques, In IEEE International Conference on Computer Vision, pages 1803–1811, 2019. https://arxiv.org/abs/1903.05134, Code: https://github.com/JiahuiYu/slimmable_networks
  3. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017, Learning efficient convolutional networks through network slimming, In Computer Vision (ICCV), 2017 IEEE International Conference on, pp. 2755–2763. IEEE, 2017. https://arxiv.org/abs/1708.06519
  4. Sergio Guadarrama, Nathan Silberman, 2016, TensorFlow-Slim: A lightweight library for defining, training and evaluating complex models in TensorFlow, Google Research, Code: https://github.com/google-research/tf-slim
  5. Yu and T. S. Huang. 2019, Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers, Preprint arXiv:1903.11728, 2019. PDF: https://arxiv.org/pdf/1903.11728v1.pdf
  6. Jiahui Yu, Thomas Huang, June 2019, AutoSlim: Towards One-Shot Architecture Search for Channel Numbers, https://arxiv.org/abs/1903.11728, Code: https://github.com/JiahuiYu/slimmable_networks
  7. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014, Fitnets: Hints for thin deep nets, arXiv preprint arXiv:1412.6550, 2014. https://arxiv.org/abs/1412.6550
  8. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen, 2019, MobileNetV2: Inverted Residuals and Linear Bottlenecks, Mar 2019, https://arxiv.org/abs/1801.04381, Code: https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet (MobileNetV2 uses some slimming techniques with TensorFlow-Slim.)
  9. Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, Xiaojun Chang, 2021, Dynamic Slimmable Network, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 8607-8617, https://arxiv.org/abs/2103.13258, http://openaccess.thecvf.com/content/CVPR2021/html/Li_Dynamic_Slimmable_Network_CVPR_2021_paper.html, Code: https://github.com/changlin31/DS-Net
  10. Ting-Wu Chin, Ari S. Morcos, Diana Marculescu, 2020, Pareco: Pareto-aware channel optimization for slimmable neural networks, https://arxiv.org/abs/2007.11752v2, https://openreview.net/forum?id=SPyxaz_h9Nd
  11. Won Joon Yun; Yunseok Kwak; Hankyul Baek; Soyi Jung; Mingyue Ji; Mehdi Bennis; Jihong Park, 2023, SlimFL: Federated learning with superposition coding over slimmable neural networks, IEEE/ACM Transactions on Networking (Early Access), DOI: 10.1109/TNET.2022.3231864 https://ieeexplore.ieee.org/document/10004844 https://arxiv.org/abs/2203.14094
  12. Ting-Wu Chin, Ari S. Morcos & Diana Marculescu, 2021, Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks, Lecture Notes in Computer Science book series (LNAI,volume 12977) Springer, https://link.springer.com/chapter/10.1007/978-3-030-86523-8_8, https://arxiv.org/pdf/2007.11752, Code: https://github.com/cmu-enyac/Joslim
  13. Hideaki Kuratsu, Atsuyoshi Nakamura, 2022, Slimmable Pruned Neural Networks, arXiv preprint arXiv:2212.03415, https://arxiv.org/abs/2212.03415 Code: https://github.com/hideakikuratsu/SP-Net
  14. A Ozerov, A Lambert, SK Kumaraswamy, 2021, ParaDiS: Parallelly Distributable Slimmable Neural Networks, arXiv preprint arXiv:2110.02724, https://arxiv.org/abs/2110.02724
  15. L Hou, Z Yuan, L Huang, H Shen, X Cheng, 2021, Slimmable generative adversarial networks, Proceedings of the AAAI Conference on Artificial Intelligence, 35, No. 9: AAAI-21 Technical Tracks 9, https://ojs.aaai.org/index.php/AAAI/article/view/16946, https://arxiv.org/abs/2012.05660
  16. F Yang, L Herranz, Y Cheng, 2021, Slimmable compressive autoencoders for practical neural image compression, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4998-5007, http://openaccess.thecvf.com/content/CVPR2021/html/Yang_Slimmable_Compressive_Autoencoders_for_Practical_Neural_Image_Compression_CVPR_2021_paper.html , https://arxiv.org/abs/2103.15726
  17. Zuhaib Akhtar, Mohammad Omar Khursheed, Dongsu Du, Yuzong Liu, Apr 2023, Small-footprint slimmable networks for keyword spotting, ICASSP 2023: 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), https://arxiv.org/abs/2304.12183
  18. Yu, Jiahui, 2019, Slimmable neural networks for edge devices, Masters Thesis, Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, https://www.ideals.illinois.edu/items/112252
  19. Rang Meng, Weijie Chen, Shicai Yang, Jie Song, Luojun Lin, Di Xie, Shiliang Pu, Xinchao Wang, Mingli Song, Yueting Zhuang, June 2022, Slimmable domain adaptation, http://openaccess.thecvf.com/content/CVPR2022/html/Meng_Slimmable_Domain_Adaptation_CVPR_2022_paper.html, https://arxiv.org/abs/2206.06620, Code: https://github.com/hikvision-research/SlimDA
  20. Juliano S. Assine, J. C. S. Santos Filho, Eduardo Valle, Marco Levorato, 2023, Slimmable Encoders for Flexible Split DNNs in Bandwidth and Resource Constrained IoT Systems, arXiv preprint arXiv:2306.12691, June 2023, https://arxiv.org/abs/2306.12691
  21. Li Yang, Zhezhi He, Yu Cao, Deliang Fan, Sep 2020, A Progressive Sub-Network Searching Framework for Dynamic Inference, https://arxiv.org/abs/2009.05681
  22. Dawei Li, Xiaolong Wang, and Deguang Kong. 2018. DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices, In AAAI’18, https://arxiv.org/abs/1708.04728
  23. Hankook Lee and Jinwoo Shin. 2018. Anytime neural prediction via slicing networks vertically, arXiv preprint arXiv:1807.02609. https://arxiv.org/abs/1807.02609
  24. Junjie He, Yinzhang Ding, Ming Zhang, Dongxiao Li, 2022, Towards efficient network compression via Few-Shot Slimming, Neural Networks, Volume 147, 2022, pp. 113-125, https://doi.org/10.1016/j.neunet.2021.12.011, https://pubmed.ncbi.nlm.nih.gov/34999388/
  25. Mojtaba Valipour, Mehdi Rezagholizadeh, Hossein Rajabzadeh, Marzieh Tahaei, Boxing Chen, Ali Ghodsi, 2023, SortedNet, a Place for Every Network and Every Network in its Place: Towards a Generalized Solution for Training Many-in-One Neural Networks, https://arxiv.org/abs/2309.00255 (Generalization of multi-dimensional pruning, by training a large neural network with many sub-networks across different width and depth dimensions.)
  26. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H., 2017, Mobilenets: Efficient convolutional neural networks for mobile vision applications (2017), https://doi.org/10.48550/ARXIV.1704.04861, https://arxiv.org/abs/1704.04861 (Combines depthwise separable convolutions and thinning at each layer.)
  27. N Penkov, K Balaskas, M Rapp, J Henkel, 2023, Differentiable Slimming for Memory-Efficient Transformers, IEEE Embedded Systems Letters (Early Access), DOI: 10.1109/LES.2023.3299638, https://ieeexplore.ieee.org/abstract/document/10261943

For additional research papers on slimmable networks, see https://www.aussieai.com/research/width-pruning#slimmable.

 

• Next:

• Up: Table of Contents

• Buy: Generative AI in C++: Coding Transformers and LLMs

Generative AI in C++ The new AI programming book by Aussie AI co-founders:
  • AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++