Pruning (artificial neural network)
In deep learning, pruning is the practice of removing parameters from an existing artificial neural network.[1] The goal of this process is to reduce the size (parameter count) of the neural network (and therefore the computational resources required to run it) whilst maintaining accuracy. This can be compared to the biological process of synaptic pruning which takes place in mammalian brains during development.[2]
Node (neuron) pruning
[edit]A basic algorithm for pruning is as follows:[3][4]
- Evaluate the importance of each neuron.
- Rank the neurons according to their importance (assuming there is a clearly defined measure for "importance").
- Remove the least important neuron.
- Check a termination condition (to be determined by the user) to see whether to continue pruning.
Edge (weight) pruning
[edit]Most work on neural network pruning does not remove full neurons or layers. Instead, it focuses on removing the most insignificant weights, namely, setting their values to zero. This can either be done globally by comparing weights from all layers in the network or locally by comparing weights in each layer separately.[5]
Different metrics can be used to measure the importance of each weight. Weight magnitude as well as combinations of weight and gradient information are commonly used metrics.[6][7][8]
Early work suggested also to change the values of non-pruned weights.[9]
When to prune the neural network?
[edit]Pruning can be applied at three different stages: before training, during training, or after training. When pruning is performed during or after training, additional fine-tuning epochs are typically required. Each approach involves different trade-offs between accuracy and computational cost[5].
See also
[edit]References
[edit]- ^ Blalock, Davis; Ortiz, Jose Javier Gonzalez; Frankle, Jonathan; Guttag, John (2020-03-06). "What is the State of Neural Network Pruning?". arXiv:2003.03033 [cs.LG].
- ^ Chechik, Gal; Meilijson, Isaac; Ruppin, Eytan (October 1998). "Synaptic Pruning in Development: A Computational Account". Neural Computation. 10 (7): 1759–1777. doi:10.1162/089976698300017124. ISSN 0899-7667. PMID 9744896. S2CID 14629275.
- ^ Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440.
- ^ Gildenblat, Jacob (2017-06-23). "Pruning deep neural networks to make them fast and small". Github. Retrieved 2024-02-04.
- ^ a b Cheng, Hongrong; Zhang, Miao; Shi, Javen Qinfeng (December 2024). "A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations". IEEE Transactions on Pattern Analysis and Machine Intelligence. 46 (12): 10558–10578. doi:10.1109/TPAMI.2024.3447085. ISSN 0162-8828.
- ^ LeCun, Yann; Denker, John; Solla, Sara (1989). "Optimal Brain Damage". Advances in Neural Information Processing Systems. 2. Morgan-Kaufmann.
- ^ Liu, Shiwei; Chen, Tianlong; Chen, Xiaohan; Atashgahi, Zahra; Yin, Lu; Kou, Huanyu; Shen, Li; Pechenizkiy, Mykola; Wang, Zhangyang (2022-02-06), Sparse Training via Boosting Pruning Plasticity with Neuroregeneration, arXiv, doi:10.48550/arXiv.2106.10404, arXiv:2106.10404, retrieved 2025-10-21
- ^ Zhu, Lingkai; Bezek, Can Deniz; Goksel, Orcun (2025). Petersen, Jens; Dahl, Vedrana Andersen (eds.). "FGGP: Fixed-Rate Gradient-First Gradual Pruning". Image Analysis. Cham: Springer Nature Switzerland: 3–15. doi:10.1007/978-3-031-95911-0_1. ISBN 978-3-031-95911-0.
- ^ Chechik, Gal; Meilijson, Isaac; Ruppin, Eytan (April 2001). "Effective Neuronal Learning with Ineffective Hebbian Learning Rules". Neural Computation. 13 (4): 817–840. doi:10.1162/089976601300014367. ISSN 0899-7667. PMID 11255571. S2CID 133186.
 | This artificial neural network-related article is a stub. You can help Wikipedia by expanding it. |