LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging

LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging

Jinuk Kim1, 2, Marwa El Halabi3, Mingi Ji4, Hyun Oh Song1, 2,
1Seoul National University, 2Neural Processing Research Center, 3Samsung - SAIT AI Lab, Montreal, 4Google
Paper Video Code Poster

LayerMerge reduces depth of the CNN by pruning and merging convolution and activation layers.

Overview

LayerMerge is a novel method to make convolutional neural networks (CNNs) more efficient without losing performance. Traditional methods for reducing network depth usually follow one of two approaches:

1. Pruning Convolution Layers: Aggressively removes parameters, risking loss of important information.

2. Pruning Activation Layers and Merging Layers: Eliminates redundant activation layers and merges resulting consecutive convolution layers, potentially increasing kernel sizes and negating speed gains.

LayerMerge addresses these issues by jointly pruning convolution layers and activation functions. It optimizes which layers to remove, speeding up inference while minimizing performance loss. Since this selection process involves an exponential search space, we formulate a novel surrogate optimization problem and efficiently solve it via dynamic programming.

Our results show that LayerMerge outperforms current methods for reducing network depth in tasks including image classification and generation.

Demo

We demonstrate the effectiveness of LayerMerge with MobileNetV2-1.0 on ImageNet and with DDPM on CIFAR10. The jupyter notebook code for the demo can be found here.

Results

We evaluate compression methods in ResNet-34, MobileNetV2-(1.0/1.4) on ImageNet, and DDPM on CIFAR10. Results show that LayerMerge outperforms current methods for reducing network depth in tasks including image classification and generation.

BibTeX

@inproceedings{kim2024layermerge,
      title={LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging}, 
      author={Kim, Jinuk and El Halabi, Marwa and Ji, Mingi and Song, Hyun Oh},
      booktitle = {International Conference on Machine Learning (ICML)},
      year={2024}
}
Template of this post is based on Nerfies website.