摘要
为了解决神经网络深度不断加深带来结构冗余的问题,提出了一种基于改进知识蒸馏算法的轻量型神经网络设计方法。该方法中,轻量型神经网络的卷积结构使用分组卷积与点式卷积相结合的残差结构,并结合基于风格迁移与特征重建的知识蒸馏算法对模型进行训练。网络在Cifar10和Cifar100中的实验结果表明,在准确率相当的前提下,轻量型神经网络的参数数量比普通残差网络减少了20%以上。同时,两种知识蒸馏算法结果的相近性说明分组卷积的人工设计部分对网络的影响较小。
In order to solve the problem of structural redundancy caused by the deepening of neural network depth,a lightweight neural network design method based on the improved knowledge distillation algorithm was proposed.In this method,the convolution structure of the lightweight neural network uses a residual structure combining grouped convolution and point convolution,and combines the knowledge distillation algorithm based on the style transfer and feature reconstruction to train the model.The experimental results of the network in Cifar10 and Cifar100 show that the number of parameters of the lightweight neural network is reduced by more than 20%compared with the ordinary residual network on the premise of equivalent accuracy.At the same time,the similarity of the results of the two knowledge distillation algorithms shows that the artificially designed part of the grouped convolution has little influence on the network.
作者
郭俊伦
彭书华
李俊杰
GUO Junlun;PENG Shuhua;LI Junjie(School of Automation,Beijing Information Science&Technology University,Beijing 100192,China)
出处
《北京信息科技大学学报（自然科学版）》
2021年第4期20-24,共5页
Journal of Beijing Information Science and Technology University
基金
国家自然科学基金资助项目(61801032)。
关键词
深度神经网络
轻量型神经网络
知识蒸馏
特征重建
风格迁移
deep neural network
lightweight neural network
knowledge distillation
feature reconstruction
style transfer
