中图分类号：TP399 文献标志码：A DOI: 10.16157/j.issn.0258-7998.233812
中文引用格式：顾明珠，明瑞成，邱创一，等. 一种多教师模型知识蒸馏深度神经网络模型压缩算法[J]. 电子技术应用，2023，49(8)：7-12.
英文引用格式：Gu Mingzhu，Ming Ruicheng，Qiu Chuangyi，et al. A multi-teacher knowledge distillation model compression algorithm for deep neural network[J]. Application of Electronic Technique，2023，49(8)：7-12.

A multi-teacher knowledge distillation model compression algorithm for deep neural network

Abstract：In order to minimize the accuracy loss when compressing huge deep learning models and deploying them to devices with limited computing power and storage capacity, a knowledge distillation model compression method is investigated and an improved multi-teacher model knowledge distillation compression algorithm with filtering is proposed. Taking advantage of the integration of multi-teacher models, the better-performing teacher models are screened for student instruction using the predicted cross-entropy of each teacher model as the quantitative criterion for screening, and the student models are allowed to extract information starting from the feature layer of the teacher models, while the better-performing teacher models are allowed to have more say in the instruction. The experimental results of classification models such as VGG13 on the CIFAR100 dataset show that the multi-teacher model compression method in this paper has better performance in terms of accuracy compared with other compression algorithms with the same size of the final obtained student models.

Key words :model compression；distillation of knowledge；multi-teacher model；cross entropy；feature layer