Wang, Chengjia and Yang, Guang and Papanastasiou, Giorgos and Zhang, Heye and Rodrigues, Joel JPC and de Albuquerque, Victor Hugo C (2021) Industrial Cyber-Physical Systems-based Cloud IoT Edge for Federated Heterogeneous Distillation. IEEE Transactions on Industrial Informatics, 17 (8). pp. 5511-5521. DOI https://doi.org/10.1109/TII.2020.3007407
Wang, Chengjia and Yang, Guang and Papanastasiou, Giorgos and Zhang, Heye and Rodrigues, Joel JPC and de Albuquerque, Victor Hugo C (2021) Industrial Cyber-Physical Systems-based Cloud IoT Edge for Federated Heterogeneous Distillation. IEEE Transactions on Industrial Informatics, 17 (8). pp. 5511-5521. DOI https://doi.org/10.1109/TII.2020.3007407
Wang, Chengjia and Yang, Guang and Papanastasiou, Giorgos and Zhang, Heye and Rodrigues, Joel JPC and de Albuquerque, Victor Hugo C (2021) Industrial Cyber-Physical Systems-based Cloud IoT Edge for Federated Heterogeneous Distillation. IEEE Transactions on Industrial Informatics, 17 (8). pp. 5511-5521. DOI https://doi.org/10.1109/TII.2020.3007407
Abstract
Deep convoloutional networks have achieved remarkable performance in a wide range of vision-based tasks in modern internet of things (IoT). Due to privacy issue and transmission cost, mannually annotated data for training the deep learning models are usually stored in different sites with fog and edge devices of various computing capacity. It has been proved that knowledge distillation technique can effectively compress well trained neural networks into light-weight models suitable to particular devices. However, different fog and edge devices may perform different sub-tasks, and simplely performing model compression on powerful cloud servers failed to make use of the private data sotred at different sites. To overcome these obstacles, we propose an novel knowledge distillation method for object recognition in real-world IoT sencarios. Our method enables flexible bidirectional online training of heterogeneous models distributed datasets with a new ``brain storming'' mechanism and optimizable temperature parameters. In our comparison experiments, this heterogeneous brain storming method were compared to multiple state-of-the-art single-model compression methods, as well as the newest heterogeneous and homogeneous multi-teacher knowledge distillation methods. Our methods outperformed the state of the arts in both conventional and heterogeneous tasks. Further analysis of the ablation expxeriment results shows that introducing the trainable temperature parameters into the conventional knowledge distillation loss can effectively ease the learning process of student networks in different methods. To the best of our knowledge, this is the IoT-oriented method that allows asynchronous bidirectional heterogeneous knowledge distillation in deep networks.
Item Type: | Article |
---|---|
Uncontrolled Keywords: | Clouds; Training; Task analysis; Servers; Computational modeling; Knowledge engineering; Training data; Deep learning; heterogeneous classifiers; Internet of Things (IoT); knowledge distillation (KD); online learning |
Divisions: | Faculty of Science and Health Faculty of Science and Health > Computer Science and Electronic Engineering, School of |
SWORD Depositor: | Unnamed user with email elements@essex.ac.uk |
Depositing User: | Unnamed user with email elements@essex.ac.uk |
Date Deposited: | 17 Jul 2020 09:11 |
Last Modified: | 30 Oct 2024 16:29 |
URI: | http://repository.essex.ac.uk/id/eprint/28152 |
Available files
Filename: ALL_TII-20-1728.pdf