Research Repository

Industrial Cyber-Physical Systems-based Cloud IoT Edge for Federated Heterogeneous Distillation.

Wang, Chengjia and Yang, Guang and Papanastasiou, Giorgos and Zhang, Heye and Rodrigues, Joel JPC and de Albuquerque, Victor Hugo C (2021) 'Industrial Cyber-Physical Systems-based Cloud IoT Edge for Federated Heterogeneous Distillation.' IEEE Transactions on Industrial Informatics, 17 (8). pp. 5511-5521. ISSN 1551-3203

ALL_TII-20-1728.pdf - Accepted Version

Download (2MB) | Preview


Deep convoloutional networks have achieved remarkable performance in a wide range of vision-based tasks in modern internet of things (IoT). Due to privacy issue and transmission cost, mannually annotated data for training the deep learning models are usually stored in different sites with fog and edge devices of various computing capacity. It has been proved that knowledge distillation technique can effectively compress well trained neural networks into light-weight models suitable to particular devices. However, different fog and edge devices may perform different sub-tasks, and simplely performing model compression on powerful cloud servers failed to make use of the private data sotred at different sites. To overcome these obstacles, we propose an novel knowledge distillation method for object recognition in real-world IoT sencarios. Our method enables flexible bidirectional online training of heterogeneous models distributed datasets with a new ``brain storming'' mechanism and optimizable temperature parameters. In our comparison experiments, this heterogeneous brain storming method were compared to multiple state-of-the-art single-model compression methods, as well as the newest heterogeneous and homogeneous multi-teacher knowledge distillation methods. Our methods outperformed the state of the arts in both conventional and heterogeneous tasks. Further analysis of the ablation expxeriment results shows that introducing the trainable temperature parameters into the conventional knowledge distillation loss can effectively ease the learning process of student networks in different methods. To the best of our knowledge, this is the IoT-oriented method that allows asynchronous bidirectional heterogeneous knowledge distillation in deep networks.

Item Type: Article
Uncontrolled Keywords: Clouds; Training; Task analysis; Servers; Computational modeling; Knowledge engineering; Training data; Deep learning; heterogeneous classifiers; Internet of Things (IoT); knowledge distillation (KD); online learning
Divisions: Faculty of Science and Health
Faculty of Science and Health > Computer Science and Electronic Engineering, School of
SWORD Depositor: Elements
Depositing User: Elements
Date Deposited: 17 Jul 2020 09:11
Last Modified: 18 Aug 2022 12:25

Actions (login required)

View Item View Item