No Cover Image

Journal article 20 views 2 downloads

Deep Collaborative Learning for Randomly Wired Neural Networks / Ehab Essa, Xianghua Xie

Electronics, Volume: 10, Issue: 14, Start page: 1669

Swansea University Author: Xianghua Xie

  • electronics-10-01669.pdf

    PDF | Version of Record

    Copyright: © 2021 by the authors. This is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license

    Download (790.29KB)

Abstract

A deep collaborative learning approach is introduced in which a chain of randomly wired neural networks is trained simultaneously to improve the overall generalization and form a strong ensemble model. The proposed method takes advantage of functional-preserving transfer learning and knowledge disti...

Full description

Published in: Electronics
ISSN: 2079-9292
Published: MDPI AG 2021
Online Access: Check full text

URI: https://cronfa.swan.ac.uk/Record/cronfa57529
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract: A deep collaborative learning approach is introduced in which a chain of randomly wired neural networks is trained simultaneously to improve the overall generalization and form a strong ensemble model. The proposed method takes advantage of functional-preserving transfer learning and knowledge distillation to produce an ensemble model. Knowledge distillation is an effective learning scheme for improving the performance of small neural networks by using the knowledge learned by teacher networks. Most of the previous methods learn from one or more teachers but not in a collaborative way. In this paper, we created a chain of randomly wired neural networks based on a random graph algorithm and collaboratively trained the models using functional-preserving transfer learning, so that the small network in the chain could learn from the largest one simultaneously. The training method applies knowledge distillation between randomly wired models, where each model is considered as a teacher to the next model in the chain. The decision of multiple chains of models can be combined to produce a robust ensemble model. The proposed method is evaluated on CIFAR-10, CIFAR-100, and TinyImageNet. The experimental results show that the collaborative training significantly improved the generalization of each model, which allowed for obtaining a small model that can mimic the performance of a large model and produce a more robust ensemble approach.
Keywords: randomly wired neural networks; model distillation; ensemble model; deep learning
College: College of Science
Issue: 14
Start Page: 1669