No Cover Image

Journal article 645 views 141 downloads

Deep Collaborative Learning for Randomly Wired Neural Networks

Ehab Essa, Xianghua Xie Orcid Logo

Electronics, Volume: 10, Issue: 14, Start page: 1669

Swansea University Author: Xianghua Xie Orcid Logo

  • electronics-10-01669.pdf

    PDF | Version of Record

    Copyright: © 2021 by the authors. This is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license

    Download (790.29KB)

Check full text

DOI (Published version): 10.3390/electronics10141669

Abstract

A deep collaborative learning approach is introduced in which a chain of randomly wired neural networks is trained simultaneously to improve the overall generalization and form a strong ensemble model. The proposed method takes advantage of functional-preserving transfer learning and knowledge disti...

Full description

Published in: Electronics
ISSN: 2079-9292
Published: MDPI AG 2021
Online Access: Check full text

URI: https://cronfa.swan.ac.uk/Record/cronfa57529
Tags: Add Tag
No Tags, Be the first to tag this record!
first_indexed 2021-08-05T10:45:58Z
last_indexed 2021-09-10T03:20:15Z
id cronfa57529
recordtype SURis
fullrecord <?xml version="1.0"?><rfc1807><datestamp>2021-09-09T15:26:54.2304104</datestamp><bib-version>v2</bib-version><id>57529</id><entry>2021-08-05</entry><title>Deep Collaborative Learning for Randomly Wired Neural Networks</title><swanseaauthors><author><sid>b334d40963c7a2f435f06d2c26c74e11</sid><ORCID>0000-0002-2701-8660</ORCID><firstname>Xianghua</firstname><surname>Xie</surname><name>Xianghua Xie</name><active>true</active><ethesisStudent>false</ethesisStudent></author></swanseaauthors><date>2021-08-05</date><deptcode>SCS</deptcode><abstract>A deep collaborative learning approach is introduced in which a chain of randomly wired neural networks is trained simultaneously to improve the overall generalization and form a strong ensemble model. The proposed method takes advantage of functional-preserving transfer learning and knowledge distillation to produce an ensemble model. Knowledge distillation is an effective learning scheme for improving the performance of small neural networks by using the knowledge learned by teacher networks. Most of the previous methods learn from one or more teachers but not in a collaborative way. In this paper, we created a chain of randomly wired neural networks based on a random graph algorithm and collaboratively trained the models using functional-preserving transfer learning, so that the small network in the chain could learn from the largest one simultaneously. The training method applies knowledge distillation between randomly wired models, where each model is considered as a teacher to the next model in the chain. The decision of multiple chains of models can be combined to produce a robust ensemble model. The proposed method is evaluated on CIFAR-10, CIFAR-100, and TinyImageNet. The experimental results show that the collaborative training significantly improved the generalization of each model, which allowed for obtaining a small model that can mimic the performance of a large model and produce a more robust ensemble approach.</abstract><type>Journal Article</type><journal>Electronics</journal><volume>10</volume><journalNumber>14</journalNumber><paginationStart>1669</paginationStart><paginationEnd/><publisher>MDPI AG</publisher><placeOfPublication/><isbnPrint/><isbnElectronic/><issnPrint/><issnElectronic>2079-9292</issnElectronic><keywords>randomly wired neural networks; model distillation; ensemble model; deep learning</keywords><publishedDay>13</publishedDay><publishedMonth>7</publishedMonth><publishedYear>2021</publishedYear><publishedDate>2021-07-13</publishedDate><doi>10.3390/electronics10141669</doi><url/><notes/><college>COLLEGE NANME</college><department>Computer Science</department><CollegeCode>COLLEGE CODE</CollegeCode><DepartmentCode>SCS</DepartmentCode><institution>Swansea University</institution><apcterm>SU College/Department paid the OA fee</apcterm><funders>Ser&#x2C6; Cymru COFUND Fellowship</funders><lastEdited>2021-09-09T15:26:54.2304104</lastEdited><Created>2021-08-05T11:43:50.2623825</Created><path><level id="1">Faculty of Science and Engineering</level><level id="2">School of Mathematics and Computer Science - Computer Science</level></path><authors><author><firstname>Ehab</firstname><surname>Essa</surname><order>1</order></author><author><firstname>Xianghua</firstname><surname>Xie</surname><orcid>0000-0002-2701-8660</orcid><order>2</order></author></authors><documents><document><filename>57529__20553__afb8bb4521ee40a09e94001bdc8a9987.pdf</filename><originalFilename>electronics-10-01669.pdf</originalFilename><uploaded>2021-08-05T11:45:37.5024705</uploaded><type>Output</type><contentLength>809257</contentLength><contentType>application/pdf</contentType><version>Version of Record</version><cronfaStatus>true</cronfaStatus><documentNotes>Copyright: &#xA9; 2021 by the authors. This is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license</documentNotes><copyrightCorrect>true</copyrightCorrect><language>eng</language><licence>https://creativecommons.org/licenses/by/4.0/</licence></document></documents><OutputDurs/></rfc1807>
spelling 2021-09-09T15:26:54.2304104 v2 57529 2021-08-05 Deep Collaborative Learning for Randomly Wired Neural Networks b334d40963c7a2f435f06d2c26c74e11 0000-0002-2701-8660 Xianghua Xie Xianghua Xie true false 2021-08-05 SCS A deep collaborative learning approach is introduced in which a chain of randomly wired neural networks is trained simultaneously to improve the overall generalization and form a strong ensemble model. The proposed method takes advantage of functional-preserving transfer learning and knowledge distillation to produce an ensemble model. Knowledge distillation is an effective learning scheme for improving the performance of small neural networks by using the knowledge learned by teacher networks. Most of the previous methods learn from one or more teachers but not in a collaborative way. In this paper, we created a chain of randomly wired neural networks based on a random graph algorithm and collaboratively trained the models using functional-preserving transfer learning, so that the small network in the chain could learn from the largest one simultaneously. The training method applies knowledge distillation between randomly wired models, where each model is considered as a teacher to the next model in the chain. The decision of multiple chains of models can be combined to produce a robust ensemble model. The proposed method is evaluated on CIFAR-10, CIFAR-100, and TinyImageNet. The experimental results show that the collaborative training significantly improved the generalization of each model, which allowed for obtaining a small model that can mimic the performance of a large model and produce a more robust ensemble approach. Journal Article Electronics 10 14 1669 MDPI AG 2079-9292 randomly wired neural networks; model distillation; ensemble model; deep learning 13 7 2021 2021-07-13 10.3390/electronics10141669 COLLEGE NANME Computer Science COLLEGE CODE SCS Swansea University SU College/Department paid the OA fee Serˆ Cymru COFUND Fellowship 2021-09-09T15:26:54.2304104 2021-08-05T11:43:50.2623825 Faculty of Science and Engineering School of Mathematics and Computer Science - Computer Science Ehab Essa 1 Xianghua Xie 0000-0002-2701-8660 2 57529__20553__afb8bb4521ee40a09e94001bdc8a9987.pdf electronics-10-01669.pdf 2021-08-05T11:45:37.5024705 Output 809257 application/pdf Version of Record true Copyright: © 2021 by the authors. This is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license true eng https://creativecommons.org/licenses/by/4.0/
title Deep Collaborative Learning for Randomly Wired Neural Networks
spellingShingle Deep Collaborative Learning for Randomly Wired Neural Networks
Xianghua Xie
title_short Deep Collaborative Learning for Randomly Wired Neural Networks
title_full Deep Collaborative Learning for Randomly Wired Neural Networks
title_fullStr Deep Collaborative Learning for Randomly Wired Neural Networks
title_full_unstemmed Deep Collaborative Learning for Randomly Wired Neural Networks
title_sort Deep Collaborative Learning for Randomly Wired Neural Networks
author_id_str_mv b334d40963c7a2f435f06d2c26c74e11
author_id_fullname_str_mv b334d40963c7a2f435f06d2c26c74e11_***_Xianghua Xie
author Xianghua Xie
author2 Ehab Essa
Xianghua Xie
format Journal article
container_title Electronics
container_volume 10
container_issue 14
container_start_page 1669
publishDate 2021
institution Swansea University
issn 2079-9292
doi_str_mv 10.3390/electronics10141669
publisher MDPI AG
college_str Faculty of Science and Engineering
hierarchytype
hierarchy_top_id facultyofscienceandengineering
hierarchy_top_title Faculty of Science and Engineering
hierarchy_parent_id facultyofscienceandengineering
hierarchy_parent_title Faculty of Science and Engineering
department_str School of Mathematics and Computer Science - Computer Science{{{_:::_}}}Faculty of Science and Engineering{{{_:::_}}}School of Mathematics and Computer Science - Computer Science
document_store_str 1
active_str 0
description A deep collaborative learning approach is introduced in which a chain of randomly wired neural networks is trained simultaneously to improve the overall generalization and form a strong ensemble model. The proposed method takes advantage of functional-preserving transfer learning and knowledge distillation to produce an ensemble model. Knowledge distillation is an effective learning scheme for improving the performance of small neural networks by using the knowledge learned by teacher networks. Most of the previous methods learn from one or more teachers but not in a collaborative way. In this paper, we created a chain of randomly wired neural networks based on a random graph algorithm and collaboratively trained the models using functional-preserving transfer learning, so that the small network in the chain could learn from the largest one simultaneously. The training method applies knowledge distillation between randomly wired models, where each model is considered as a teacher to the next model in the chain. The decision of multiple chains of models can be combined to produce a robust ensemble model. The proposed method is evaluated on CIFAR-10, CIFAR-100, and TinyImageNet. The experimental results show that the collaborative training significantly improved the generalization of each model, which allowed for obtaining a small model that can mimic the performance of a large model and produce a more robust ensemble approach.
published_date 2021-07-13T04:13:20Z
_version_ 1763753906162106368
score 11.037603

Similar Items