No Cover Image

E-Thesis 711 views 44 downloads

Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems / Jay Morgan

Swansea University Author: Jay Morgan

  • Morgan_Jay_PhD_Thesis_Final_Embargoed_Cronfa.pdf

    PDF | E-Thesis – open access

    Copyright: The author, Jay Morgan, 2022.

    Download (2.3MB)

DOI (Published version): 10.23889/SUthesis.59258

Abstract

Machine Learning (ML) has been a transformative technology in society by automating otherwise difficult tasks such as image recognition and natural language understand-ing. The performance of Deep Learning (DL), in particular, has improved to the point where it can be applied to automotive vehicles...

Full description

Published: Swansea 2022
Institution: Swansea University
Degree level: Doctoral
Degree name: Ph.D
Supervisor: Seisenberger, Monika ; Williams, Jane ; Paiement, Adeline
URI: https://cronfa.swan.ac.uk/Record/cronfa59258
Tags: Add Tag
No Tags, Be the first to tag this record!
first_indexed 2022-01-27T18:02:30Z
last_indexed 2022-01-29T03:42:56Z
id cronfa59258
recordtype RisThesis
fullrecord <?xml version="1.0" encoding="utf-8"?><rfc1807 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><bib-version>v2</bib-version><id>59258</id><entry>2022-01-27</entry><title>Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems</title><swanseaauthors><author><sid>df9a27bcf77b4769c2ebbb702b587491</sid><ORCID>0000-0003-3719-362X</ORCID><firstname>Jay</firstname><surname>Morgan</surname><name>Jay Morgan</name><active>true</active><ethesisStudent>false</ethesisStudent></author></swanseaauthors><date>2022-01-27</date><deptcode>MACS</deptcode><abstract>Machine Learning (ML) has been a transformative technology in society by automating otherwise difficult tasks such as image recognition and natural language understand-ing. The performance of Deep Learning (DL), in particular, has improved to the point where it can be applied to automotive vehicles – a situation in which trust is placed on the ML systems to operate correctly and safely. Yet, while fundamental ML algorithms can be formally verified for safety without much trouble, the same may not be said for DL. A key problem preventing the trustworthiness of DL is the existence of adver-sarial examples, where small changes in input result in catastrophic misclassifications, thereby undermining their use in safety-critical systems.Using pre-existing knowledge from domain experts has been shown to successfully in-crease not only the performance but critically the resilience of DL models to adversarial examples. The current thesis developed four different strategies of integrating prior expert knowledge into DL models: feature specialisation, specialised information pro-cessing, stimulation of attention mechanisms, and augmentation of training data. Prior knowledge from three scientific domains was used (Quantum Chemistry, Corpus Lin-guistics and Astrophysics) as case studies to provide a comprehensive framework for evaluation of the strategies performance given different types of data (i.e., text-based, image-based, and graph-based) and model architectures (e.g. recurrent, graph, and convolutional). For the Quantum Chemistry and Corpus Linguistics case studies, two novel datasets are introduced to facilitate the training of prior knowledge informed DL models. Each of the four proposed strategies were tested independently on the case studies to understand their isolated contribution, as well as combined with other strategies to evaluate their interaction.The results show that, combined, the four prior knowledge integration strategies (a) are an effective method of increasing model performance; (b) result in fewer misclas-sifications as a result of misleading features; (c) lead to increased model robustness to adversarial examples; (d) create informative representations by visualising learnt representations of prior knowledge; (e) lessen the number of training samples needed to achieve adequate model performance; and (f) lead to better generalisation to dif-ferent problem tasks other than those the model was trained for. The findings show the prior knowledge integration strategies used here improve the performance of ML while being more resilient to adversarial examples. This can lead to more trustworthy ML systems in practice.</abstract><type>E-Thesis</type><journal/><volume/><journalNumber/><paginationStart/><paginationEnd/><publisher/><placeOfPublication>Swansea</placeOfPublication><isbnPrint/><isbnElectronic/><issnPrint/><issnElectronic/><keywords>Trustable Machine Learning, Machine Learning, Prior knowledge, Feature specialisation, Attention, Adversarial Examples, Data augmentation, Specialised information processing</keywords><publishedDay>21</publishedDay><publishedMonth>1</publishedMonth><publishedYear>2022</publishedYear><publishedDate>2022-01-21</publishedDate><doi>10.23889/SUthesis.59258</doi><url/><notes>ORCiD identifier: https://orcid.org/0000-0003-3719-362X</notes><college>COLLEGE NANME</college><department>Mathematics and Computer Science School</department><CollegeCode>COLLEGE CODE</CollegeCode><DepartmentCode>MACS</DepartmentCode><institution>Swansea University</institution><supervisor>Seisenberger, Monika ; Williams, Jane ; Paiement, Adeline</supervisor><degreelevel>Doctoral</degreelevel><degreename>Ph.D</degreename><degreesponsorsfunders>College of Science/Hilary Clinton School of Law PhD Scholarship</degreesponsorsfunders><apcterm/><funders/><projectreference/><lastEdited>2024-07-11T15:25:44.5994866</lastEdited><Created>2022-01-27T17:56:23.0953648</Created><path><level id="1">Faculty of Science and Engineering</level><level id="2">School of Mathematics and Computer Science - Computer Science</level></path><authors><author><firstname>Jay</firstname><surname>Morgan</surname><orcid>0000-0003-3719-362X</orcid><order>1</order></author></authors><documents><document><filename>59258__22238__66f03ae563d64ef89dd92703140c4a10.pdf</filename><originalFilename>Morgan_Jay_PhD_Thesis_Final_Embargoed_Cronfa.pdf</originalFilename><uploaded>2022-01-27T18:17:28.4514905</uploaded><type>Output</type><contentLength>2412366</contentLength><contentType>application/pdf</contentType><version>E-Thesis – open access</version><cronfaStatus>true</cronfaStatus><embargoDate>2024-01-21T00:00:00.0000000</embargoDate><documentNotes>Copyright: The author, Jay Morgan, 2022.</documentNotes><copyrightCorrect>true</copyrightCorrect><language>eng</language></document></documents><OutputDurs/></rfc1807>
spelling v2 59258 2022-01-27 Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems df9a27bcf77b4769c2ebbb702b587491 0000-0003-3719-362X Jay Morgan Jay Morgan true false 2022-01-27 MACS Machine Learning (ML) has been a transformative technology in society by automating otherwise difficult tasks such as image recognition and natural language understand-ing. The performance of Deep Learning (DL), in particular, has improved to the point where it can be applied to automotive vehicles – a situation in which trust is placed on the ML systems to operate correctly and safely. Yet, while fundamental ML algorithms can be formally verified for safety without much trouble, the same may not be said for DL. A key problem preventing the trustworthiness of DL is the existence of adver-sarial examples, where small changes in input result in catastrophic misclassifications, thereby undermining their use in safety-critical systems.Using pre-existing knowledge from domain experts has been shown to successfully in-crease not only the performance but critically the resilience of DL models to adversarial examples. The current thesis developed four different strategies of integrating prior expert knowledge into DL models: feature specialisation, specialised information pro-cessing, stimulation of attention mechanisms, and augmentation of training data. Prior knowledge from three scientific domains was used (Quantum Chemistry, Corpus Lin-guistics and Astrophysics) as case studies to provide a comprehensive framework for evaluation of the strategies performance given different types of data (i.e., text-based, image-based, and graph-based) and model architectures (e.g. recurrent, graph, and convolutional). For the Quantum Chemistry and Corpus Linguistics case studies, two novel datasets are introduced to facilitate the training of prior knowledge informed DL models. Each of the four proposed strategies were tested independently on the case studies to understand their isolated contribution, as well as combined with other strategies to evaluate their interaction.The results show that, combined, the four prior knowledge integration strategies (a) are an effective method of increasing model performance; (b) result in fewer misclas-sifications as a result of misleading features; (c) lead to increased model robustness to adversarial examples; (d) create informative representations by visualising learnt representations of prior knowledge; (e) lessen the number of training samples needed to achieve adequate model performance; and (f) lead to better generalisation to dif-ferent problem tasks other than those the model was trained for. The findings show the prior knowledge integration strategies used here improve the performance of ML while being more resilient to adversarial examples. This can lead to more trustworthy ML systems in practice. E-Thesis Swansea Trustable Machine Learning, Machine Learning, Prior knowledge, Feature specialisation, Attention, Adversarial Examples, Data augmentation, Specialised information processing 21 1 2022 2022-01-21 10.23889/SUthesis.59258 ORCiD identifier: https://orcid.org/0000-0003-3719-362X COLLEGE NANME Mathematics and Computer Science School COLLEGE CODE MACS Swansea University Seisenberger, Monika ; Williams, Jane ; Paiement, Adeline Doctoral Ph.D College of Science/Hilary Clinton School of Law PhD Scholarship 2024-07-11T15:25:44.5994866 2022-01-27T17:56:23.0953648 Faculty of Science and Engineering School of Mathematics and Computer Science - Computer Science Jay Morgan 0000-0003-3719-362X 1 59258__22238__66f03ae563d64ef89dd92703140c4a10.pdf Morgan_Jay_PhD_Thesis_Final_Embargoed_Cronfa.pdf 2022-01-27T18:17:28.4514905 Output 2412366 application/pdf E-Thesis – open access true 2024-01-21T00:00:00.0000000 Copyright: The author, Jay Morgan, 2022. true eng
title Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems
spellingShingle Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems
Jay Morgan
title_short Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems
title_full Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems
title_fullStr Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems
title_full_unstemmed Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems
title_sort Strategies to use Prior Knowledge to Improve the Performance of Deep Learning (Subtitle) An Approach Towards Trustable Machine Learning Systems
author_id_str_mv df9a27bcf77b4769c2ebbb702b587491
author_id_fullname_str_mv df9a27bcf77b4769c2ebbb702b587491_***_Jay Morgan
author Jay Morgan
author2 Jay Morgan
format E-Thesis
publishDate 2022
institution Swansea University
doi_str_mv 10.23889/SUthesis.59258
college_str Faculty of Science and Engineering
hierarchytype
hierarchy_top_id facultyofscienceandengineering
hierarchy_top_title Faculty of Science and Engineering
hierarchy_parent_id facultyofscienceandengineering
hierarchy_parent_title Faculty of Science and Engineering
department_str School of Mathematics and Computer Science - Computer Science{{{_:::_}}}Faculty of Science and Engineering{{{_:::_}}}School of Mathematics and Computer Science - Computer Science
document_store_str 1
active_str 0
description Machine Learning (ML) has been a transformative technology in society by automating otherwise difficult tasks such as image recognition and natural language understand-ing. The performance of Deep Learning (DL), in particular, has improved to the point where it can be applied to automotive vehicles – a situation in which trust is placed on the ML systems to operate correctly and safely. Yet, while fundamental ML algorithms can be formally verified for safety without much trouble, the same may not be said for DL. A key problem preventing the trustworthiness of DL is the existence of adver-sarial examples, where small changes in input result in catastrophic misclassifications, thereby undermining their use in safety-critical systems.Using pre-existing knowledge from domain experts has been shown to successfully in-crease not only the performance but critically the resilience of DL models to adversarial examples. The current thesis developed four different strategies of integrating prior expert knowledge into DL models: feature specialisation, specialised information pro-cessing, stimulation of attention mechanisms, and augmentation of training data. Prior knowledge from three scientific domains was used (Quantum Chemistry, Corpus Lin-guistics and Astrophysics) as case studies to provide a comprehensive framework for evaluation of the strategies performance given different types of data (i.e., text-based, image-based, and graph-based) and model architectures (e.g. recurrent, graph, and convolutional). For the Quantum Chemistry and Corpus Linguistics case studies, two novel datasets are introduced to facilitate the training of prior knowledge informed DL models. Each of the four proposed strategies were tested independently on the case studies to understand their isolated contribution, as well as combined with other strategies to evaluate their interaction.The results show that, combined, the four prior knowledge integration strategies (a) are an effective method of increasing model performance; (b) result in fewer misclas-sifications as a result of misleading features; (c) lead to increased model robustness to adversarial examples; (d) create informative representations by visualising learnt representations of prior knowledge; (e) lessen the number of training samples needed to achieve adequate model performance; and (f) lead to better generalisation to dif-ferent problem tasks other than those the model was trained for. The findings show the prior knowledge integration strategies used here improve the performance of ML while being more resilient to adversarial examples. This can lead to more trustworthy ML systems in practice.
published_date 2022-01-21T15:25:43Z
_version_ 1804293052707110912
score 11.037319

Similar Items