No Cover Image

Journal article 769 views 23 downloads

Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication

Hassan Eshkiki Orcid Logo, Benjamin Mora Orcid Logo, Xianghua Xie Orcid Logo

IEEE Transactions on Neural Networks and Learning Systems, Volume: 35, Issue: 1, Pages: 634 - 647

Swansea University Authors: Hassan Eshkiki Orcid Logo, Benjamin Mora Orcid Logo, Xianghua Xie Orcid Logo

  • 60038.VoR.pdf

    PDF | Version of Record

    © 2022 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License.

    Download (3.96MB)

Check full text

DOI (Published version): 10.1109/tnnls.2022.3176197

Abstract

This article proposes the Mediterranean matrix multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m,n, and p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to t...

Full description

Published in: IEEE Transactions on Neural Networks and Learning Systems
ISSN: 2162-237X 2162-2388
Published: Institute of Electrical and Electronics Engineers (IEEE) 2024
Online Access: Check full text

URI: https://cronfa.swan.ac.uk/Record/cronfa60038
first_indexed 2022-05-16T16:49:00Z
last_indexed 2024-11-14T12:16:35Z
id cronfa60038
recordtype SURis
fullrecord <?xml version="1.0"?><rfc1807><datestamp>2024-09-17T16:36:10.1309391</datestamp><bib-version>v2</bib-version><id>60038</id><entry>2022-05-16</entry><title>Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication</title><swanseaauthors><author><sid>c9972b26a83de11ffe211070f26fe16b</sid><ORCID>0000-0001-7795-453X</ORCID><firstname>Hassan</firstname><surname>Eshkiki</surname><name>Hassan Eshkiki</name><active>true</active><ethesisStudent>false</ethesisStudent></author><author><sid>557f93dfae240600e5bd4398bf203821</sid><ORCID>0000-0002-2945-3519</ORCID><firstname>Benjamin</firstname><surname>Mora</surname><name>Benjamin Mora</name><active>true</active><ethesisStudent>false</ethesisStudent></author><author><sid>b334d40963c7a2f435f06d2c26c74e11</sid><ORCID>0000-0002-2701-8660</ORCID><firstname>Xianghua</firstname><surname>Xie</surname><name>Xianghua Xie</name><active>true</active><ethesisStudent>false</ethesisStudent></author></swanseaauthors><date>2022-05-16</date><deptcode>MACS</deptcode><abstract>This article proposes the Mediterranean matrix multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m,n, and p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this article demonstrates a first application to machine learning inference by showing that weights of fully connected layers can be compressed between 30&#xD7; and 100&#xD7; with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators.</abstract><type>Journal Article</type><journal>IEEE Transactions on Neural Networks and Learning Systems</journal><volume>35</volume><journalNumber>1</journalNumber><paginationStart>634</paginationStart><paginationEnd>647</paginationEnd><publisher>Institute of Electrical and Electronics Engineers (IEEE)</publisher><placeOfPublication/><isbnPrint/><isbnElectronic/><issnPrint>2162-237X</issnPrint><issnElectronic>2162-2388</issnElectronic><keywords>Complexity theory, Approximation algorithms, Neural networks, Monte Carlo methods, Heart, Transforms, Inference algorithms</keywords><publishedDay>5</publishedDay><publishedMonth>1</publishedMonth><publishedYear>2024</publishedYear><publishedDate>2024-01-05</publishedDate><doi>10.1109/tnnls.2022.3176197</doi><url/><notes/><college>COLLEGE NANME</college><department>Mathematics and Computer Science School</department><CollegeCode>COLLEGE CODE</CollegeCode><DepartmentCode>MACS</DepartmentCode><institution>Swansea University</institution><apcterm>SU College/Department paid the OA fee</apcterm><funders>This work was supported in part by the UK Government through the Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/R51312X/1 and in part by the Natural Environment Research Council (NERC) under Grant NE/W502911/1.</funders><projectreference/><lastEdited>2024-09-17T16:36:10.1309391</lastEdited><Created>2022-05-16T17:37:16.9524707</Created><path><level id="1">Faculty of Science and Engineering</level><level id="2">School of Mathematics and Computer Science - Computer Science</level></path><authors><author><firstname>Hassan</firstname><surname>Eshkiki</surname><orcid>0000-0001-7795-453X</orcid><order>1</order></author><author><firstname>Benjamin</firstname><surname>Mora</surname><orcid>0000-0002-2945-3519</orcid><order>2</order></author><author><firstname>Xianghua</firstname><surname>Xie</surname><orcid>0000-0002-2701-8660</orcid><order>3</order></author></authors><documents><document><filename>60038__31356__0b683a7201f540b89f3f8ca0f373040d.pdf</filename><originalFilename>60038.VoR.pdf</originalFilename><uploaded>2024-09-17T16:33:17.0281858</uploaded><type>Output</type><contentLength>4153630</contentLength><contentType>application/pdf</contentType><version>Version of Record</version><cronfaStatus>true</cronfaStatus><documentNotes>&#xA9; 2022 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License.</documentNotes><copyrightCorrect>true</copyrightCorrect><language>eng</language><licence>https://creativecommons.org/licenses/by/4.0/</licence></document></documents><OutputDurs/></rfc1807>
spelling 2024-09-17T16:36:10.1309391 v2 60038 2022-05-16 Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication c9972b26a83de11ffe211070f26fe16b 0000-0001-7795-453X Hassan Eshkiki Hassan Eshkiki true false 557f93dfae240600e5bd4398bf203821 0000-0002-2945-3519 Benjamin Mora Benjamin Mora true false b334d40963c7a2f435f06d2c26c74e11 0000-0002-2701-8660 Xianghua Xie Xianghua Xie true false 2022-05-16 MACS This article proposes the Mediterranean matrix multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m,n, and p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this article demonstrates a first application to machine learning inference by showing that weights of fully connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators. Journal Article IEEE Transactions on Neural Networks and Learning Systems 35 1 634 647 Institute of Electrical and Electronics Engineers (IEEE) 2162-237X 2162-2388 Complexity theory, Approximation algorithms, Neural networks, Monte Carlo methods, Heart, Transforms, Inference algorithms 5 1 2024 2024-01-05 10.1109/tnnls.2022.3176197 COLLEGE NANME Mathematics and Computer Science School COLLEGE CODE MACS Swansea University SU College/Department paid the OA fee This work was supported in part by the UK Government through the Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/R51312X/1 and in part by the Natural Environment Research Council (NERC) under Grant NE/W502911/1. 2024-09-17T16:36:10.1309391 2022-05-16T17:37:16.9524707 Faculty of Science and Engineering School of Mathematics and Computer Science - Computer Science Hassan Eshkiki 0000-0001-7795-453X 1 Benjamin Mora 0000-0002-2945-3519 2 Xianghua Xie 0000-0002-2701-8660 3 60038__31356__0b683a7201f540b89f3f8ca0f373040d.pdf 60038.VoR.pdf 2024-09-17T16:33:17.0281858 Output 4153630 application/pdf Version of Record true © 2022 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. true eng https://creativecommons.org/licenses/by/4.0/
title Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
spellingShingle Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
Hassan Eshkiki
Benjamin Mora
Xianghua Xie
title_short Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
title_full Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
title_fullStr Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
title_full_unstemmed Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
title_sort Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
author_id_str_mv c9972b26a83de11ffe211070f26fe16b
557f93dfae240600e5bd4398bf203821
b334d40963c7a2f435f06d2c26c74e11
author_id_fullname_str_mv c9972b26a83de11ffe211070f26fe16b_***_Hassan Eshkiki
557f93dfae240600e5bd4398bf203821_***_Benjamin Mora
b334d40963c7a2f435f06d2c26c74e11_***_Xianghua Xie
author Hassan Eshkiki
Benjamin Mora
Xianghua Xie
author2 Hassan Eshkiki
Benjamin Mora
Xianghua Xie
format Journal article
container_title IEEE Transactions on Neural Networks and Learning Systems
container_volume 35
container_issue 1
container_start_page 634
publishDate 2024
institution Swansea University
issn 2162-237X
2162-2388
doi_str_mv 10.1109/tnnls.2022.3176197
publisher Institute of Electrical and Electronics Engineers (IEEE)
college_str Faculty of Science and Engineering
hierarchytype
hierarchy_top_id facultyofscienceandengineering
hierarchy_top_title Faculty of Science and Engineering
hierarchy_parent_id facultyofscienceandengineering
hierarchy_parent_title Faculty of Science and Engineering
department_str School of Mathematics and Computer Science - Computer Science{{{_:::_}}}Faculty of Science and Engineering{{{_:::_}}}School of Mathematics and Computer Science - Computer Science
document_store_str 1
active_str 0
description This article proposes the Mediterranean matrix multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m,n, and p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this article demonstrates a first application to machine learning inference by showing that weights of fully connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators.
published_date 2024-01-05T14:19:49Z
_version_ 1821415507663781888
score 11.048194

Similar Items