Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication

Eshkiki, Hassan; Mora, Benjamin; Xie, Xianghua

doi:10.1109/tnnls.2022.3176197

Journal article 594 views 53 downloads

Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication

Hassan Eshkiki

, Benjamin Mora

, Xianghua Xie

IEEE Transactions on Neural Networks and Learning Systems, Pages: 1 - 14

Swansea University Authors: Hassan Eshkiki , Benjamin Mora , Xianghua Xie

PDF | Accepted Manuscript

Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0).
Download (3.89MB)

Check full text

DOI (Published version): 10.1109/tnnls.2022.3176197

Abstract

This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the pr...

Full description

Published in:	IEEE Transactions on Neural Networks and Learning Systems
ISSN:	2162-237X 2162-2388
Published:	Institute of Electrical and Electronics Engineers (IEEE) 2022
Online Access:	Check full text
URI:	https://cronfa.swan.ac.uk/Record/cronfa60038
Tags:	Add Tag No Tags, Be the first to tag this record!

first_indexed	2022-05-16T16:49:00Z
last_indexed	2023-04-19T03:18:47Z
id	cronfa60038
recordtype	SURis
fullrecord	<?xml version="1.0" encoding="utf-8"?><rfc1807 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><bib-version>v2</bib-version><id>60038</id><entry>2022-05-16</entry><title>Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication</title><swanseaauthors><author><sid>c9972b26a83de11ffe211070f26fe16b</sid><ORCID>0000-0001-7795-453X</ORCID><firstname>Hassan</firstname><surname>Eshkiki</surname><name>Hassan Eshkiki</name><active>true</active><ethesisStudent>false</ethesisStudent></author><author><sid>557f93dfae240600e5bd4398bf203821</sid><ORCID>0000-0002-2945-3519</ORCID><firstname>Benjamin</firstname><surname>Mora</surname><name>Benjamin Mora</name><active>true</active><ethesisStudent>false</ethesisStudent></author><author><sid>b334d40963c7a2f435f06d2c26c74e11</sid><ORCID>0000-0002-2701-8660</ORCID><firstname>Xianghua</firstname><surname>Xie</surname><name>Xianghua Xie</name><active>true</active><ethesisStudent>false</ethesisStudent></author></swanseaauthors><date>2022-05-16</date><deptcode>SCS</deptcode><abstract>This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this paper demonstrates a first application to machine learning inference by showing that weights of fully-connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators.</abstract><type>Journal Article</type><journal>IEEE Transactions on Neural Networks and Learning Systems</journal><volume>0</volume><journalNumber/><paginationStart>1</paginationStart><paginationEnd>14</paginationEnd><publisher>Institute of Electrical and Electronics Engineers (IEEE)</publisher><placeOfPublication/><isbnPrint/><isbnElectronic/><issnPrint>2162-237X</issnPrint><issnElectronic>2162-2388</issnElectronic><keywords>Complexity theory, Approximation algorithms, Neural networks, Monte Carlo methods, Heart, Transforms, Inference algorithms</keywords><publishedDay>1</publishedDay><publishedMonth>1</publishedMonth><publishedYear>2022</publishedYear><publishedDate>2022-01-01</publishedDate><doi>10.1109/tnnls.2022.3176197</doi><url>http://dx.doi.org/10.1109/tnnls.2022.3176197</url><notes>In Press.</notes><college>COLLEGE NANME</college><department>Computer Science</department><CollegeCode>COLLEGE CODE</CollegeCode><DepartmentCode>SCS</DepartmentCode><institution>Swansea University</institution><apcterm>SU College/Department paid the OA fee</apcterm><funders>This work was supported in part by the UK Government through the Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/R51312X/1 and in part by the Natural Environment Research Council (NERC) under Grant NE/W502911/1.</funders><projectreference/><lastEdited>2023-11-08T15:53:00.3476012</lastEdited><Created>2022-05-16T17:37:16.9524707</Created><path><level id="1">Faculty of Science and Engineering</level><level id="2">School of Mathematics and Computer Science - Computer Science</level></path><authors><author><firstname>Hassan</firstname><surname>Eshkiki</surname><orcid>0000-0001-7795-453X</orcid><order>1</order></author><author><firstname>Benjamin</firstname><surname>Mora</surname><orcid>0000-0002-2945-3519</orcid><order>2</order></author><author><firstname>Xianghua</firstname><surname>Xie</surname><orcid>0000-0002-2701-8660</orcid><order>3</order></author></authors><documents><document><filename>60038__24511__86ec71af591f437aa3f0505e3a023b37.pdf</filename><originalFilename>60038.pdf</originalFilename><uploaded>2022-07-08T14:34:15.9357207</uploaded><type>Output</type><contentLength>4081808</contentLength><contentType>application/pdf</contentType><version>Accepted Manuscript</version><cronfaStatus>true</cronfaStatus><documentNotes>Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0).</documentNotes><copyrightCorrect>true</copyrightCorrect><language>eng</language><licence>https://creativecommons.org/licenses/by/4.0/</licence></document></documents><OutputDurs/></rfc1807>
spelling	v2 60038 2022-05-16 Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication c9972b26a83de11ffe211070f26fe16b 0000-0001-7795-453X Hassan Eshkiki Hassan Eshkiki true false 557f93dfae240600e5bd4398bf203821 0000-0002-2945-3519 Benjamin Mora Benjamin Mora true false b334d40963c7a2f435f06d2c26c74e11 0000-0002-2701-8660 Xianghua Xie Xianghua Xie true false 2022-05-16 SCS This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this paper demonstrates a first application to machine learning inference by showing that weights of fully-connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators. Journal Article IEEE Transactions on Neural Networks and Learning Systems 0 1 14 Institute of Electrical and Electronics Engineers (IEEE) 2162-237X 2162-2388 Complexity theory, Approximation algorithms, Neural networks, Monte Carlo methods, Heart, Transforms, Inference algorithms 1 1 2022 2022-01-01 10.1109/tnnls.2022.3176197 http://dx.doi.org/10.1109/tnnls.2022.3176197 In Press. COLLEGE NANME Computer Science COLLEGE CODE SCS Swansea University SU College/Department paid the OA fee This work was supported in part by the UK Government through the Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/R51312X/1 and in part by the Natural Environment Research Council (NERC) under Grant NE/W502911/1. 2023-11-08T15:53:00.3476012 2022-05-16T17:37:16.9524707 Faculty of Science and Engineering School of Mathematics and Computer Science - Computer Science Hassan Eshkiki 0000-0001-7795-453X 1 Benjamin Mora 0000-0002-2945-3519 2 Xianghua Xie 0000-0002-2701-8660 3 60038__24511__86ec71af591f437aa3f0505e3a023b37.pdf 60038.pdf 2022-07-08T14:34:15.9357207 Output 4081808 application/pdf Accepted Manuscript true Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0). true eng https://creativecommons.org/licenses/by/4.0/
title	Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
spellingShingle	Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication Hassan Eshkiki Benjamin Mora Xianghua Xie
title_short	Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
title_full	Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
title_fullStr	Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
title_full_unstemmed	Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
title_sort	Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
author_id_str_mv	c9972b26a83de11ffe211070f26fe16b 557f93dfae240600e5bd4398bf203821 b334d40963c7a2f435f06d2c26c74e11
author_id_fullname_str_mv	c9972b26a83de11ffe211070f26fe16b_*_Hassan Eshkiki 557f93dfae240600e5bd4398bf203821__Benjamin Mora b334d40963c7a2f435f06d2c26c74e11_**_Xianghua Xie
author	Hassan Eshkiki Benjamin Mora Xianghua Xie
author2	Hassan Eshkiki Benjamin Mora Xianghua Xie
format	Journal article
container_title	IEEE Transactions on Neural Networks and Learning Systems
container_volume	0
container_start_page	1
publishDate	2022
institution	Swansea University
issn	2162-237X 2162-2388
doi_str_mv	10.1109/tnnls.2022.3176197
publisher	Institute of Electrical and Electronics Engineers (IEEE)
college_str	Faculty of Science and Engineering
hierarchytype
hierarchy_top_id	facultyofscienceandengineering
hierarchy_top_title	Faculty of Science and Engineering
hierarchy_parent_id	facultyofscienceandengineering
hierarchy_parent_title	Faculty of Science and Engineering
department_str	School of Mathematics and Computer Science - Computer Science{{{_:::_}}}Faculty of Science and Engineering{{{_:::_}}}School of Mathematics and Computer Science - Computer Science
url	http://dx.doi.org/10.1109/tnnls.2022.3176197
document_store_str	1
active_str	0
description	This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this paper demonstrates a first application to machine learning inference by showing that weights of fully-connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators.
published_date	2022-01-01T15:53:03Z
_version_	1782011693848592384
score	11.013148

Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication

Similar Items