No Cover Image

Conference Paper/Proceeding/Abstract 52 views 12 downloads

Random Matrix Theory for Stochastic Gradient Descent

Chanju Park, Matteo Favoni, Biagio Lucini, Gert Aarts Orcid Logo

Proceedings of The 41st International Symposium on Lattice Field Theory — PoS(LATTICE2024), Volume: 466, Start page: 031

Swansea University Authors: Matteo Favoni, Gert Aarts Orcid Logo

  • 70970.VoR.pdf

    PDF | Version of Record

    ©Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

    Download (954.7KB)

Check full text

DOI (Published version): 10.22323/1.466.0031

Abstract

Investigating the dynamics of learning in machine learning algorithms is of paramount importance for understanding how and why an approach may be successful. The tools of physics and statistics provide a robust setting for such investigations. Here, we apply concepts from random matrix theory to des...

Full description

Published in: Proceedings of The 41st International Symposium on Lattice Field Theory — PoS(LATTICE2024)
ISSN: 1824-8039
Published: Trieste, Italy Sissa Medialab 2025
Online Access: Check full text

URI: https://cronfa.swan.ac.uk/Record/cronfa70970
Abstract: Investigating the dynamics of learning in machine learning algorithms is of paramount importance for understanding how and why an approach may be successful. The tools of physics and statistics provide a robust setting for such investigations. Here, we apply concepts from random matrix theory to describe stochastic weight matrix dynamics, using the framework of Dyson Brownian motion. We derive the linear scaling rule between the learning rate (step size) and the batch size, and identify universal and non-universal aspects of weight matrix dynamics. We test our findings in the (near-)solvable case of the Gaussian Restricted Boltzmann Machine and in a linear one-hidden-layer neural network.
College: Faculty of Science and Engineering
Funders: GA, MF and BL are supported by STFC Consolidated Grant ST/X000648/1. BL is further supported by the UKRI EPSRC ExCALIBUR ExaTEPP project EP/X017168/1. CP is supported by the UKRI AIMLAC CDT EP/S023992/1.
Start Page: 031

Similar Items