Neural Spectrum Alignment: Empirical Study

Dmitry Kopitkov, Indelman Vadim

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Expressiveness and generalization of deep models was recently addressed via the connection between neural networks (NNs) and kernel learning, where first-order dynamics of NN during a gradient-descent (GD) optimization were related to gradient similarity kernel, also known as Neural Tangent Kernel (NTK)[9]. In the majority of works this kernel is considered to be time-invariant[9, 13]. In contrast, we empirically explore these properties along the optimization and show that in practice top eigenfunctions of NTK align toward the target function learned by NN which improves the overall optimization performance. Moreover, these top eigenfunctions serve as basis functions for NN output - a function represented by NN is spanned almost completely by them for the entire optimization process. Further, we study how learning rate decay affects the neural spectrum. We argue that the presented phenomena may lead to a more complete theoretical understanding behind NN learning.

Original languageEnglish
Title of host publicationArtificial Neural Networks and Machine Learning – ICANN 2020 - 29th International Conference on Artificial Neural Networks, Proceedings
EditorsIgor Farkaš, Paolo Masulli, Stefan Wermter
PublisherSpringer Science and Business Media Deutschland GmbH
Pages168-179
Number of pages12
ISBN (Print)9783030616151
DOIs
StatePublished - 2020
Event29th International Conference on Artificial Neural Networks, ICANN 2020 - Bratislava, Slovakia
Duration: 15 Sep 202018 Sep 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12397 LNCS

Conference

Conference29th International Conference on Artificial Neural Networks, ICANN 2020
Country/TerritorySlovakia
CityBratislava
Period15/09/2018/09/20

Keywords

  • Deep learning
  • Kernel learning
  • Neural tangent kernel

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'Neural Spectrum Alignment: Empirical Study'. Together they form a unique fingerprint.

Cite this