On the similarity between the laplace and neural tangent kernels

Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, Ronen Basri

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Recent theoretical work has shown that massively overparameterized neural networks are equivalent to kernel regressors that use Neural Tangent Kernels (NTKs). Experiments show that these kernel methods perform similarly to real neural networks. Here we show that NTK for fully connected networks with ReLU activation is closely related to the standard Laplace kernel. We show theoretically that for normalized data on the hypersphere both kernels have the same eigenfunctions and their eigenvalues decay polynomially at the same rate, implying that their Reproducing Kernel Hilbert Spaces (RKHS) include the same sets of functions. This means that both kernels give rise to classes of functions with the same smoothness properties. The two kernels differ for data off the hypersphere, but experiments indicate that when data is properly normalized these differences are not significant. Finally, we provide experiments on real data comparing NTK and the Laplace kernel, along with a larger class of γ-exponential kernels. We show that these perform almost identically. Our results suggest that much insight about neural networks can be obtained from analysis of the well-known Laplace kernel, which has a simple closed form.

Original languageEnglish
Title of host publicationNIPS'20
Subtitle of host publicationProceedings of the 34th International Conference on Neural Information Processing Systems
EditorsH. Larochelle
StatePublished - 6 Dec 2020
Event34th Conference on Neural Information Processing Systems, NeurIPS 2020 - Virtual, Online
Duration: 6 Dec 202012 Dec 2020

Publication series

NameAdvances in Neural Information Processing Systems
ISSN (Print)1049-5258


Conference34th Conference on Neural Information Processing Systems, NeurIPS 2020
CityVirtual, Online

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing


Dive into the research topics of 'On the similarity between the laplace and neural tangent kernels'. Together they form a unique fingerprint.

Cite this