Size-independent sample complexity of neural networks

Noah Golowich; Alexander Rakhlin; Ohad Shamir

doi:10.1093/imaiai/iaz007

Size-independent sample complexity of neural networks

Noah Golowich, Alexander Rakhlin, Ohad Shamir

Department of Computer Science and Applied Mathematics

Weizmann Institute of Science

Research output: Contribution to journal › Article › peer-review

Abstract

We study the sample complexity of learning neural networks by providing new bounds on their Rademacher complexity, assuming norm constraints on the parameter matrix of each layer. Compared to previous work, these complexity bounds have improved dependence on the network depth and, under some additional assumptions, are fully independent of the network size (both depth and width). These results are derived using some novel techniques, which may be of independent interest.

Original language	English
Pages (from-to)	473-504
Number of pages	32
Journal	Information and Inference: A Journal of the IMA
Volume	9
Issue number	2
Early online date	4 May 2020
DOIs	https://doi.org/10.1093/imaiai/iaz007
State	Published - Jun 2020

Access to Document

10.1093/imaiai/iaz007

https://arxiv.org/abs/1712.06541License: Other

Cite this

@article{1f66686f75da4d538efb9c157e6dfa92,

title = "Size-independent sample complexity of neural networks",

abstract = "We study the sample complexity of learning neural networks by providing new bounds on their Rademacher complexity, assuming norm constraints on the parameter matrix of each layer. Compared to previous work, these complexity bounds have improved dependence on the network depth and, under some additional assumptions, are fully independent of the network size (both depth and width). These results are derived using some novel techniques, which may be of independent interest.",

author = "Noah Golowich and Alexander Rakhlin and Ohad Shamir",

note = "We thank Ziwei Ji and Matus Telgarsky for pointing that an additional union bound is required in the proof of Theorem 4.3, resulting in an extra logarithmic factor. Noah Golowich is grateful for research funding from Harvard University, including the Herchel Smith and PRISE fellowships.",

year = "2020",

month = jun,

doi = "10.1093/imaiai/iaz007",

language = "الإنجليزيّة",

volume = "9",

pages = "473--504",

journal = "Information and Inference: A Journal of the IMA",

issn = "2049-8764",

publisher = "Oxford University Press",

number = "2",

}

TY - JOUR

T1 - Size-independent sample complexity of neural networks

AU - Golowich, Noah

AU - Rakhlin, Alexander

AU - Shamir, Ohad

N1 - We thank Ziwei Ji and Matus Telgarsky for pointing that an additional union bound is required in the proof of Theorem 4.3, resulting in an extra logarithmic factor. Noah Golowich is grateful for research funding from Harvard University, including the Herchel Smith and PRISE fellowships.

PY - 2020/6

Y1 - 2020/6

N2 - We study the sample complexity of learning neural networks by providing new bounds on their Rademacher complexity, assuming norm constraints on the parameter matrix of each layer. Compared to previous work, these complexity bounds have improved dependence on the network depth and, under some additional assumptions, are fully independent of the network size (both depth and width). These results are derived using some novel techniques, which may be of independent interest.

AB - We study the sample complexity of learning neural networks by providing new bounds on their Rademacher complexity, assuming norm constraints on the parameter matrix of each layer. Compared to previous work, these complexity bounds have improved dependence on the network depth and, under some additional assumptions, are fully independent of the network size (both depth and width). These results are derived using some novel techniques, which may be of independent interest.

U2 - 10.1093/imaiai/iaz007

DO - 10.1093/imaiai/iaz007

M3 - مقالة

SN - 2049-8764

VL - 9

SP - 473

EP - 504

JO - Information and Inference: A Journal of the IMA

JF - Information and Inference: A Journal of the IMA

IS - 2

ER -

Size-independent sample complexity of neural networks

Abstract

Access to Document

Fingerprint

Cite this