Implicit Bias of Gradient Descent on Linear Convolutional Networks

Suriya Gunasekar; Jason D. Lee; Daniel Soudry; Nathan Srebro

Implicit Bias of Gradient Descent on Linear Convolutional Networks

Suriya Gunasekar, Jason D. Lee, Daniel Soudry, Nathan Srebro

Technion - Israel Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the `_2/L bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the `₂ maximum margin solution.

Original language	English
Title of host publication	32nd Conference on Neural Information Processing Systems, NeurIPS 2018
Pages	9461-9471
Number of pages	11
State	Published - 2018
Event	32nd Conference on Neural Information Processing Systems, NeurIPS 2018 - Montreal, Canada Duration: 2 Dec 2018 → 8 Dec 2018

Conference

Conference	32nd Conference on Neural Information Processing Systems, NeurIPS 2018
Country/Territory	Canada
City	Montreal
Period	2/12/18 → 8/12/18

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Information Systems
Signal Processing

Access to Document

https://proceedings.neurips.cc/paper/2018/file/0e98aeeb54acf612b9eb4e48a269814c-Paper.pdf

Cite this

@inproceedings{a922f64d245a4b4892e049e08d3fd7ce,

title = "Implicit Bias of Gradient Descent on Linear Convolutional Networks",

abstract = "We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the `2/L bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the `2 maximum margin solution.",

author = "Suriya Gunasekar and Lee, {Jason D.} and Daniel Soudry and Nathan Srebro",

note = "Publisher Copyright: {\textcopyright} 2018 Curran Associates Inc.All rights reserved.; 32nd Conference on Neural Information Processing Systems, NeurIPS 2018 ; Conference date: 02-12-2018 Through 08-12-2018",

year = "2018",

language = "الإنجليزيّة",

pages = "9461--9471",

booktitle = "32nd Conference on Neural Information Processing Systems, NeurIPS 2018",

}

Gunasekar, S, Lee, JD, Soudry, D & Srebro, N 2018, Implicit Bias of Gradient Descent on Linear Convolutional Networks. in 32nd Conference on Neural Information Processing Systems, NeurIPS 2018. pp. 9461-9471, 32nd Conference on Neural Information Processing Systems, NeurIPS 2018, Montreal, Canada, 2/12/18. <https://proceedings.neurips.cc/paper/2018/file/0e98aeeb54acf612b9eb4e48a269814c-Paper.pdf>

TY - GEN

T1 - Implicit Bias of Gradient Descent on Linear Convolutional Networks

AU - Gunasekar, Suriya

AU - Lee, Jason D.

AU - Soudry, Daniel

AU - Srebro, Nathan

PY - 2018

Y1 - 2018

N2 - We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the `2/L bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the `2 maximum margin solution.

AB - We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the `2/L bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the `2 maximum margin solution.

UR - http://www.scopus.com/inward/record.url?scp=85057771475&partnerID=8YFLogxK

M3 - منشور من مؤتمر

SP - 9461

EP - 9471

BT - 32nd Conference on Neural Information Processing Systems, NeurIPS 2018

T2 - 32nd Conference on Neural Information Processing Systems, NeurIPS 2018

Y2 - 2 December 2018 through 8 December 2018

ER -

Implicit Bias of Gradient Descent on Linear Convolutional Networks

Abstract

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this