On Margin Maximization in Linear and ReLU Networks

Gal Vardi; Ohad Shamir; Nathan Srebro

On Margin Maximization in Linear and ReLU Networks

Gal Vardi, Ohad Shamir, Nathan Srebro

Department of Computer Science and Applied Mathematics

Weizmann Institute of Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The implicit bias of neural networks has been extensively studied in recent years. Lyu and Li [2019] showed that in homogeneous networks trained with the exponential or the logistic loss, gradient flow converges to a KKT point of the max margin problem in parameter space. However, that leaves open the question of whether this point will generally be an actual optimum of the max margin problem. In this paper, we study this question in detail, for several neural network architectures involving linear and ReLU activations. Perhaps surprisingly, we show that in many cases, the KKT point is not even a local optimum of the max margin problem. On the flip side, we identify multiple settings where a local or global optimum can be guaranteed.

Original language	English
Title of host publication	NIPS'22
Subtitle of host publication	Proceedings of the 36th International Conference on Neural Information Processing Systems
Editors	S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh
Pages	37024-37036
Number of pages	13
ISBN (Electronic)	9781713871088
State	Published - 3 Apr 2022
Event	36th Conference on Neural Information Processing Systems, NeurIPS 2022 - New Orleans, United States Duration: 28 Nov 2022 → 9 Dec 2022

Publication series

Name	Advances in Neural Information Processing Systems
Volume	35

Conference

Conference	36th Conference on Neural Information Processing Systems, NeurIPS 2022
Country/Territory	United States
City	New Orleans
Period	28/11/22 → 9/12/22

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Information Systems
Signal Processing

Access to Document

https://proceedings.neurips.cc/paper_files/paper/2022/file/f062da1973ac9ac61fc6d44dd7fa309f-Paper-Conference.pdf

Cite this

Vardi, G., Shamir, O., & Srebro, N. (2022). On Margin Maximization in Linear and ReLU Networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), NIPS'22: Proceedings of the 36th International Conference on Neural Information Processing Systems (pp. 37024-37036). Article 2683 (Advances in Neural Information Processing Systems; Vol. 35). https://proceedings.neurips.cc/paper_files/paper/2022/file/f062da1973ac9ac61fc6d44dd7fa309f-Paper-Conference.pdf

@inproceedings{d112e88bb1ef408cb7a24266340f64b7,

title = "On Margin Maximization in Linear and ReLU Networks",

abstract = "The implicit bias of neural networks has been extensively studied in recent years. Lyu and Li [2019] showed that in homogeneous networks trained with the exponential or the logistic loss, gradient flow converges to a KKT point of the max margin problem in parameter space. However, that leaves open the question of whether this point will generally be an actual optimum of the max margin problem. In this paper, we study this question in detail, for several neural network architectures involving linear and ReLU activations. Perhaps surprisingly, we show that in many cases, the KKT point is not even a local optimum of the max margin problem. On the flip side, we identify multiple settings where a local or global optimum can be guaranteed.",

author = "Gal Vardi and Ohad Shamir and Nathan Srebro",

note = "Publisher Copyright: {\textcopyright} 2022 Neural information processing systems foundation. All rights reserved.; 36th Conference on Neural Information Processing Systems, NeurIPS 2022 ; Conference date: 28-11-2022 Through 09-12-2022",

year = "2022",

month = apr,

day = "3",

language = "الإنجليزيّة",

series = "Advances in Neural Information Processing Systems",

pages = "37024--37036",

editor = "S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh",

booktitle = "NIPS'22",

}

Vardi, G , Shamir, O & Srebro, N 2022, On Margin Maximization in Linear and ReLU Networks. in S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho & A Oh (eds), NIPS'22: Proceedings of the 36th International Conference on Neural Information Processing Systems., 2683, Advances in Neural Information Processing Systems, vol. 35, pp. 37024-37036, 36th Conference on Neural Information Processing Systems, NeurIPS 2022, New Orleans, United States, 28/11/22. <https://proceedings.neurips.cc/paper_files/paper/2022/file/f062da1973ac9ac61fc6d44dd7fa309f-Paper-Conference.pdf>

On Margin Maximization in Linear and ReLU Networks. / Vardi, Gal ; Shamir, Ohad; Srebro, Nathan.
NIPS'22: Proceedings of the 36th International Conference on Neural Information Processing Systems. ed. / S. Koyejo; S. Mohamed; A. Agarwal; D. Belgrave; K. Cho; A. Oh. 2022. p. 37024-37036 2683 (Advances in Neural Information Processing Systems; Vol. 35).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - On Margin Maximization in Linear and ReLU Networks

AU - Vardi, Gal

AU - Shamir, Ohad

AU - Srebro, Nathan

PY - 2022/4/3

Y1 - 2022/4/3

N2 - The implicit bias of neural networks has been extensively studied in recent years. Lyu and Li [2019] showed that in homogeneous networks trained with the exponential or the logistic loss, gradient flow converges to a KKT point of the max margin problem in parameter space. However, that leaves open the question of whether this point will generally be an actual optimum of the max margin problem. In this paper, we study this question in detail, for several neural network architectures involving linear and ReLU activations. Perhaps surprisingly, we show that in many cases, the KKT point is not even a local optimum of the max margin problem. On the flip side, we identify multiple settings where a local or global optimum can be guaranteed.

AB - The implicit bias of neural networks has been extensively studied in recent years. Lyu and Li [2019] showed that in homogeneous networks trained with the exponential or the logistic loss, gradient flow converges to a KKT point of the max margin problem in parameter space. However, that leaves open the question of whether this point will generally be an actual optimum of the max margin problem. In this paper, we study this question in detail, for several neural network architectures involving linear and ReLU activations. Perhaps surprisingly, we show that in many cases, the KKT point is not even a local optimum of the max margin problem. On the flip side, we identify multiple settings where a local or global optimum can be guaranteed.

UR - http://www.scopus.com/inward/record.url?scp=85163153335&partnerID=8YFLogxK

M3 - منشور من مؤتمر

T3 - Advances in Neural Information Processing Systems

SP - 37024

EP - 37036

BT - NIPS'22

A2 - Koyejo, S.

A2 - Mohamed, S.

A2 - Agarwal, A.

A2 - Belgrave, D.

A2 - Cho, K.

A2 - Oh, A.

T2 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022

Y2 - 28 November 2022 through 9 December 2022

ER -

On Margin Maximization in Linear and ReLU Networks

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this