Two Regimes of Generalization for Non-Linear Metric Learning

Mark Kozdoba; Shie Mannor

Two Regimes of Generalization for Non-Linear Metric Learning

Technion - Israel Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

A common approach to metric learning is to seek an embedding of the input data that behaves well with respect to the labels. While generalization bounds for linear embeddings are known, the non-linear case is not well understood. In this work we fill this gap by
providing uniform generalization guarantees for the case where the metric is induced by a neural network type embedding of the data. Specifically, we discover and analyze two regimes of behavior of the networks, which are roughly related to the sparsity of the last layer. The bounds corresponding to the first regime are based on the spectral and
-norms of the weight matrices, while the second regime bounds use the
-norm at the last layer, and are significantly stronger when the last layer is dense. In addition, we empirically evaluate the behavior of the bounds for networks trained with SGD on the MNIST and 20newsgroups datasets. In particular, we demonstrate that both regimes occur naturally on realistic data.

Original language	English
Title of host publication	Tenth International Conference on Learning Representations
Number of pages	11
State	Published - 2022
Event	Tenth International Conference on Learning Representations - Virtual Duration: 25 Apr 2022 → 29 Apr 2022 Conference number: 10th https://iclr.cc/Conferences/2022

Conference

Conference	Tenth International Conference on Learning Representations
Abbreviated title	ICLR
Period	25/04/22 → 29/04/22
Internet address	https://iclr.cc/Conferences/2022

Access to Document

https://openreview.net/forum?id=zPLQSnfd14w

Cite this

@inproceedings{1c013d426df243a78195b1728036d731,

title = "Two Regimes of Generalization for Non-Linear Metric Learning",

abstract = "A common approach to metric learning is to seek an embedding of the input data that behaves well with respect to the labels. While generalization bounds for linear embeddings are known, the non-linear case is not well understood. In this work we fill this gap by providing uniform generalization guarantees for the case where the metric is induced by a neural network type embedding of the data. Specifically, we discover and analyze two regimes of behavior of the networks, which are roughly related to the sparsity of the last layer. The bounds corresponding to the first regime are based on the spectral and -norms of the weight matrices, while the second regime bounds use the -norm at the last layer, and are significantly stronger when the last layer is dense. In addition, we empirically evaluate the behavior of the bounds for networks trained with SGD on the MNIST and 20newsgroups datasets. In particular, we demonstrate that both regimes occur naturally on realistic data.",

author = "Mark Kozdoba and Shie Mannor",

year = "2022",

language = "الإنجليزيّة",

booktitle = "Tenth International Conference on Learning Representations",

note = "Tenth International Conference on Learning Representations ; Conference date: 25-04-2022 Through 29-04-2022",

url = "https://iclr.cc/Conferences/2022",

}

TY - GEN

T1 - Two Regimes of Generalization for Non-Linear Metric Learning

AU - Kozdoba, Mark

AU - Mannor, Shie

N1 - Conference code: 10th

PY - 2022

Y1 - 2022

N2 - A common approach to metric learning is to seek an embedding of the input data that behaves well with respect to the labels. While generalization bounds for linear embeddings are known, the non-linear case is not well understood. In this work we fill this gap by providing uniform generalization guarantees for the case where the metric is induced by a neural network type embedding of the data. Specifically, we discover and analyze two regimes of behavior of the networks, which are roughly related to the sparsity of the last layer. The bounds corresponding to the first regime are based on the spectral and -norms of the weight matrices, while the second regime bounds use the -norm at the last layer, and are significantly stronger when the last layer is dense. In addition, we empirically evaluate the behavior of the bounds for networks trained with SGD on the MNIST and 20newsgroups datasets. In particular, we demonstrate that both regimes occur naturally on realistic data.

AB - A common approach to metric learning is to seek an embedding of the input data that behaves well with respect to the labels. While generalization bounds for linear embeddings are known, the non-linear case is not well understood. In this work we fill this gap by providing uniform generalization guarantees for the case where the metric is induced by a neural network type embedding of the data. Specifically, we discover and analyze two regimes of behavior of the networks, which are roughly related to the sparsity of the last layer. The bounds corresponding to the first regime are based on the spectral and -norms of the weight matrices, while the second regime bounds use the -norm at the last layer, and are significantly stronger when the last layer is dense. In addition, we empirically evaluate the behavior of the bounds for networks trained with SGD on the MNIST and 20newsgroups datasets. In particular, we demonstrate that both regimes occur naturally on realistic data.

M3 - منشور من مؤتمر

BT - Tenth International Conference on Learning Representations

T2 - Tenth International Conference on Learning Representations

Y2 - 25 April 2022 through 29 April 2022

ER -