Sparse algorithms are not stable: A no-free-lunch theorem

Huan Xu; Constantine Caramanis; Shie Mannor

doi:10.1109/TPAMI.2011.177

Sparse algorithms are not stable: A no-free-lunch theorem

Huan Xu, Constantine Caramanis, Shie Mannor

Electrical and Computer Engineering

Technion - Israel Institute of Technology

Research output: Contribution to journal › Article › peer-review

Abstract

We consider two desired properties of learning algorithms: sparsity and algorithmic stability. Both properties are believed to lead to good generalization ability. We show that these two properties are fundamentally at odds with each other: A sparse algorithm cannot be stable and vice versa. Thus, one has to trade off sparsity and stability in designing a learning algorithm. In particular, our general result implies that l₁-regularized regression (Lasso) cannot be stable, while l₂-regularized regression is known to have strong stability properties and is therefore not sparse.

Original language	English
Article number	5989836
Pages (from-to)	187-193
Number of pages	7
Journal	IEEE Transactions on Pattern Analysis and Machine Intelligence
Volume	34
Issue number	1
DOIs	https://doi.org/10.1109/TPAMI.2011.177
State	Published - 2012

Keywords

Lasso
Stability
regularization
sparsity

All Science Journal Classification (ASJC) codes

Software
Computer Vision and Pattern Recognition
Computational Theory and Mathematics
Artificial Intelligence
Applied Mathematics

Access to Document

10.1109/TPAMI.2011.177

Cite this

@article{19fcbcb6d81f4a0bb1c0c34989503367,

title = "Sparse algorithms are not stable: A no-free-lunch theorem",

abstract = "We consider two desired properties of learning algorithms: sparsity and algorithmic stability. Both properties are believed to lead to good generalization ability. We show that these two properties are fundamentally at odds with each other: A sparse algorithm cannot be stable and vice versa. Thus, one has to trade off sparsity and stability in designing a learning algorithm. In particular, our general result implies that l1-regularized regression (Lasso) cannot be stable, while l2-regularized regression is known to have strong stability properties and is therefore not sparse.",

keywords = "Lasso, Stability, regularization, sparsity",

author = "Huan Xu and Constantine Caramanis and Shie Mannor",

note = "Funding Information: This work was supported by NUS startup grant R-265-000-384-133, US National Science Foundation (NSF) (grants EFRI-0735905, CNS-0721532, CNS-0831580), DTRA (grant HDTRA1-08-0029), and Israel Science Foundation (contract 890015). A preliminary version of this work was presented in the 46th Allerton Conference on Communication, Control, and Computing.",

year = "2012",

doi = "10.1109/TPAMI.2011.177",

language = "الإنجليزيّة",

volume = "34",

pages = "187--193",

journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence",

issn = "0162-8828",

publisher = "IEEE Computer Society",

number = "1",

}

TY - JOUR

T1 - Sparse algorithms are not stable

T2 - A no-free-lunch theorem

AU - Xu, Huan

AU - Caramanis, Constantine

AU - Mannor, Shie

N1 - Funding Information: This work was supported by NUS startup grant R-265-000-384-133, US National Science Foundation (NSF) (grants EFRI-0735905, CNS-0721532, CNS-0831580), DTRA (grant HDTRA1-08-0029), and Israel Science Foundation (contract 890015). A preliminary version of this work was presented in the 46th Allerton Conference on Communication, Control, and Computing.

PY - 2012

Y1 - 2012

N2 - We consider two desired properties of learning algorithms: sparsity and algorithmic stability. Both properties are believed to lead to good generalization ability. We show that these two properties are fundamentally at odds with each other: A sparse algorithm cannot be stable and vice versa. Thus, one has to trade off sparsity and stability in designing a learning algorithm. In particular, our general result implies that l1-regularized regression (Lasso) cannot be stable, while l2-regularized regression is known to have strong stability properties and is therefore not sparse.

AB - We consider two desired properties of learning algorithms: sparsity and algorithmic stability. Both properties are believed to lead to good generalization ability. We show that these two properties are fundamentally at odds with each other: A sparse algorithm cannot be stable and vice versa. Thus, one has to trade off sparsity and stability in designing a learning algorithm. In particular, our general result implies that l1-regularized regression (Lasso) cannot be stable, while l2-regularized regression is known to have strong stability properties and is therefore not sparse.

KW - Lasso

KW - Stability

KW - regularization

KW - sparsity

UR - http://www.scopus.com/inward/record.url?scp=81855191843&partnerID=8YFLogxK

U2 - 10.1109/TPAMI.2011.177

DO - 10.1109/TPAMI.2011.177

M3 - مقالة

SN - 0162-8828

VL - 34

SP - 187

EP - 193

JO - IEEE Transactions on Pattern Analysis and Machine Intelligence

JF - IEEE Transactions on Pattern Analysis and Machine Intelligence

IS - 1

M1 - 5989836

ER -

Sparse algorithms are not stable: A no-free-lunch theorem

Abstract

Keywords

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this