Proper Learning, Helly Number, and an Optimal SVM Bound

Olivier Bousquet, Steve Hanneke, Shay Moran, Nikita Zhivotovskiy

Research output: Contribution to journalConference articlepeer-review

Abstract

The classical PAC sample complexity bounds are stated for any Empirical Risk Minimizer (ERM) and contain an extra multiplicative logarithmic factor log 1/ε which is known to be necessary for ERM in general. It has been recently shown by Hanneke (2016a) that the optimal sample complexity of PAC learning for any VC class ℂ does not include this log factor and is achieved by a particular improper learning algorithm, which outputs a specific majority-vote of hypotheses in ℂ. This leaves the question of when this bound can be achieved by proper learning algorithms, which are restricted to always output a hypothesis from ℂ. In this paper we aim to characterize the classes for which the optimal sample complexity can be achieved by a proper learning algorithm. We identify that these classes can be characterized by the dual Helly number, which is a combinatorial parameter that arises in discrete geometry and abstract convexity. In particular, under general conditions on C, we show that the dual Helly number is bounded if and only if there is a proper learner that obtains the optimal dependence on ε. As further implications of our techniques we resolve a long-standing open problem posed by Vapnik and Chervonenkis (1974) on the performance of the Support Vector Machine in ℝn by proving that the sample complexity of SVM in the realizable case is Θ (n/ε + 1/ε log 1/δ ) . This gives the first optimal PAC bound for Halfspaces in ℝn achieved by a proper learning algorithm, and moreover is computationally efficient.

Original languageEnglish
Pages (from-to)582-609
Number of pages28
JournalProceedings of Machine Learning Research
Volume125
StatePublished - 2020
Externally publishedYes
Event33rd Conference on Learning Theory, COLT 2020 - Virtual, Online, Austria
Duration: 9 Jul 202012 Jul 2020

Keywords

  • PAC Learning
  • Proper Learning
  • Sample Complexity
  • Statistical Learning Theory
  • SVM

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability

Fingerprint

Dive into the research topics of 'Proper Learning, Helly Number, and an Optimal SVM Bound'. Together they form a unique fingerprint.

Cite this