Thinking Inside the Ball: Near-Optimal Minimization of the Maximal Loss

Yair Carmon; Arun Jambulapati; Yujia Jin; Aaron Sidford

Thinking Inside the Ball: Near-Optimal Minimization of the Maximal Loss

Yair Carmon, Arun Jambulapati, Yujia Jin, Aaron Sidford

School of Computer Science

Tel Aviv University

Research output: Contribution to journal › Conference article › peer-review

Abstract

We characterize the complexity of minimizing max_i∈[N_] f_i(x) for convex, Lipschitz functions f₁, . . ., f_N. For non-smooth functions, existing methods require O(Nε^-2) queries to a first-order oracle to compute an ε-suboptimal point and O^e(Nε^-1) queries if the f_i are O(1/ε)-smooth. We develop methods with improved complexity bounds of O^e(Nε^-2/3 + ε^-8/3) in the non-smooth case and O^e(Nε^-2/3 + √Nε^-1) in the O(1/ε)-smooth case. Our methods consist of a recently proposed ball optimization oracle acceleration algorithm (which we refine) and a careful implementation of said oracle for the softmax function. We also prove an oracle complexity lower bound scaling as Ω(Nε^-2/3), showing that our dependence on N is optimal up to polylogarithmic factors.

Original language	English
Pages (from-to)	866-882
Number of pages	17
Journal	Proceedings of Machine Learning Research
Volume	134
State	Published - 2021
Event	34th Conference on Learning Theory, COLT 2021 - Boulder, United States Duration: 15 Aug 2021 → 19 Aug 2021

Keywords

Ball optimization oracle
Convex optimization
Min-max problems
Monteiro-Svaiter acceleration
Stochastic first-order methods

All Science Journal Classification (ASJC) codes

Artificial Intelligence
Software
Control and Systems Engineering
Statistics and Probability

Cite this

@article{9bc61822bbf4439ab84db923bff5263a,

title = "Thinking Inside the Ball: Near-Optimal Minimization of the Maximal Loss",

abstract = "We characterize the complexity of minimizing maxi∈[N] fi(x) for convex, Lipschitz functions f1, . . ., fN. For non-smooth functions, existing methods require O(Nε-2) queries to a first-order oracle to compute an ε-suboptimal point and Oe(Nε-1) queries if the fi are O(1/ε)-smooth. We develop methods with improved complexity bounds of Oe(Nε-2/3 + ε-8/3) in the non-smooth case and Oe(Nε-2/3 + √Nε-1) in the O(1/ε)-smooth case. Our methods consist of a recently proposed ball optimization oracle acceleration algorithm (which we refine) and a careful implementation of said oracle for the softmax function. We also prove an oracle complexity lower bound scaling as Ω(Nε-2/3), showing that our dependence on N is optimal up to polylogarithmic factors.",

keywords = "Ball optimization oracle, Convex optimization, Min-max problems, Monteiro-Svaiter acceleration, Stochastic first-order methods",

author = "Yair Carmon and Arun Jambulapati and Yujia Jin and Aaron Sidford",

note = "Publisher Copyright: {\textcopyright} 2021 Y. Carmon, A. Jambulapati, Y. Jin & A. Sidford.; 34th Conference on Learning Theory, COLT 2021 ; Conference date: 15-08-2021 Through 19-08-2021",

year = "2021",

language = "الإنجليزيّة",

volume = "134",

pages = "866--882",

journal = "Proceedings of Machine Learning Research",

issn = "2640-3498",

publisher = "ML Research Press",

}

TY - JOUR

T1 - Thinking Inside the Ball

T2 - 34th Conference on Learning Theory, COLT 2021

AU - Carmon, Yair

AU - Jambulapati, Arun

AU - Jin, Yujia

AU - Sidford, Aaron

PY - 2021

Y1 - 2021

N2 - We characterize the complexity of minimizing maxi∈[N] fi(x) for convex, Lipschitz functions f1, . . ., fN. For non-smooth functions, existing methods require O(Nε-2) queries to a first-order oracle to compute an ε-suboptimal point and Oe(Nε-1) queries if the fi are O(1/ε)-smooth. We develop methods with improved complexity bounds of Oe(Nε-2/3 + ε-8/3) in the non-smooth case and Oe(Nε-2/3 + √Nε-1) in the O(1/ε)-smooth case. Our methods consist of a recently proposed ball optimization oracle acceleration algorithm (which we refine) and a careful implementation of said oracle for the softmax function. We also prove an oracle complexity lower bound scaling as Ω(Nε-2/3), showing that our dependence on N is optimal up to polylogarithmic factors.

AB - We characterize the complexity of minimizing maxi∈[N] fi(x) for convex, Lipschitz functions f1, . . ., fN. For non-smooth functions, existing methods require O(Nε-2) queries to a first-order oracle to compute an ε-suboptimal point and Oe(Nε-1) queries if the fi are O(1/ε)-smooth. We develop methods with improved complexity bounds of Oe(Nε-2/3 + ε-8/3) in the non-smooth case and Oe(Nε-2/3 + √Nε-1) in the O(1/ε)-smooth case. Our methods consist of a recently proposed ball optimization oracle acceleration algorithm (which we refine) and a careful implementation of said oracle for the softmax function. We also prove an oracle complexity lower bound scaling as Ω(Nε-2/3), showing that our dependence on N is optimal up to polylogarithmic factors.

KW - Ball optimization oracle

KW - Convex optimization

KW - Min-max problems

KW - Monteiro-Svaiter acceleration

KW - Stochastic first-order methods

UR - http://www.scopus.com/inward/record.url?scp=85162719529&partnerID=8YFLogxK

M3 - مقالة من مؤنمر

SN - 2640-3498

VL - 134

SP - 866

EP - 882

JO - Proceedings of Machine Learning Research

JF - Proceedings of Machine Learning Research

Y2 - 15 August 2021 through 19 August 2021

ER -

Thinking Inside the Ball: Near-Optimal Minimization of the Maximal Loss

Abstract

Keywords

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this