Minimax Regret for Stochastic Shortest Path

Alon Cohen; Yonathan Efroni; Yishay Mansour; Aviv Rosenberg

Minimax Regret for Stochastic Shortest Path

Alon Cohen, Yonathan Efroni, Yishay Mansour, Aviv Rosenberg

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We study the Stochastic Shortest Path (SSP) problem in which an agent has to reach a goal state in minimum total expected cost. In the learning formulation of the problem, the agent has no prior knowledge about the costs and dynamics of the model. She repeatedly interacts with the model for K episodes, and has to minimize her regret. In this work we show that the minimax regret for this setting is (equation presented) Õ (√(B²_* + B_*)|S||A|K) where B_* is a bound on the expected cost of the optimal policy from any state, S is the state space, and A is the action space. This matches the Ω(√B²_*|S||A|K) lower bound of Rosenberg et al. [2020] for B_* ≥ 1, and improves their regret bound by a factor of √|S|. For B_* < 1 we prove a matching lower bound of Ω(√B_*|S||A|K). Our algorithm is based on a novel reduction from SSP to finite-horizon MDPs. To that end, we provide an algorithm for the finite-horizon setting whose leading term in the regret depends polynomially on the expected cost of the optimal policy and only logarithmically on the horizon.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Editors	Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Pages	28350-28361
Number of pages	12
ISBN (Electronic)	9781713845393
State	Published - 2021
Event	35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online Duration: 6 Dec 2021 → 14 Dec 2021

Publication series

Name	Advances in Neural Information Processing Systems
Volume	34

Conference

Conference	35th Conference on Neural Information Processing Systems, NeurIPS 2021
City	Virtual, Online
Period	6/12/21 → 14/12/21

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

Cohen, A., Efroni, Y., Mansour, Y., & Rosenberg, A. (2021). Minimax Regret for Stochastic Shortest Path. In MA. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021 (pp. 28350-28361). (Advances in Neural Information Processing Systems; Vol. 34).

Cohen, Alon ; Efroni, Yonathan ; Mansour, Yishay et al. / Minimax Regret for Stochastic Shortest Path. Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. editor / Marc'Aurelio Ranzato ; Alina Beygelzimer ; Yann Dauphin ; Percy S. Liang ; Jenn Wortman Vaughan. 2021. pp. 28350-28361 (Advances in Neural Information Processing Systems).

@inproceedings{6d3a344b842f40aca50208a4b24da0ca,

title = "Minimax Regret for Stochastic Shortest Path",

abstract = "We study the Stochastic Shortest Path (SSP) problem in which an agent has to reach a goal state in minimum total expected cost. In the learning formulation of the problem, the agent has no prior knowledge about the costs and dynamics of the model. She repeatedly interacts with the model for K episodes, and has to minimize her regret. In this work we show that the minimax regret for this setting is (equation presented) {\~O} (√(B2* + B*)|S||A|K) where B* is a bound on the expected cost of the optimal policy from any state, S is the state space, and A is the action space. This matches the Ω(√B2*|S||A|K) lower bound of Rosenberg et al. [2020] for B* ≥ 1, and improves their regret bound by a factor of √|S|. For B* < 1 we prove a matching lower bound of Ω(√B*|S||A|K). Our algorithm is based on a novel reduction from SSP to finite-horizon MDPs. To that end, we provide an algorithm for the finite-horizon setting whose leading term in the regret depends polynomially on the expected cost of the optimal policy and only logarithmically on the horizon.",

author = "Alon Cohen and Yonathan Efroni and Yishay Mansour and Aviv Rosenberg",

note = "Publisher Copyright: {\textcopyright} 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

year = "2021",

language = "الإنجليزيّة",

series = "Advances in Neural Information Processing Systems",

pages = "28350--28361",

editor = "Marc'Aurelio Ranzato and Alina Beygelzimer and Yann Dauphin and Liang, {Percy S.} and {Wortman Vaughan}, Jenn",

booktitle = "Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021",

}

Cohen, A, Efroni, Y, Mansour, Y & Rosenberg, A 2021, Minimax Regret for Stochastic Shortest Path. in MA Ranzato, A Beygelzimer, Y Dauphin, PS Liang & J Wortman Vaughan (eds), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Advances in Neural Information Processing Systems, vol. 34, pp. 28350-28361, 35th Conference on Neural Information Processing Systems, NeurIPS 2021, Virtual, Online, 6/12/21.

Minimax Regret for Stochastic Shortest Path. / Cohen, Alon; Efroni, Yonathan; Mansour, Yishay et al.
Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. ed. / Marc'Aurelio Ranzato; Alina Beygelzimer; Yann Dauphin; Percy S. Liang; Jenn Wortman Vaughan. 2021. p. 28350-28361 (Advances in Neural Information Processing Systems; Vol. 34).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Minimax Regret for Stochastic Shortest Path

AU - Cohen, Alon

AU - Efroni, Yonathan

AU - Mansour, Yishay

AU - Rosenberg, Aviv

PY - 2021

Y1 - 2021

N2 - We study the Stochastic Shortest Path (SSP) problem in which an agent has to reach a goal state in minimum total expected cost. In the learning formulation of the problem, the agent has no prior knowledge about the costs and dynamics of the model. She repeatedly interacts with the model for K episodes, and has to minimize her regret. In this work we show that the minimax regret for this setting is (equation presented) Õ (√(B2* + B*)|S||A|K) where B* is a bound on the expected cost of the optimal policy from any state, S is the state space, and A is the action space. This matches the Ω(√B2*|S||A|K) lower bound of Rosenberg et al. [2020] for B* ≥ 1, and improves their regret bound by a factor of √|S|. For B* < 1 we prove a matching lower bound of Ω(√B*|S||A|K). Our algorithm is based on a novel reduction from SSP to finite-horizon MDPs. To that end, we provide an algorithm for the finite-horizon setting whose leading term in the regret depends polynomially on the expected cost of the optimal policy and only logarithmically on the horizon.

AB - We study the Stochastic Shortest Path (SSP) problem in which an agent has to reach a goal state in minimum total expected cost. In the learning formulation of the problem, the agent has no prior knowledge about the costs and dynamics of the model. She repeatedly interacts with the model for K episodes, and has to minimize her regret. In this work we show that the minimax regret for this setting is (equation presented) Õ (√(B2* + B*)|S||A|K) where B* is a bound on the expected cost of the optimal policy from any state, S is the state space, and A is the action space. This matches the Ω(√B2*|S||A|K) lower bound of Rosenberg et al. [2020] for B* ≥ 1, and improves their regret bound by a factor of √|S|. For B* < 1 we prove a matching lower bound of Ω(√B*|S||A|K). Our algorithm is based on a novel reduction from SSP to finite-horizon MDPs. To that end, we provide an algorithm for the finite-horizon setting whose leading term in the regret depends polynomially on the expected cost of the optimal policy and only logarithmically on the horizon.

UR - http://www.scopus.com/inward/record.url?scp=85131872495&partnerID=8YFLogxK

M3 - منشور من مؤتمر

T3 - Advances in Neural Information Processing Systems

SP - 28350

EP - 28361

BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

A2 - Ranzato, Marc'Aurelio

A2 - Beygelzimer, Alina

A2 - Dauphin, Yann

A2 - Liang, Percy S.

A2 - Wortman Vaughan, Jenn

T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

Y2 - 6 December 2021 through 14 December 2021

ER -

Minimax Regret for Stochastic Shortest Path

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this