Finite sample analyses for TD(0) with function approximation

Gal Dalal; Balázs Szörényi; Gugan Thoppe; Shie Mannor

Finite sample analyses for TD(0) with function approximation

Gal Dalal
, Balázs Szörényi
, Gugan Thoppe
, Shie Mannor

Technion - Israel Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

TD(0) is one of the most commonly used algorithms in reinforcement learning. Despite this, there is no existing finite sample analysis for TD(0) with function approximation, even for the linear case. Our work is the first to provide such results. Works that managed to obtain convergence rates for online Temporal Difference (TD) methods analyzed somewhat modified versions of them that include projections and stepsize dependent on unknown problem parameters. Our analysis obviates these artificial alterations by exploiting strong properties of TD(0). We provide convergence rates both in expectation and with high-probability. Both are based on relatively unknown, recently developed stochastic approximation techniques.

Original language	English
Title of host publication	32nd AAAI Conference on Artificial Intelligence, AAAI 2018
Pages	6144-6160
Number of pages	17
ISBN (Electronic)	9781577358008
State	Published - 2018
Event	32nd AAAI Conference on Artificial Intelligence, AAAI 2018 - New Orleans, United States Duration: 2 Feb 2018 → 7 Feb 2018

Publication series

Name	32nd AAAI Conference on Artificial Intelligence, AAAI 2018

Conference

Conference	32nd AAAI Conference on Artificial Intelligence, AAAI 2018
Country/Territory	United States
City	New Orleans
Period	2/02/18 → 7/02/18

ASJC Scopus subject areas

Artificial Intelligence

Access to Document

https://aaai.org/papers/12079-finite-sample-analyses-for-td0-with-function-approximation/

Cite this

@inproceedings{3d446b16a77b46c18f8192cd5d765939,

title = "Finite sample analyses for TD(0) with function approximation",

abstract = "TD(0) is one of the most commonly used algorithms in reinforcement learning. Despite this, there is no existing finite sample analysis for TD(0) with function approximation, even for the linear case. Our work is the first to provide such results. Works that managed to obtain convergence rates for online Temporal Difference (TD) methods analyzed somewhat modified versions of them that include projections and stepsize dependent on unknown problem parameters. Our analysis obviates these artificial alterations by exploiting strong properties of TD(0). We provide convergence rates both in expectation and with high-probability. Both are based on relatively unknown, recently developed stochastic approximation techniques.",

author = "Gal Dalal and Bal{\'a}zs Sz{\"o}r{\'e}nyi and Gugan Thoppe and Shie Mannor",

note = "Publisher Copyright: Copyright {\textcopyright} 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.; 32nd AAAI Conference on Artificial Intelligence, AAAI 2018 ; Conference date: 02-02-2018 Through 07-02-2018",

year = "2018",

language = "English",

series = "32nd AAAI Conference on Artificial Intelligence, AAAI 2018",

pages = "6144--6160",

booktitle = "32nd AAAI Conference on Artificial Intelligence, AAAI 2018",

}

Dalal, G, Szörényi, B, Thoppe, G & Mannor, S 2018, Finite sample analyses for TD(0) with function approximation. in 32nd AAAI Conference on Artificial Intelligence, AAAI 2018. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 6144-6160, 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, New Orleans, United States, 2/02/18. <https://aaai.org/papers/12079-finite-sample-analyses-for-td0-with-function-approximation/>

TY - GEN

T1 - Finite sample analyses for TD(0) with function approximation

AU - Dalal, Gal

AU - Szörényi, Balázs

AU - Thoppe, Gugan

AU - Mannor, Shie

PY - 2018

Y1 - 2018

N2 - TD(0) is one of the most commonly used algorithms in reinforcement learning. Despite this, there is no existing finite sample analysis for TD(0) with function approximation, even for the linear case. Our work is the first to provide such results. Works that managed to obtain convergence rates for online Temporal Difference (TD) methods analyzed somewhat modified versions of them that include projections and stepsize dependent on unknown problem parameters. Our analysis obviates these artificial alterations by exploiting strong properties of TD(0). We provide convergence rates both in expectation and with high-probability. Both are based on relatively unknown, recently developed stochastic approximation techniques.

AB - TD(0) is one of the most commonly used algorithms in reinforcement learning. Despite this, there is no existing finite sample analysis for TD(0) with function approximation, even for the linear case. Our work is the first to provide such results. Works that managed to obtain convergence rates for online Temporal Difference (TD) methods analyzed somewhat modified versions of them that include projections and stepsize dependent on unknown problem parameters. Our analysis obviates these artificial alterations by exploiting strong properties of TD(0). We provide convergence rates both in expectation and with high-probability. Both are based on relatively unknown, recently developed stochastic approximation techniques.

UR - https://www.scopus.com/pages/publications/85060436473

M3 - Conference contribution

T3 - 32nd AAAI Conference on Artificial Intelligence, AAAI 2018

SP - 6144

EP - 6160

BT - 32nd AAAI Conference on Artificial Intelligence, AAAI 2018

T2 - 32nd AAAI Conference on Artificial Intelligence, AAAI 2018

Y2 - 2 February 2018 through 7 February 2018

ER -

Finite sample analyses for TD(0) with function approximation

Abstract

Publication series

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this