Abstract
We consider stochastic optimization problems
where data is drawn from a Markov chain. Existing methods for this setting crucially rely on
knowing the mixing time of the chain, which in
real-world applications is usually unknown. We
propose the first optimization method that does
not require the knowledge of the mixing time, yet
obtains the optimal asymptotic convergence rate
when applied to convex problems. We further
show that our approach can be extended to: (i)
finding stationary points in non-convex optimization with Markovian data, and (ii) obtaining better
dependence on the mixing time in temporal difference (TD) learning; in both cases, our method
is completely oblivious to the mixing time. Our
method relies on a novel combination of multilevel Monte Carlo (MLMC) gradient estimation
together with an adaptive learning method.
where data is drawn from a Markov chain. Existing methods for this setting crucially rely on
knowing the mixing time of the chain, which in
real-world applications is usually unknown. We
propose the first optimization method that does
not require the knowledge of the mixing time, yet
obtains the optimal asymptotic convergence rate
when applied to convex problems. We further
show that our approach can be extended to: (i)
finding stationary points in non-convex optimization with Markovian data, and (ii) obtaining better
dependence on the mixing time in temporal difference (TD) learning; in both cases, our method
is completely oblivious to the mixing time. Our
method relies on a novel combination of multilevel Monte Carlo (MLMC) gradient estimation
together with an adaptive learning method.
Original language | English |
---|---|
Pages (from-to) | 5429-5446 |
Journal | Proceedings of Machine Learning Research |
Volume | 162 |
State | Published - 2022 |
Event | Proceedings of the 39th International Conference on Machine Learning - Baltimore, United States Duration: 17 Jul 2022 → 23 Jul 2022 https://proceedings.mlr.press/v162/ |