Data-Driven Methods for Markov Decision Problems with Parameter Uncertainty

In nearly every real sequential decision problem, there is some uncertainty concerning the actual parameters of the decision problem. In this tutorial we survey different approaches to tackle the problem of finding an optimal, or at least a reasonable, policy when the parameters are not known in advance. Not knowing the parameters leads to two problems: first, the policy is not optimal, and second, the estimated return of the policy the decision maker chooses is typically overly optimistic. We advocate for using robust optimization to circumvent these effects and survey different approaches for different settings where the model is data driven, and hence some uncertainty in the parameters must be taken into account.