¹Computer Science and Operations Research University of Montreal P.O. Box 6128, Montr´eal Qu´ebec, H3C 3J7, Canada

---

Abstract—We describe a general method to transform a non-Markovian sequential decision problem into a supervised learning problem using a K-bestpaths algorithm. We consider an application in financial portfolio management where we can train a controller to directly optimize a Sharpe Ratio (or other risk-averse non-additive) utility function. We illustrate the approach by demonstrating experimental results using a kernel-based controller architecture that would not normally be considered in traditional reinforcement learning or approximate dynamic programming.We further show that using a non-additive criterion (incremental Sharpe Ratio) yields a noisy K-best-paths extraction problem, that can give substantially improved performance.

[PDF]

Cite: Nicolas Chapados, Yoshua Bengio, "Noisy K Best-Paths for Approximate Dynamic Programming with Application to Portfolio Optimization," Journal of Computers vol. 2, no. 1, pp. 12-19, 2007.