OPIM 7400 Stochastic Dynamic Programming with Applications***

Dan Zhang, Leeds School of Business, University of Colorado at Boulder

Last offered in Spring 2012

The lecture slides, with the exception of Approximate Dynamic Programming, are based on the book by Marty Puterman (1994), titled Markov Decision Processes: Discrete Stochastic Dynamic Programming. You may also find the following books highly relevant:

The syllabus and selected lecture slides are available for download in pdf format. The syllabus gives a list of course materials used for the class.


Introduction to Dynamic Programming

Applications of Dynamic Programming

Finite Horizon Markov Decision Processes

Infinite Horizon Discounted Markov Decision Processes

Infinite Horizon Average Reward Markov Decision Processes

Structural Properties

Continuous Time Models

Introduction to Approximate Dynamic Programming

Example Matlab Code

A set of matlab code is developed to illustrate several commonly used algorithms to solve dynamic programs. The code is for the eRite-Way example on pages 42-47 of Porteus (2002) book titled Foundations of Stochastic Inventory Theory. [Unfortunately, I cannot post copyrighted material from the book.]

Value iteration for finite horizon MDP
Value iteration for infinite horizon discounted MDP
Policy iteration for infinite horizon discounted MDP
Relative value iteration for infinite horizon average reward MDP

***I would like to thank
Felipe Caro (UCLA, Anderson School of Management),Ioana Popescu (INSEAD), and Peng Sun (Duke, Fuqua School of Business) for discussions related to this course and for syllabi of similar PhD courses they offered.

This page was updated on July 5, 2012