Dan Zhang, Leeds School of Business, University of Colorado at Boulder

Last offered in Spring 2012

The lecture slides, with the exception of Approximate Dynamic Programming, are based on the book by Marty Puterman (1994), titled Markov Decision Processes: Discrete Stochastic Dynamic Programming. You may also find the following books highly relevant:

- Dynamic Programming & Optimal Control, Vol. I
- Dynamic Programming and Optimal Control, Vol. II, 4th Edition: Approximate Dynamic Programming
- Neuro-Dynamic Programming
- Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). You may find more information about this book here.
- Approximate Dynamic Programming: Solving the Curses of Dimensionality
- Introduction to Stochastic Dynamic Programming. This is a concise and elegant introduction to stochastic dynamic programming.

The syllabus and selected lecture slides are available for download in pdf format. The syllabus gives a list of course materials used for the class.

Syllabus

Introduction to Dynamic Programming

Applications of Dynamic Programming

Finite Horizon Markov Decision Processes

Infinite Horizon Discounted Markov Decision Processes

Infinite Horizon Average Reward Markov Decision Processes

Structural Properties

Continuous Time Models

Introduction to Approximate Dynamic Programming

Example Matlab Code

A set of matlab code is developed to illustrate several commonly used algorithms to solve dynamic programs. The code is for the eRite-Way example on pages 42-47 of Porteus (2002) book titled Foundations of Stochastic Inventory Theory. [Unfortunately, I cannot post copyrighted material from the book.]

Value iteration for finite horizon MDP

Value iteration for infinite horizon discounted MDP

Policy iteration for infinite horizon discounted MDP

Relative value iteration for infinite horizon average reward MDP

***I would like to thank Felipe Caro (UCLA, Anderson School of Management),Ioana Popescu (INSEAD), and Peng Sun (Duke, Fuqua School of Business) for discussions related to this course and for syllabi of similar PhD courses they offered.

This page was updated on July 5, 2012