This book presents classical Markov Decision Processes (MDP) for real-life applications and
optimization. MDP allows users to develop and formally support approximate and simple decision
rules and this book showcases state-of-the-art applications in which MDP was key to the
solution approach. The book is divided into six parts. Part 1 is devoted to the
state-of-the-art theoretical foundation of MDP including approximate methods such as policy
improvement successive approximation and infinite state spaces as well as an instructive
chapter on Approximate Dynamic Programming. It then continues with five parts of specific and
non-exhaustive application areas. Part 2 covers MDP healthcare applications which includes
different screening procedures appointment scheduling ambulance scheduling and blood
management. Part 3 explores MDP modeling within transportation. This ranges from public to
private transportation from airports and traffic lights to car parking or charging your
electric car . Part 4 contains three chapters that illustrates the structure of approximate
policies for production or manufacturing structures. In Part 5 communications is highlighted
as an important application area for MDP. It includes Gittins indices down-to-earth call
centers and wireless sensor networks. Finally Part 6 is dedicated to financial modeling
offering an instructive review to account for financial portfolios and derivatives under
proportional transactional costs. The MDP applications in this book illustrate a variety of
both standard and non-standard aspects of MDP modeling and its practical use. This book should
appeal to readers for practitioning academic research and educational purposes with a
background in among others operations research mathematics computer science and industrial
engineering.
