Many real-world decision problems have multiple objectives. For example when choosing a
medical treatment plan we want to maximize the efficacy of the treatment but also minimize
the side effects. These objectives typically conflict e.g. we can often increase the efficacy
of the treatment but at the cost of more severe side effects. In this book we outline how to
deal with multiple objectives in decision-theoretic planning and reinforcement learning
algorithms. To illustrate this we employ the popular problem classes of multi-objective Markov
decision processes (MOMDPs) and multi-objective coordination graphs (MO-CoGs). First we
discuss different use cases for multi-objective decision making and why they often necessitate
explicitly multi-objective algorithms. We advocate a utility-based approach to multi-objective
decision making i.e. that what constitutes an optimal solution to a multi-objective decision
problem should be derived from the available information about user utility. We show how
different assumptions about user utility and what types of policies are allowed lead to
different solution concepts which we outline in a taxonomy of multi-objective decision
problems. Second we show how to create new methods for multi-objective decision making using
existing single-objective methods as a basis. Focusing on planning we describe two ways to
creating multi-objective algorithms: in the inner loop approach the inner workings of a
single-objective method are adapted to work with multi-objective solution concepts in the
outer loop approach a wrapper is created around a single-objective method that solves the
multi-objective problem as a series of single-objective problems. After discussing the creation
of such methods for the planning setting we discuss how these approaches apply to the learning
setting. Next we discuss three promising application domains for multi-objective decision
making algorithms: energy health and infrastructure and transportation. Finally we conclude
by outlining important open problems and promising future directions.
