GAME THEORY, MAXIMUM ENTROPY, MINIMUM DISCREPANCY AND ROBUST BAYESIAN DECISION THEORY.pdf

The Annals of Statistics
2004, Vol. 32, No. 4, 1367–1433
DOI
© Institute of Mathematical Statistics, 2004
GAME THEORY, MAXIMUM ENTROPY, MINIMUM
DISCREPANCY AND ROBUST BAYESIAN
DECISION THEORY1
BY PETER D. GRÜNWALD AND A. PHILIP DAWID
CWI Amsterdam and University College London
We describe and develop a close relationship between two problems that
have customarily been regarded as distinct: that of maximizing entropy, and
that of minimizing worst-case expected loss. Using a formulation grounded
in the equilibrium theory of zero-sum games between Decision Maker and
Nature, these two problems are shown to be dual to each other, the solution to
each providing that to the other. Although Topsøe described this connection
for the Shannon entropy over 20 years ago, it does not appear to be widely
known even in that important special case.
We here generalize this theory to apply to arbitrary decision problems
and loss functions. We indicate how an appropriate generalized definition of
entropy can be associated with such a problem, and we show that, subject to
certain regularity conditions, the above-mentioned duality continues to apply
in this extended context. This simultaneously provides a possible rationale for
maximizing entropy and a tool for finding robust Bayes acts. We also describe
the essential identity between the problem of maximizing entropy and that of
minimizing a related discrepancy or divergence between distributions. This
leads to an extension, to arbitrary discrepancies, of a well-known minimax
theorem for the case of Kullback–Leibler divergence (the “redundancy-
capacity theorem” of information theory).
For the important case of families of distributions having certain mean
values specified, we develop simple sufficient conditions and methods for
identifying the desired solutions. We use this theory to introduce a new
concept of “generalized exponential family” linked to the