The minimum information principle for discriminative learning
Year: 2004
Type: preprint
Abstract: Exponential models of distributions are widely used in machine learning for classification and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we argue that for classification tasks, mutual information is a more suitable information theoretic measure to be optimized. We show how the principle of minimum mutual information generalizes that of maximum entropy, and provides a comprehensive framework for building discriminative classifiers. A game theoretic interpretation of our approach is then given, and several generalization bounds provided. We present iterative algorithms for solving the minimum information problem and its convex dual, and demonstrate their performance on various classification tasks. The results show that minimum information classifiers outperform the corresponding maximum entropy models.
Source: arXiv (Cornell University)
Authors: Amir Globerson, Naftali Tishby
Institution: Hebrew University of Jerusalem
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2964243709
