Monthly
288 pp. per issue
6 x 9, illustrated
ISSN
0899-7667
E-ISSN
1530-888X
2014 Impact factor:
2.21

Neural Computation

August 15, 1997, Vol. 9, No. 6, Pages 1357-1380
(doi: 10.1162/neco.1997.9.6.1357)
© 1997 Massachusetts Institute of Technology
Information Controller to Maximize and Minimize Information
Article PDF (201.16 KB)
Abstract

We propose an information controller to maximize and simultaneously minimize information in neural networks. The controller can be used to improve generalization performance and interpret explicitly internal representations. An apparently contradictory operation of the simultaneous maximization and minimization is possible by assuming two kinds of information: collective information and individual information, representing collective activities of hidden units and activities of individual input-hidden connections, respectively. By maximizing the collective information and minimizing the individual information, simple networks can be generated in terms of the number of connections and number of hidden units. This method was applied to the inference of the maximum onset principle and the sonority principle of artificial languages. In these problems, it was shown that the individual information could be sufficiently minimized and simultaneously the collective information was maximized. Experimental results confirmed improved generalization performance by the suppression of overtraining. In addition, internal representations created by information maximization and minimization could easily be interpreted.