Monthly
288 pp. per issue
6 x 9, illustrated
ISSN
0899-7667
E-ISSN
1530-888X
2014 Impact factor:
2.21

Neural Computation

July 1, 1996, Vol. 8, No. 5, Pages 1075-1084.
(doi: 10.1162/neco.1996.8.5.1075)
© 1996 Massachusetts Institute of Technology
Online Steepest Descent Yields Weights with Nonnormal Limiting Distribution
Article PDF (518.14 KB)
Abstract

We study the asymptotic properties of the sequence of iterates of weight-vector estimates obtained by training a feedforward neural network with a basic gradient-descent method using a fixed learning rate and no batch-processing. Earlier results based on stochastic approximation techniques (Kuan and Hornik 1991; Finnoff 1993; Bucklew et al. 1993) have established the existence of a gaussian limiting distribution for the weights, but they apply only in the limiting case of a zero learning rate. We here prove, from an exact analysis of the one-dimensional case and constant learning rate, weak convergence to a distribution that is not gaussian in general. We also run simulations to compare and contrast the results of our analysis with those of stochastic approximation.