288 pp. per issue
6 x 9, illustrated
2014 Impact factor:

Neural Computation

July 1, 1996, Vol. 8, No. 5, Pages 1075-1084
(doi: 10.1162/neco.1996.8.5.1075)
© 1996 Massachusetts Institute of Technology
Online Steepest Descent Yields Weights with Nonnormal Limiting Distribution
Article PDF (518.14 KB)

We study the asymptotic properties of the sequence of iterates of weight-vector estimates obtained by training a feedforward neural network with a basic gradient-descent method using a fixed learning rate and no batch-processing. Earlier results based on stochastic approximation techniques (Kuan and Hornik 1991; Finnoff 1993; Bucklew et al. 1993) have established the existence of a gaussian limiting distribution for the weights, but they apply only in the limiting case of a zero learning rate. We here prove, from an exact analysis of the one-dimensional case and constant learning rate, weak convergence to a distribution that is not gaussian in general. We also run simulations to compare and contrast the results of our analysis with those of stochastic approximation.