Monthly
288 pp. per issue
6 x 9, illustrated
ISSN
0899-7667
E-ISSN
1530-888X
2014 Impact factor:
2.21

Neural Computation

January 2020, Vol. 32, No. 1, Pages 205-260
(doi: 10.1162/neco_a_01247)
© 2019 Massachusetts Institute of Technology
Iterative Retrieval and Block Coding in Autoassociative and Heteroassociative Memory
Article PDF (4.23 MB)
Abstract
Neural associative memories (NAM) are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Gripon and Berrou (2011) investigated NAM employing block coding, a particular sparse coding method, and reported a significant increase in storage capacity. Here we verify and extend their results for both heteroassociative and recurrent autoassociative networks. For this we provide a new analysis of iterative retrieval in finite autoassociative and heteroassociative networks that allows estimating storage capacity for random and block patterns. Furthermore, we have implemented various retrieval algorithms for block coding and compared them in simulations to our theoretical results and previous simulation data. In good agreement of theory and experiments, we find that finite networks employing block coding can store significantly more memory patterns. However, due to the reduced information per block pattern, it is not possible to significantly increase stored information per synapse. Asymptotically, the information retrieval capacity converges to the known limits C=ln20.69 and C=(ln2)/40.17 also for block coding. We have also implemented very large recurrent networks up to n=2·106 neurons, showing that maximal capacity C0.2 bit per synapse occurs for finite networks having a size n105 similar to cortical macrocolumns.