Activity traces preserve temporal information

Self-connected hidden units integrated a sequence while local traces supplied information needed for learning.

Michael C. Mozer

AI topics

Explore related entries. Larger tags appear on more entries.

The contribution

Mozer designed a recurrent architecture in which hidden units combine new input with retained activity. The associated learning method focuses error propagation and computes local traces during forward processing. This offered a way to learn temporal patterns without retaining the entire input sequence and every intermediate activation in a separate buffer.

What this does not establish

The method is specialized to the proposed architecture. Its compact traces do not imply unlimited recall or remove every difficulty of learning long dependencies.

Why this date?

The original journal paper appeared in 1989. LSTM cites it under the variant title A focused back-propagation algorithm for temporal sequence recognition; this entry uses the title printed by the original journal.

This entry follows the linked publication. Read the source and date conventions.

The original work

A Focused Backpropagation Algorithm for Temporal Pattern Recognition

Authors: Michael C. Mozer
Publication: Complex Systems, 3 (4), 349-381
Source type: Journal article

The links above support the description and dating of this entry. Full text may be open or publisher-restricted.

Cited By

Atlas entries whose reference lists cite this work or the publication version noted below.

Long Short-Term Memory (1997)
Cites Mozer (1989), Complex Systems 3, 349-381, with a title variant. Check the reference list.

Citations

Works in the Atlas cited by this entry.

Finding Structure in Time (1988)
Listed in the source bibliography. Check the reference list.

Bibliography coverage: Reference section indexed. Open the source.

Scanned bibliography pages processed with Windows OCR. The readable citation metadata was checked against the images and OCR; raw OCR is retained with page numbers and hashes. Spelling and source-publication inconsistencies may remain. Only reviewed Atlas matches become citation links.

Read the indexed bibliography

Bibliography (PDF pages 16, 17)

1. D.E. Rumelhart and J.L. McClelland. On learning the past tenses of English verbs. Parallel Distributed Processing, volume II, 1986, 216-271. 2. T.J. Sejnowski and C.R. Rosenberg. Parallel networks that learn to pronounce English text. Complex Systems 1 (1987), 145-168. 3. G. Hinton. Learning distributed representations of concepts. Proceedings of the Eighth Annual Conference of the Cognitive Science Society (1987), 1-12. 4. P. Smolensky. Schema selection and stochastic inference in modular environments. Proceedings AAAI-83 (1983), 109-113. 5. J. Freyd. Dynamic mental representations. Psychological Review 94 (1987), 427-438. 6. J.L. Elman and J.L. McClelland. Exploiting lawful variability in the speech wave. In Invariance and variability in speech processes (1986), 360-380. 7. J.L. Elman and D. Zipser. Learning the hidden structure of speech. Journal of the Acoustical Society of America, in press. 8. T.K. Landauer, C.A. Kamm and S. Singhal. Teaching a minimally structured back propagation network to recognize speech. Proceedings of the Ninth Annual Conference of the Cognitive Science Society (1987), 531-536. 9. A. Lapedes and R. Farber. Nonlinear signal processing using neural networks. Report LA-UR-87-2662, Los Alamos, 1987. 10. J.L. McClelland and J.L. Elman. Interactive processes in speech perception: The TRACE model. Parallel Distributed Processing, volume II (1986), 58-121. 11. D.C. Plaut, S. Nowlan and G.E. Hinton. Experiments on learning by back propagation. Technical Report CMU, Carnegie-Mellon University, 1986. 12. D. Tank and J. Hopfield. Proceedings of the National Academy of Sciences 84 (1987), 1896. No article title supplied in the reference. 13. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano and K. Lang. Phoneme recognition using time-delay neural networks. Technical Report 1-0006, ATR Interpreting Telephony Research Laboratories, 1987. 14. G. Hinton. Connectionist learning procedures. Artificial Intelligence (1988), in press. 15. K. Lang. Connectionist speech recognition. Unpublished Ph.D. thesis proposal, Carnegie-Mellon University, 1987. 16. J.L. Elman. Finding structure in time. CRL Technical Report 8801, University of California, San Diego, Center for Research in Language, 1988. 17. W.S. Stornetta, T. Hogg and B.A. Huberman. A dynamical approach to temporal pattern processing. Proceedings of the IEEE Conference on Neural Information Processing Systems, 1987. 18. R.L. Watrous and L. Shastri. Learning acoustic features from speech data using connectionist networks. Proceedings of the Ninth Annual Conference of the Cognitive Science Society (1987), 518-530. 19. M.I. Jordan. Attractor dynamics and parallelism in a connectionist sequential machine. Proceedings of the Eighth Annual Conference of the Cognitive Science Society (1987), 531-546. 20. D.E. Rumelhart, G.E. Hinton and R.J. Williams. Learning internal representations by error propagation. Parallel Distributed Processing, volume I (1986), 318-362. 21. L. Almeida. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. IEEE First Annual International Conference on Neural Networks, 1987. 22. F. Pineda. Generalization of back propagation to recurrent networks. Memo S1A-63-87, Johns Hopkins University Applied Physics Laboratory, 1987. 23. W. Wickelgren. Context-sensitive coding, associative memory, and serial order in (speech) behavior. Psychological Review 76 (1969), 1-15. 24. M.C. Mozer. Early parallel processing in reading: A connectionist approach. Attention and Performance XII (1987), 83-104. 25. M.C. Mozer. The perception of multiple objects: A parallel, distributed processing approach. ICS Technical Report 8803, University of California, San Diego, 1988. 26. Y. Miyata. The learning and planning of actions. ICS Technical Report 8802, University of California, San Diego, 1988. 27. J. Bachrach. Learning to represent state. Unpublished master's thesis, University of Massachusetts, Amherst, 1988. 28. J.L. Bybee and D.I. Slobin. Rules and schemas in the development and use of the English past tense. Language 58 (1982), 265-289. 29. Steven Nowlan. Personal communication. 30. R.J. Williams and D. Zipser. Experimental analysis of the real-time recurrent learning algorithm. Connection Science 1, in press. 31. M. Gori, Y. Bengio and R. De Mori. BPS: A learning algorithm for capturing the dynamic nature of speech. Proceedings of the 1989 First International Joint Conference on Neural Networks, volume 2 (1989), 417-423. 32. Yoshiro Miyata. Personal communication.

Download the bibliography index · Download Atlas citation links

These links cover verified matches within the Atlas. They are not total scholarly citation counts. A citation alone does not establish that the later system implements the same method.

AI topics

The contribution

What this does not establish

Why this date?

Comments