Prior activity becomes context

Feeding earlier hidden activity back into a network created a learned representation of temporal context.

Jeffrey L. Elman

AI topics

Explore related entries. Larger tags appear on more entries.

The contribution

Elman explored a network whose hidden activity at one time step becomes part of the input for the next. Its internal state therefore reflects both current input and earlier processing. The report studied how such dynamic memory could support sequence prediction and linguistic structure without explicitly storing a fixed window of prior symbols.

What this does not establish

A recurrent state is a learned, compressed context, not a lossless transcript. The surviving 1988 newsletter link contains the report abstract; the author archive supplies the later published version.

Why this date?

The April 1988 CRL newsletter identifies Technical Report 8801. LSTM cites this report. The Cognitive Science article appeared in 1990, and that journal version is the one cited by word2vec and LLaMA.

This entry follows the linked publication. Read the source and date conventions.

The original work

Finding Structure in Time

Authors: Jeffrey L. Elman
Publication: UC San Diego, Center for Research in Language, Technical Report 8801
Source type: Technical report

The links above support the description and dating of this entry. Full text may be open or publisher-restricted.

Cited By

Atlas entries whose reference lists cite this work or the publication version noted below.

A Focused Backpropagation Algorithm for Temporal Pattern Recognition (1989)
Listed in the source bibliography. Check the reference list.
Finite State Automata and Simple Recurrent Networks (1989)
Cites the 1988 technical report. The extracted report number reads 9901; the Atlas uses the independently verified report number 8801. Check the reference list.
Long Short-Term Memory (1997)
Cites the 1988 technical report. Check the reference list.
Efficient Estimation of Word Representations in Vector Space (2013)
Reference 8 cites the later 1990 journal version. Check the reference list.
LLaMA: Open and Efficient Foundation Language Models (2023)
Cites the later 1990 journal version. Check the reference list.

Citations

Works in the Atlas cited by this entry.

No references to other Atlas entries have been recorded for this work yet.

Bibliography coverage: Reference section indexed. Open the source.

The author's publication archive states that the 1988 technical report is no longer available and was superseded by the 1990 journal paper: https://jeffelman.ucsd.edu/research/publications/. The complete bibliography of that author-designated replacement is indexed here. All three reference pages were processed with Windows OCR; image and text hashes are retained. This does not establish the bibliography of the unavailable 1988 report. The Atlas chronology continues to date the first report to 1988.

Read the indexed bibliography

References, printed pages 209-211 (PDF pages 31, 32, 33)

REFERENCES Berwick, R.C.. & Weinberg, A.S. (1984). The grammatical of linguistic performance. Cambridge, MA: MIT Pres. Chomsky, N. (1957). Syntactic structures. The Hague: Moutin. Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge. MA: MIT Pres. Cottrell, G.W., Munro. P.W., & Zipser. D. (1987). Image comprasion by back propagation: A demonstration of extensional programming. In N.E. Sharkey (Ed.), Adpanæs in cognitive science (Vol. 2). Chichester. England: Ellis Horwood. Elman. J.L. (1989). Structurd representations and connectionist models. (CRL Tech. Rep. No. 8901). San Diego: University of California, Center for Research in Language. Elman, J.L., & Zipser, D. (1988). Discovering the hidden structure of speech. Journal of the Acoustical Society of America, U, 1615-1626. Fodor. J & Pylyshyn. Z. (1988). Connectionism and cognitive architecture: A critical analysis. In S. Pinker & J. Mehler (Eds.), Connections and symbols (pp. 3-71). Cambridge, MA: MIT Press. Fowler. C. (1977). Timing control in speeh production. Bloomington. IN: Indiana University Linguistics Club. Fowler , C. (1980). Coarticulation and theories of extrinsic timing control. Journal of Phonetics. 8. 113-133. Frazier, L. , & Fodor, J.D. (1978). The sausage machine: A new two-stage parsing model. Cognition. 6. 291-325. Greenberg, J.H. (1963). Universals of language. Cambridge, MA: MIT Press. Grosjean, F. (1980). Spokel word recognition process— and the gating paradigm. Perception & Psychophysics. 28, 267-283. Hanson, S.J., & Keg), J. (1987). Parsnip: A connectionist network that learns natural language grammar from exposure to natural language sentence. Ninth A naual Confeænce of the Cognitive Science Society, Seattle, Washington. Hillsdale, NJ: Erlbaum. Hinton, G.E., McClelland, J.L., & Rumelhart, D.E. (1986). Distributed repreentations. In D.E. Rumelhart & J.L. McClelland (Eds.), hm'lel distributed procsing: 210 ELMAN (ions in the microstructure of cognition (Vol. l, pp. 77-109). Cambridge, MA: MIT Press. Jordan, M.I. (1986). Serial order: A parallel distributed processing approach (Tech. Rep. No. 804). San Diego: University of California, Institute for Cognitive Science. Jordan, M.I., & Rosenbaum, D.A. (1988). Action (Tech. Rep. No. 88-26). Amherst: Univer- sity of Massachusetts, Department of Computer Science. Kelso, J.A.S., Saltzman, E., & Tuner, B. (1986). The dynamical theory of speech production: Data and theory. Journal of Phonetics, 14, 29-60. Lashley, KS. (1951). The problem of serial order in behavior. In L.A. Jeffress (Ed.), Cerebral mechanisms in behavior. New York: Wiley. Lehman, W.P. (1962). Historical linguistics: An introduction. New York: Holt, Rinehart, and Winston. MacNeilage, P.F. (1970). Motor control of serial ordering of speech. Psychological Review. 77, 182-196. MacWhinney, B. (1978). The acquisition of morphophonology. Monographs of the Society for Research in Child Development,' 43, (Serial No. l). Marcus, M. (1980). A theory of syntactic recognition for natural language. Cambridge, MA: MIT Press. Marslen-Wilson, W., & Tyler, L.K. (1980). The temporal structure of spoken language under- standing. Cognition, 8, I-II. Pineda, F.J. (1988). Generalization of back propagation to recurrent and higher order neural networks. In D.Z. Anderson (Ed.), Neural information processing systems. New York: American Institute of Physics. Pinker, S. (1984). Language learnability and language development. Cambridge, MA: Harvard University Press. Rumelhart, D.E., Hinton, G.E., & Williams, R.J. (1986). Learning internal representations by error propagation. In D.E. Rumelhart & J.L. McClelland (Eds.), Parallel distributed processing: Explorations in the microstructure ofcognition (Vol. l, pp. 318-362). Cam- bridge, MA: MIT Press. Salasoo, A., & Pisoni, D.B. (1985). Interaction of knowledge sources in spoken word identifi- cation. Journal of Memory and Language, 24, 210-231. Saltzman, E. , & Kelso, J. A.S. (1987). Skilled actions: A task dynamic approach. Psychological Review, 94, 84-106. Schwanennugel, P.J., & Shoben, E.J. (1985). The influence of sentence constraint on the scope of facilitation for upcoming words. Journal of Memory and Language, 24, 232-252. Sejnowski, T.J ., & Rosenberg, C.R. (1987). Parallel networks that learn to pronounce English text. Complex Systems, l, 145-168. Servan-Schreiber, D., Cleeremans, A. , & McClelland, J. L. (1988). Encoding sequential struc- ture in simple recurrent networks (CMU Tech. Rep. No. CMU-CS-88-183). Pittsburgh, PA: Carnegie-Mellon University, Computer Science Department. Smolensky, P. (1987). On variable binding and the representation of symbolic structures in connectionist systems (Tech. Rep. No. CU-CS-355-87). Boulder, CO: University of Colorado, Department of Computer Science. Smolensky, P. (1988). On the proper treatment of connectionism. The Behavioral and Brain Sciences, 11. Stornetta, W.S., Hogg, T. , & Huberman, B.A. (1987). A dynamical approach to temporal pattern processing. Proceedings of the IEEE Conference on Neural Idormation Pro- cessing Systems. Denver, CO. Swinney, D. (1979). Lexical access during sentence comprehension: (Re)consideration Of con- text effects. Journal of Verbal Learning and Verbal Behavior, 6, 645-659. FINDING STRUCTURE IN TIME 211 Tabossi, P. (1988). Effects of context on the immediate interpretation of unambiguous nouns. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 153-162. Tabossi, P. , Colombo, L. , & Job, R. (1987). Accessing lexical ambiguity: Effects of context and dominance. Psychological Research, 49, 161-167. Tank, D.W., & Hopfield, J. J. (1987, June). Neural computation by concentrating information in time. Proceedings of the IEEE International Conference on Neural Networks. San Diego, CA. Van Gelder, T.J. (in press). Compositionality: Variations on a classical theme. Cognitive Science. Waibel, A. , Hanazawa, T., Hinton, G. , Shikano, K. , & Lang, K. (1987). Phoneme recognition using time-delay neural networks (ATR Tech. Rep. TR-I-0006). Japan: ATR Interpret- ing Telephony Research Laboratories. Watrous, R.L., & Shastri, L. (1987). Learning phonetic features using connectionist networks: An experiment in speech recognition. Proceedings of the IEEE International Confer- ence on Neural Networks. San Diego, CA. Williams, R.J., & Zipser, D. (1988). A learning algorithm for continually runningfully recur- rent neural networks (Tech. Rep. No. 8805). San Diego: University of California, Insti- tute for Cognitive Science.

Download the bibliography index · Download Atlas citation links

These links cover verified matches within the Atlas. They are not total scholarly citation counts. A citation alone does not establish that the later system implements the same method.

AI topics

The contribution

What this does not establish

Why this date?

Comments