Long Short-Term Memory

Memory cells and gates addressed difficulties in learning long-range dependencies.

Sepp Hochreiter and Jürgen Schmidhuber

AI topics

Explore related entries. Larger tags appear on more entries.

The contribution

LSTM introduced a recurrent architecture designed to preserve useful error signals over long intervals. Its memory mechanism and gates provided a way to learn which information to retain and when to use it in sequence tasks.

What this does not establish

LSTM mitigates particular learning difficulties; it does not guarantee perfect recall or unlimited reasoning over long sequences.

Why this date?

The original journal paper appeared in November 1997.

This entry follows the linked publication. Read the source and date conventions.

The original work

Long Short-Term Memory

Authors: Sepp Hochreiter and Jürgen Schmidhuber
Publication: Neural Computation, 9 (8), 1735–1780
Source type: Journal article

The links above support the description and dating of this entry. Full text may be open or publisher-restricted.

Cited By

Atlas entries whose reference lists cite this work or the publication version noted below.

Deep Residual Learning for Image Recognition (2015)
Listed in the source bibliography. Check the reference list.
Attention Is All You Need (2017)
Reference 13 cites Hochreiter and Schmidhuber, Neural Computation 9(8), 1997. Check the reference list.
LLaMA: Open and Efficient Foundation Language Models (2023)
Listed in the source bibliography. Check the reference list.

Citations

Works in the Atlas cited by this entry.

The Utility Driven Dynamic Error Propagation Network (1987)
The bibliography cites Robinson and Fallside (1987), CUED/F-INFENG/TR.1. Check the reference list.
Finding Structure in Time (1988)
Cites the 1988 technical report. Check the reference list.
A Focused Backpropagation Algorithm for Temporal Pattern Recognition (1989)
Cites Mozer (1989), Complex Systems 3, 349-381, with a title variant. Check the reference list.
Finite State Automata and Simple Recurrent Networks (1989)
Cites Cleeremans, Servan-Schreiber and McClelland (1989). Check the reference list.
Learning Sequential Structure with the Real-Time Recurrent Learning Algorithm (1989)
Cites Smith and Zipser (1989), International Journal of Neural Systems 1 (2), 125-131. Check the reference list.

Bibliography coverage: Reference section indexed. Open the source.

Raw extraction retains OCR errors, ligatures, page headers and column-order artifacts. Entries have not all been normalized or individually verified.

Read the indexed bibliography

Reference section 1 (PDF pages 30, 31, 32)

Almeida, L. B. (1987). A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In IEEE 1st International Conference on Neural Networks, San Diego, volume 2, pages 609{618. Baldi, P. and Pineda, F. (1991). Contrastive learning and neural oscillator. Neural Computation, 3:526{545. Bengio, Y. and Frasconi, P. (1994). Credit assignment through time: Alternatives to backpropagation. In Cowan, J. D., Tesauro, G., and Alspector, J., editors, Advances in Neural Information Processing Systems 6, pages 75{82. San Mateo, CA: Morgan Kaufmann. Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is di cult. IEEE Transactions on Neural Networks, 5(2):157{166. Cleeremans, A., Servan-Schreiber, D., and McClelland, J. L. (1989). Finite-state automata and simple recurrent networks. Neural Computation, 1:372{381. de Vries, B. and Principe, J. C. (1991). A theory for neural networks with time delays. In Lippmann, R. P., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing Systems 3, pages 162{168. San Mateo, CA: Morgan Kaufmann. Doya, K. (1992). Bifurcations in the learning of recurrent neural networks. In Proceedings of 1992 IEEE International Symposium on Circuits and Systems, pages 2777{2780. Doya, K. and Yoshizawa, S. (1989). Adaptive neural oscillator using continuous-time backpropagation learning. Neural Networks, 2:375{385. Elman, J. L. (1988). Finding structure in time. Technical Report CRL Technical Report 8801, Center for Research in Language, University of California, San Diego. Fahlman, S. E. (1991). The recurrent cascade-correlation learning algorithm. In Lippmann, R. P., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing Systems 3, pages 190{196. San Mateo, CA: Morgan Kaufmann. Hochreiter, J. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut fur Informatik, Lehrstuhl Prof. Brauer, Technische Universitat Munchen. See www7.informatik.tu-muenchen.de/~hochreit. 30 Hochreiter, S. and Schmidhuber, J. (1995). Long short-term memory. Technical Report FKI-20795, Fakultat fur Informatik, Technische Universitat Munchen. Hochreiter, S. and Schmidhuber, J. (1996). Bridging long time lags by weight guessing and \Long Short-Term Memory". In Silva, F. L., Principe, J. C., and Almeida, L. B., editors, Spatiotemporal models in biological and arti cial systems, pages 65{72. IOS Press, Amsterdam, Netherlands. Serie: Frontiers in Arti cial Intelligence and Applications, Volume 37. Hochreiter, S. and Schmidhuber, J. (1997). LSTM can solve hard long time lag problems. In Advances in Neural Information Processing Systems 9. MIT Press, Cambridge MA. Presented at NIPS 96. Lang, K., Waibel, A., and Hinton, G. E. (1990). A time-delay neural network architecture for isolated word recognition. Neural Networks, 3:23{43. Miller, C. B. and Giles, C. L. (1993). Experimental comparison of the e ect of order in recurrent neural networks. International Journal of Pattern Recognition and Arti cial Intelligence, 7(4):849{872. Mozer, M. C. (1989). A focused back-propagation algorithm for temporal sequence recognition. Complex Systems, 3:349{381. Mozer, M. C. (1992). Induction of multiscale temporal structure. In Lippman, D. S., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing Systems 4, pages 275{282. San Mateo, CA: Morgan Kaufmann. Pearlmutter, B. A. (1989). Learning state space trajectories in recurrent neural networks. Neural Computation, 1(2):263{269. Pearlmutter, B. A. (1995). Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions on Neural Networks, 6(5):1212{1228. Pineda, F. J. (1987). Generalization of back-propagation to recurrent neural networks. Physical Review Letters, 19(59):2229{2232. Pineda, F. J. (1988). Dynamics and architecture for neural computation. Journal of Complexity, 4:216{245. Plate, T. A. (1993). Holographic recurrent networks. In S. J. Hanson, J. D. C. and Giles, C. L., editors, Advances in Neural Information Processing Systems 5, pages 34{41. San Mateo, CA: Morgan Kaufmann. Pollack, J. B. (1991). Language induction by phase transition in dynamical recognizers. In Lippmann, R. P., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing Systems 3, pages 619{626. San Mateo, CA: Morgan Kaufmann. Puskorius, G. V. and Feldkamp, L. A. (1994). Neurocontrol of nonlinear dynamical systems with Kalman lter trained recurrent networks. IEEE Transactions on Neural Networks, 5(2):279{ 297. Ring, M. B. (1993). Learning sequential tasks by incrementally adding higher orders. In S. J. Hanson, J. D. C. and Giles, C. L., editors, Advances in Neural Information Processing Systems 5, pages 115{122. Morgan Kaufmann. Robinson, A. J. and Fallside, F. (1987). The utility driven dynamic error propagation network. Technical Report CUED/F-INFENG/TR.1, Cambridge University Engineering Department. Schmidhuber, J. (1989). The Neural Bucket Brigade: A local learning algorithm for dynamic feedforward and recurrent networks. Connection Science, 1(4):403{412. 31 Schmidhuber, J. (1992a). A xed size storage O(n3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation, 4(2):243{248. Schmidhuber, J. (1992b). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234{242. Schmidhuber, J. (1992c). Learning unambiguous reduced sequence descriptions. In Moody, J. E., Hanson, S. J., and Lippman, R. P., editors, Advances in Neural Information Processing Systems 4, pages 291{298. San Mateo, CA: Morgan Kaufmann. Schmidhuber, J. (1993). Netzwerkarchitekturen, Zielfunktionen und Kettenregel. Habilitationsschrift, Institut fur Informatik, Technische Universitat Munchen. Schmidhuber, J. and Hochreiter, S. (1996). Guessing can outperform many long time lag algorithms. Technical Report IDSIA-19-96, IDSIA. Silva, G. X., Amaral, J. D., Langlois, T., and Almeida, L. B. (1996). Faster training of recurrent networks. In Silva, F. L., Principe, J. C., and Almeida, L. B., editors, Spatiotemporal models in biological and arti cial systems, pages 168{175. IOS Press, Amsterdam, Netherlands. Serie: Frontiers in Arti cial Intelligence and Applications, Volume 37. Smith, A. W. and Zipser, D. (1989). Learning sequential structures with the real-time recurrent learning algorithm. International Journal of Neural Systems, 1(2):125{131. Sun, G., Chen, H., and Lee, Y. (1993). Time warping invariant neural networks. In S. J. Hanson, J. D. C. and Giles, C. L., editors, Advances in Neural Information Processing Systems 5, pages 180{187. San Mateo, CA: Morgan Kaufmann. Watrous, R. L. and Kuhn, G. M. (1992). Induction of nite-state languages using second-order recurrent networks. Neural Computation, 4:406{414. Werbos, P. J. (1988). Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1. Williams, R. J. (1989). Complexity of exact gradient computation algorithms for recurrent neural networks. Technical Report Technical Report NU-CCS-89-27, Boston: Northeastern University, College of Computer Science. Williams, R. J. and Peng, J. (1990). An e cient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation, 4:491{501. Williams, R. J. and Zipser, D. (1992). Gradient-based learning algorithms for recurrent networks and their computational complexity. In Back-propagation: Theory, Architectures and Applications. Hillsdale, NJ: Erlbaum.

Download the bibliography index · Download Atlas citation links

These links cover verified matches within the Atlas. They are not total scholarly citation counts. A citation alone does not establish that the later system implements the same method.

AI topics

The contribution

What this does not establish

Why this date?

Comments