Training deeper networks layer by layer

Greedy layerwise learning provided an effective training strategy for deep belief networks.

Geoffrey E. Hinton, Simon Osindero and Yee-Whye Teh

AI topics

Explore related entries. Larger tags appear on more entries.

The contribution

The authors introduced a learning algorithm that trained a deep generative model one layer at a time, followed by a fine-tuning procedure. The work helped renew interest in learning useful hierarchies of representations.

What this does not establish

Layerwise pretraining is a particular method for particular models. It is not a necessary step in all modern deep learning.

Why this date?

The linked Neural Computation paper was published in 2006.

This entry follows the linked publication. Read the source and date conventions.

The original work

A Fast Learning Algorithm for Deep Belief Nets

Authors: Geoffrey E. Hinton, Simon Osindero and Yee-Whye Teh
Publication: Neural Computation, 18, 1527–1554
Source type: Journal article

Author-hosted original paper (PDF)

The links above support the description and dating of this entry. Full text may be open or publisher-restricted.

Cited By

Atlas entries whose reference lists cite this work or the publication version noted below.

Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014)
Listed in the source bibliography. Check the reference list.
Generative Adversarial Networks (2014)
Listed in the source bibliography. Check the reference list.

Citations

Works in the Atlas cited by this entry.

Gradient-Based Learning Applied to Document Recognition (1998)
Listed in the source bibliography. Check the reference list.

Bibliography coverage: Reference section indexed. Open the source.

Raw extraction retains OCR errors, ligatures, page headers and column-order artifacts. Entries have not all been normalized or individually verified.

Read the indexed bibliography

Reference section 1 (PDF pages 10, 11)

Belongie, S., Malik, J., and Puzicha, J. (2002). Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(4):509–522. Carreira-Perpinan, M. A. and Hinton, G. E. (2005). On contrastive divergence learning. In Artiﬁcial Intelligence and Statistics, 2005. Decoste, D. and Schoelkopf, B. (2002). Training invariant support vector machines. Machine Learning, 46:161–190. Freund, Y. (1995). Boosting a weak learning algorithm by majority. Information and Computation, 12(2):256 – 285. Friedman, J. and Stuetzle, W. (1981). Projection pursuit regression. Journal of the American Statistical Association, 76:817–823. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1711–1800. Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. (1995). The wake-sleep algorithm for self-organizing neural networks. Science, 268:1158–1161. LeCun, Y., Bottou, L., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324. Lee, T. S. and Mumford, D. (2003). Hierarchical bayesian inference in the visual cortex. Journal of the Optical Society of America, A., 20:1434–1448. Marks, T. K. and Movellan, J. R. (2001). Diffusion networks, product of experts, and factor analysis. In Proc. Int. Conf. on Independent Component Analysis, pages 481–485. Mayraz, G. and Hinton, G. E. (2001). Recognizing handwritten digits using hierarchical products of experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24:189–197. Neal, R. (1992). Connectionist learning of belief networks. Artiﬁcial Intelligence, 56:71–113. Neal, R. M. and Hinton, G. E. (1998). A new view of the EM algorithm that justiﬁes incremental, sparse and other variants. In Jordan, M. I., editor, Learning in Graphical Models, pages 355—368. Kluwer Academic Publishers. Ning, F., Delhomme, D., LeCun, Y., Piano, F., Bottou, L., and Barbano, P. (2005). Toward automatic phenotyping of developing embryos from videos. IEEE Transactions on Image Processing, 14(9):1360–1371. Pearl, J. (1988). Probabilistic Inference in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo, CA. Roth, S. and Black, M. J. (2005). Fields of experts: A framework for learning image priors. In IEEE Conf. on Computer Vision and Pattern Recognition. Sanger, T. D. (1989). Optimal unsupervised learning in a single-layer linear feedforward neural. Neural Networks, 2(6):459–473. Simard, P. Y., Steinkraus, D., and Platt, J. (2003). Best practice for convolutional neural networks applied to visual document analysis. In International Conference on Document Analysis and Recogntion (ICDAR), IEEE Computer Society, Los Alamitos, pages 958–962. Teh, Y. and Hinton, G. E. (2001). Rate-coded restricted Boltzmann machines for face recognition. In Advances in Neural Information Processing Systems, volume 13. Teh, Y., Welling, M., Osindero, S., and Hinton, G. E. (2003). Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 4:1235– 1260. Welling, M., Hinton, G., and Osindero, S. (2003). Learning sparse topographic representations with products of Student-t distributions. In S. Becker, S. T. and Obermayer, K., editors, Advances in Neural Information Processing Systems 15, pages 1359–1366. MIT Press, Cambridge, MA. Welling, M., Rosen-Zvi, M., and Hinton, G. E. (2005). Exponential family harmoniums with an application to information retrieval. In Advances in Neural Information Processing Systems 17, pages 1481–1488. MIT Press, Cambridge, MA.

Download the bibliography index · Download Atlas citation links

These links cover verified matches within the Atlas. They are not total scholarly citation counts. A citation alone does not establish that the later system implements the same method.

AI topics

The contribution

What this does not establish

Why this date?

Comments