PyTorch bridges flexibility and performance

An imperative programming style supported flexible research with efficient tensor computation.

Adam Paszke and colleagues

AI topics

Explore related entries. Larger tags appear on more entries.

The contribution

The paper describes PyTorch as a deep-learning library combining a familiar imperative programming model with automatic differentiation and accelerated computation. It explains system choices intended to support both experimentation and performance.

What this does not establish

This is a systems contribution. The publication year does not mark the first public availability of the library.

Why this date?

The paper appeared at NeurIPS 2019; its arXiv submission is dated 3 December 2019.

This entry follows the linked publication. Read the source and date conventions.

The original work

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Authors: Adam Paszke and colleagues
Publication: arXiv:1912.01703; NeurIPS 2019
Source type: Preprint / conference paper

Paper and version history

The links above support the description and dating of this entry. Full text may be open or publisher-restricted.

Cited By

Atlas entries whose reference lists cite this work or the publication version noted below.

Learning Transferable Visual Models From Natural Language Supervision (2021)
Listed in the source bibliography. Check the reference list.
Large Language Models are Zero-Shot Reasoners (2022)
Listed in the source bibliography. Check the reference list.

Citations

Works in the Atlas cited by this entry.

No references to other Atlas entries have been recorded for this work yet.

Bibliography coverage: Reference section indexed. Open the source.

Raw extraction retains OCR errors, ligatures, page headers and column-order artifacts. Entries have not all been normalized or individually verified.

Read the indexed bibliography

Reference section 1 (PDF pages 9, 10, 11, 12)

[1] Yangqing "Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor" Darrell. "caffe: Convolutional architecture for fast feature embedding". "arXiv preprint arXiv:1408.5093", "2014". [2] Frank Seide and Amit Agarwal. Cntk: Microsoft’s open-source deep-learning toolkit. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 2135–2135, New York, NY, USA, 2016. ACM. 9 [3] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Largescale machine learning on heterogeneous systems, 2015. Software available from tensorﬂow.org. [4] Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688, May 2016. [5] Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: a next-generation open source framework for deep learning. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS), 2015. [6] Ronan Collobert, Samy Bengio, and Johnny Mariéthoz. Torch: a modular machine learning software library. Technical report, Idiap, 2002. [7] G. Neubig, C. Dyer, Y. Goldberg, A. Matthews, W. Ammar, A. Anastasopoulos, M. Ballesteros, D. Chiang, D. Clothiaux, T. Cohn, K. Duh, M. Faruqui, C. Gan, D. Garrette, Y. Ji, L. Kong, A. Kuncoro, G. Kumar, C. Malaviya, P. Michel, Y. Oda, M. Richardson, N. Saphra, S. Swayamdipta, and P. Yin. DyNet: The Dynamic Neural Network Toolkit. ArXiv e-prints, January 2017. [8] Philip S. Abrams. An APL Machine. PhD thesis, Stanford University, 1970. [9] The MathWorks, Inc., Natick, Massachusetts, United States. MATLAB and Statistics Toolbox. [10] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. [11] Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM review, 59(1):65–98, 2017. [12] Travis Oliphant. NumPy: A guide to NumPy. http://www.numpy.org/. USA: Trelgol Publishing, 2006. [13] Gaël Guennebaud, Benoît Jacob, et al. Eigen v3. http://eigen.tuxfamily.org, 2010. [14] Y LeCun and L Bottou. Lush reference manual. Technical report, code available at http://lush.sourceforge.net, 2002. [15] Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: A survey. J. Mach. Learn. Res., 18(1):5595–5637, January 2017. [16] Dougal Maclaurin. Modeling, Inference and Optimization with Composable Differentiable Procedures. PhD thesis, Harvard University, April 2016. [17] Matthew Johnson et. al. Jax. https://github.com/google/jax, 2018. [18] Mike Innes et. al. Flux.jl. https://github.com/FluxML/Flux.jl, 2018. [19] Eric Jones, Travis Oliphant, Pearu Peterson, et al. SciPy: Open source scientiﬁc tools for Python, 2001–. http://www.scipy.org/. [20] Wes McKinney. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference, 51-56, 2010. [21] Pierre Sermanet, Koray Kavukcuoglu, and Yann LeCun. Eblearn: Open-source energy-based learning in c++. In 2009 21st IEEE International Conference on Tools with Artiﬁcial Intelligence, pages 693–697. IEEE, 2009. 10 [22] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan D. Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efﬁcient primitives for deep learning. CoRR, abs/1410.0759, 2014. [23] Andrew Lavin. maxdnn: An efﬁcient convolution kernel for deep learning with maxwell gpus, January 2015. [24] Andrew Lavin and Scott Gray. Fast algorithms for convolutional neural networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4013–4021, 2016. [25] Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In NIPS 2011, 2011. [26] Richard Gabriel. The rise of worse is better. http://dreamsongs.com/RiseOfWorseIsBetter.html. [27] Yann LeCun and Corinna Cortes. http://yann.lecun.com/exdb/mnist/. MNIST handwritten digit database. [28] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy P. Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. Starcraft II: A new challenge for reinforcement learning. CoRR, abs/1708.04782, 2017. [29] DMLC. Dlpack: Open in memory tensor structure. https://github.com/dmlc/dlpack. [30] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS Workshop, 2017. [31] Dan Piponi. Automatic differentiation, C++ templates, and photogrammetry. J. Graphics, GPU, & Game Tools, 9(4):41–55, 2004. [32] Holger Leuck and Hans-Hellmut Nagel. Automatic differentiation facilitates of-integration into steering-angle-based road vehicle tracking. In 1999 Conference on Computer Vision and Pattern Recognition (CVPR ’99), 23-25 June 1999, Ft. Collins, CO, USA, pages 2360–2365, 1999. [33] The Python team. The cpython https://wiki.python.org/moin/GlobalInterpreterLock. global interpreter lock. [34] Giovanni Petrantoni and Jörg Wollenschläger. Nimtorch. https://github.com/fragcolorxyz/nimtorch. [35] Austin Huang, Junji Hashimoto, https://github.com/hasktorch/hasktorch. and Sam Stites. Hasktorch. [36] G. Synnaeve, Z. Lin, J. Gehring, D. Gant, V. Mella, V. Khalidov, N. Carion, and N. Usunier. Forward modeling for partial observation strategy games - a starcraft defogger. In Advances in Neural Information Processing Systems, pages 10761–10771, 2018. [37] The PyTorch team. Torch Script. https://pytorch.org/docs/stable/jit.html. [38] Justin Luitjens. Cuda streams. GPU technology conference, 2014. [39] Emery D. Berger, Kathryn S. McKinley, Robert D. Blumofe, and Paul R. Wilson. Hoard: A scalable memory allocator for multithreaded applications. In Proceedings of the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS IX, pages 117–128, New York, NY, USA, 2000. ACM. [40] J. Evans. A scalable concurrent malloc(3) implementation for freebsd. In In BSDCan — The Technical BSD Conference, May 2006. [41] S. Ghemawat and P. Menage. Tcmalloc: Thread-caching malloc. 11 [42] Benjamin Recht, Christopher Ré, Stephen J. Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain., pages 693–701, 2011. [43] Matthew Hertz and Emery D. Berger. Quantifying the performance of garbage collection vs. explicit memory management. In Proceedings of the 20th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, OOPSLA ’05, pages 313–326, New York, NY, USA, 2005. ACM. [44] The PyTorch team. Pytorch Autograd Proﬁler. https://pytorch.org/docs/1.0.1/autograd.html#proﬁler.

Download the bibliography index · Download Atlas citation links

These links cover verified matches within the Atlas. They are not total scholarly citation counts. A citation alone does not establish that the later system implements the same method.

AI topics

The contribution

What this does not establish

Why this date?

Comments