Faster Neural Network Training with Approximate Tensor Operations

Menachem Adelman; Kfir Y. Levy; Ido Hakimi; Mark Silberstein

Faster Neural Network Training with Approximate Tensor Operations

Menachem Adelman, Kfir Y. Levy, Ido Hakimi, Mark Silberstein

Technion - Israel Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i.e., matrix multiplications and convolutions. We introduce new sampling techniques, study their theoretical properties, and prove that they provide the same convergence guarantees when applied to SGD training. We apply approximate tensor operations to single and multi-node training of MLP and CNN networks on MNIST, CIFAR-10 and ImageNet datasets. We demonstrate up to 66% reduction in the amount of computations and communication, and up to 1.37x faster training time while maintaining negligible or no impact on the final test accuracy.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Editors	Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Pages	27877-27889
Number of pages	13
ISBN (Electronic)	9781713845393
State	Published - 2021
Event	35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online Duration: 6 Dec 2021 → 14 Dec 2021

Publication series

Name	Advances in Neural Information Processing Systems
Volume	33

Conference

Conference	35th Conference on Neural Information Processing Systems, NeurIPS 2021
City	Virtual, Online
Period	6/12/21 → 14/12/21

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Information Systems
Signal Processing

Access to Document

https://openreview.net/pdf?id=Rz-hPxb6ODl

Cite this

Adelman, M., Levy, K. Y., Hakimi, I., & Silberstein, M. (2021). Faster Neural Network Training with Approximate Tensor Operations. In MA. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021 (pp. 27877-27889). (Advances in Neural Information Processing Systems; Vol. 33). https://openreview.net/pdf?id=Rz-hPxb6ODl

Adelman, Menachem ; Levy, Kfir Y. ; Hakimi, Ido et al. / Faster Neural Network Training with Approximate Tensor Operations. Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. editor / Marc'Aurelio Ranzato ; Alina Beygelzimer ; Yann Dauphin ; Percy S. Liang ; Jenn Wortman Vaughan. 2021. pp. 27877-27889 (Advances in Neural Information Processing Systems).

@inproceedings{f3e889059002417f86408caca744b858,

title = "Faster Neural Network Training with Approximate Tensor Operations",

abstract = "We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i.e., matrix multiplications and convolutions. We introduce new sampling techniques, study their theoretical properties, and prove that they provide the same convergence guarantees when applied to SGD training. We apply approximate tensor operations to single and multi-node training of MLP and CNN networks on MNIST, CIFAR-10 and ImageNet datasets. We demonstrate up to 66\% reduction in the amount of computations and communication, and up to 1.37x faster training time while maintaining negligible or no impact on the final test accuracy.",

author = "Menachem Adelman and Levy, \{Kfir Y.\} and Ido Hakimi and Mark Silberstein",

note = "Publisher Copyright: {\textcopyright} 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

year = "2021",

language = "الإنجليزيّة",

series = "Advances in Neural Information Processing Systems",

pages = "27877--27889",

editor = "Marc'Aurelio Ranzato and Alina Beygelzimer and Yann Dauphin and Liang, \{Percy S.\} and \{Wortman Vaughan\}, Jenn",

booktitle = "Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021",

}

Adelman, M, Levy, KY, Hakimi, I & Silberstein, M 2021, Faster Neural Network Training with Approximate Tensor Operations. in MA Ranzato, A Beygelzimer, Y Dauphin, PS Liang & J Wortman Vaughan (eds), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Advances in Neural Information Processing Systems, vol. 33, pp. 27877-27889, 35th Conference on Neural Information Processing Systems, NeurIPS 2021, Virtual, Online, 6/12/21. <https://openreview.net/pdf?id=Rz-hPxb6ODl>

Faster Neural Network Training with Approximate Tensor Operations. / Adelman, Menachem; Levy, Kfir Y.; Hakimi, Ido et al.
Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. ed. / Marc'Aurelio Ranzato; Alina Beygelzimer; Yann Dauphin; Percy S. Liang; Jenn Wortman Vaughan. 2021. p. 27877-27889 (Advances in Neural Information Processing Systems; Vol. 33).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Faster Neural Network Training with Approximate Tensor Operations

AU - Adelman, Menachem

AU - Levy, Kfir Y.

AU - Hakimi, Ido

AU - Silberstein, Mark

PY - 2021

Y1 - 2021

N2 - We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i.e., matrix multiplications and convolutions. We introduce new sampling techniques, study their theoretical properties, and prove that they provide the same convergence guarantees when applied to SGD training. We apply approximate tensor operations to single and multi-node training of MLP and CNN networks on MNIST, CIFAR-10 and ImageNet datasets. We demonstrate up to 66% reduction in the amount of computations and communication, and up to 1.37x faster training time while maintaining negligible or no impact on the final test accuracy.

AB - We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i.e., matrix multiplications and convolutions. We introduce new sampling techniques, study their theoretical properties, and prove that they provide the same convergence guarantees when applied to SGD training. We apply approximate tensor operations to single and multi-node training of MLP and CNN networks on MNIST, CIFAR-10 and ImageNet datasets. We demonstrate up to 66% reduction in the amount of computations and communication, and up to 1.37x faster training time while maintaining negligible or no impact on the final test accuracy.

UR - http://www.scopus.com/inward/record.url?scp=85131866071&partnerID=8YFLogxK

M3 - منشور من مؤتمر

T3 - Advances in Neural Information Processing Systems

SP - 27877

EP - 27889

BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

A2 - Ranzato, Marc'Aurelio

A2 - Beygelzimer, Alina

A2 - Dauphin, Yann

A2 - Liang, Percy S.

A2 - Wortman Vaughan, Jenn

T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

Y2 - 6 December 2021 through 14 December 2021

ER -

Adelman M, Levy KY, Hakimi I, Silberstein M. Faster Neural Network Training with Approximate Tensor Operations. In Ranzato MA, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, editors, Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. 2021. p. 27877-27889. (Advances in Neural Information Processing Systems).

Faster Neural Network Training with Approximate Tensor Operations

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this