Binarized neural networks

Itay Hubara; Matthieu Courbariaux; Daniel Soudry; El-yaniv Ran; Yoshua Bengio

Binarized neural networks

Itay Hubara
, Matthieu Courbariaux
, Daniel Soudry
, El-yaniv Ran
, Yoshua Bengio

Technion - Israel Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At train-time the binary weights and activations are used for computing the parameter gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs, we conducted two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. We also report our preliminary results on the challenging ImageNet dataset. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.

Original language	English
Title of host publication	30th Annual Conference on Neural Information Processing Systems, NIPS 2016
Pages	4114-4122
Number of pages	9
State	Published - 2016
Event	30th Annual Conference on Neural Information Processing Systems, NIPS 2016 - Barcelona, Spain Duration: 5 Dec 2016 → 10 Dec 2016

Conference

Conference	30th Annual Conference on Neural Information Processing Systems, NIPS 2016
Country/Territory	Spain
City	Barcelona
Period	5/12/16 → 10/12/16

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Access to Document

https://proceedings.neurips.cc/paper/2016/file/d8330f857a17c53d217014ee776bfd50-Paper.pdf

Cite this

@inproceedings{a7740c10683e46db983a92381b261dfd,

title = "Binarized neural networks",

abstract = "We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At train-time the binary weights and activations are used for computing the parameter gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs, we conducted two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. We also report our preliminary results on the challenging ImageNet dataset. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.",

author = "Itay Hubara and Matthieu Courbariaux and Daniel Soudry and El-yaniv Ran and Yoshua Bengio",

note = "Funding Information: We would like to express our appreciation to Elad Hoffer, for his technical assistance and constructive comments. We thank our fellow MILA lab members who took the time to read the article and give us some feedback. We thank the developers of Torch, Collobert et al. (2011) a Lua based environment, and Theano (Bergstra et al., 2010; Bastien et al., 2012), a Python library which allowed us to easily develop a fast and optimized code for GPU. We also thank the developers of Pylearn2 (Goodfellow et al., 2013) and Lasagne (Dieleman et al., 2015), two Deep Learning libraries built on the top of Theano. We thank Yuxin Wu for helping us compare our GPU kernels with cuBLAS. We are also grateful for funding from NSERC, the Canada Research Chairs, Compute Canada, and CIFAR. We are also grateful for funding from CIFAR, NSERC, IBM, Samsung. This research was also supported by The Israel Science Foundation (grant No. 1890/14). Publisher Copyright: {\textcopyright} 2016 NIPS Foundation - All Rights Reserved.; 30th Annual Conference on Neural Information Processing Systems, NIPS 2016 ; Conference date: 05-12-2016 Through 10-12-2016",

year = "2016",

language = "English",

pages = "4114--4122",

booktitle = "30th Annual Conference on Neural Information Processing Systems, NIPS 2016",

}

Hubara, I, Courbariaux, M, Soudry, D, Ran, E & Bengio, Y 2016, Binarized neural networks. in 30th Annual Conference on Neural Information Processing Systems, NIPS 2016. pp. 4114-4122, 30th Annual Conference on Neural Information Processing Systems, NIPS 2016, Barcelona, Spain, 5/12/16. <https://proceedings.neurips.cc/paper/2016/file/d8330f857a17c53d217014ee776bfd50-Paper.pdf>

TY - GEN

T1 - Binarized neural networks

AU - Hubara, Itay

AU - Courbariaux, Matthieu

AU - Soudry, Daniel

AU - Ran, El-yaniv

AU - Bengio, Yoshua

N1 - Funding Information: We would like to express our appreciation to Elad Hoffer, for his technical assistance and constructive comments. We thank our fellow MILA lab members who took the time to read the article and give us some feedback. We thank the developers of Torch, Collobert et al. (2011) a Lua based environment, and Theano (Bergstra et al., 2010; Bastien et al., 2012), a Python library which allowed us to easily develop a fast and optimized code for GPU. We also thank the developers of Pylearn2 (Goodfellow et al., 2013) and Lasagne (Dieleman et al., 2015), two Deep Learning libraries built on the top of Theano. We thank Yuxin Wu for helping us compare our GPU kernels with cuBLAS. We are also grateful for funding from NSERC, the Canada Research Chairs, Compute Canada, and CIFAR. We are also grateful for funding from CIFAR, NSERC, IBM, Samsung. This research was also supported by The Israel Science Foundation (grant No. 1890/14). Publisher Copyright: © 2016 NIPS Foundation - All Rights Reserved.

PY - 2016

Y1 - 2016

N2 - We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At train-time the binary weights and activations are used for computing the parameter gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs, we conducted two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. We also report our preliminary results on the challenging ImageNet dataset. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.

AB - We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At train-time the binary weights and activations are used for computing the parameter gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs, we conducted two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. We also report our preliminary results on the challenging ImageNet dataset. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.

UR - https://www.scopus.com/pages/publications/85013626529

M3 - Conference contribution

SP - 4114

EP - 4122

BT - 30th Annual Conference on Neural Information Processing Systems, NIPS 2016

T2 - 30th Annual Conference on Neural Information Processing Systems, NIPS 2016

Y2 - 5 December 2016 through 10 December 2016

ER -

Binarized neural networks

Abstract

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this