Sign Based Derivative Filtering for Stochastic Gradient Descent

Konstantin Berestizshevsky; Guy Even

doi:https://doi.org/10.1007/978-3-030-30484-3_18

Sign Based Derivative Filtering for Stochastic Gradient Descent

Konstantin Berestizshevsky, Guy Even

School of Electrical Engineering

Tel Aviv University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We study the performance of stochastic gradient descent (SGD) in deep neural network (DNN) models. We show that during a single training epoch the signs of the partial derivatives of the loss with respect to a single parameter are distributed almost uniformly over the minibatches. We propose an optimization routine, where we maintain a moving average history of the sign of each derivative. This history is used to classify new derivatives as “exploratory” if they disagree with the sign of the history. Conversely, we classify the new derivatives as “exploiting” if they agree with the sign of the history. Each derivative is weighed according to our classification, providing control over exploration and exploitation. The proposed approach leads to training a model with higher accuracy as we demonstrate through a series of experiments.

Original language	English
Title of host publication	Artificial Neural Networks and Machine Learning – ICANN 2019
Subtitle of host publication	Deep Learning - 28th International Conference on Artificial Neural Networks, Proceedings
Editors	Igor V. Tetko, Pavel Karpov, Fabian Theis, Vera Kurková
Pages	208-219
Number of pages	12
DOIs	https://doi.org/10.1007/978-3-030-30484-3_18
State	Published - 2019
Event	28th International Conference on Artificial Neural Networks, ICANN 2019 - Munich, Germany Duration: 17 Sep 2019 → 19 Sep 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11728 LNCS

Conference

Conference	28th International Conference on Artificial Neural Networks, ICANN 2019
Country/Territory	Germany
City	Munich
Period	17/09/19 → 19/09/19

Keywords

Deep learning
Gradients
Neural networks
Optimization

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
General Computer Science

Access to Document

https://doi.org/10.1007/978-3-030-30484-3_18

Cite this

Berestizshevsky, K., & Even, G. (2019). Sign Based Derivative Filtering for Stochastic Gradient Descent. In I. V. Tetko, P. Karpov, F. Theis, & V. Kurková (Eds.), Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning - 28th International Conference on Artificial Neural Networks, Proceedings (pp. 208-219). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11728 LNCS). https://doi.org/10.1007/978-3-030-30484-3_18

Berestizshevsky, Konstantin ; Even, Guy. / Sign Based Derivative Filtering for Stochastic Gradient Descent. Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning - 28th International Conference on Artificial Neural Networks, Proceedings. editor / Igor V. Tetko ; Pavel Karpov ; Fabian Theis ; Vera Kurková. 2019. pp. 208-219 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{48bf3433feaf41baa2912a5eb363faaa,

title = "Sign Based Derivative Filtering for Stochastic Gradient Descent",

abstract = "We study the performance of stochastic gradient descent (SGD) in deep neural network (DNN) models. We show that during a single training epoch the signs of the partial derivatives of the loss with respect to a single parameter are distributed almost uniformly over the minibatches. We propose an optimization routine, where we maintain a moving average history of the sign of each derivative. This history is used to classify new derivatives as “exploratory” if they disagree with the sign of the history. Conversely, we classify the new derivatives as “exploiting” if they agree with the sign of the history. Each derivative is weighed according to our classification, providing control over exploration and exploitation. The proposed approach leads to training a model with higher accuracy as we demonstrate through a series of experiments.",

keywords = "Deep learning, Gradients, Neural networks, Optimization",

author = "Konstantin Berestizshevsky and Guy Even",

note = "Publisher Copyright: {\textcopyright} 2019, Springer Nature Switzerland AG.; 28th International Conference on Artificial Neural Networks, ICANN 2019 ; Conference date: 17-09-2019 Through 19-09-2019",

year = "2019",

doi = "https://doi.org/10.1007/978-3-030-30484-3_18",

language = "الإنجليزيّة",

isbn = "9783030304836",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

pages = "208--219",

editor = "Tetko, {Igor V.} and Pavel Karpov and Fabian Theis and Vera Kurkov{\'a}",

booktitle = "Artificial Neural Networks and Machine Learning – ICANN 2019",

}

Berestizshevsky, K & Even, G 2019, Sign Based Derivative Filtering for Stochastic Gradient Descent. in IV Tetko, P Karpov, F Theis & V Kurková (eds), Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning - 28th International Conference on Artificial Neural Networks, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11728 LNCS, pp. 208-219, 28th International Conference on Artificial Neural Networks, ICANN 2019, Munich, Germany, 17/09/19. https://doi.org/10.1007/978-3-030-30484-3_18

Sign Based Derivative Filtering for Stochastic Gradient Descent. / Berestizshevsky, Konstantin; Even, Guy.
Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning - 28th International Conference on Artificial Neural Networks, Proceedings. ed. / Igor V. Tetko; Pavel Karpov; Fabian Theis; Vera Kurková. 2019. p. 208-219 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11728 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Sign Based Derivative Filtering for Stochastic Gradient Descent

AU - Berestizshevsky, Konstantin

AU - Even, Guy

PY - 2019

Y1 - 2019

N2 - We study the performance of stochastic gradient descent (SGD) in deep neural network (DNN) models. We show that during a single training epoch the signs of the partial derivatives of the loss with respect to a single parameter are distributed almost uniformly over the minibatches. We propose an optimization routine, where we maintain a moving average history of the sign of each derivative. This history is used to classify new derivatives as “exploratory” if they disagree with the sign of the history. Conversely, we classify the new derivatives as “exploiting” if they agree with the sign of the history. Each derivative is weighed according to our classification, providing control over exploration and exploitation. The proposed approach leads to training a model with higher accuracy as we demonstrate through a series of experiments.

AB - We study the performance of stochastic gradient descent (SGD) in deep neural network (DNN) models. We show that during a single training epoch the signs of the partial derivatives of the loss with respect to a single parameter are distributed almost uniformly over the minibatches. We propose an optimization routine, where we maintain a moving average history of the sign of each derivative. This history is used to classify new derivatives as “exploratory” if they disagree with the sign of the history. Conversely, we classify the new derivatives as “exploiting” if they agree with the sign of the history. Each derivative is weighed according to our classification, providing control over exploration and exploitation. The proposed approach leads to training a model with higher accuracy as we demonstrate through a series of experiments.

KW - Deep learning

KW - Gradients

KW - Neural networks

KW - Optimization

UR - http://www.scopus.com/inward/record.url?scp=85072855078&partnerID=8YFLogxK

U2 - https://doi.org/10.1007/978-3-030-30484-3_18

DO - https://doi.org/10.1007/978-3-030-30484-3_18

M3 - منشور من مؤتمر

SN - 9783030304836

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 208

EP - 219

BT - Artificial Neural Networks and Machine Learning – ICANN 2019

A2 - Tetko, Igor V.

A2 - Karpov, Pavel

A2 - Theis, Fabian

A2 - Kurková, Vera

T2 - 28th International Conference on Artificial Neural Networks, ICANN 2019

Y2 - 17 September 2019 through 19 September 2019

ER -

Berestizshevsky K, Even G. Sign Based Derivative Filtering for Stochastic Gradient Descent. In Tetko IV, Karpov P, Theis F, Kurková V, editors, Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning - 28th International Conference on Artificial Neural Networks, Proceedings. 2019. p. 208-219. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: https://doi.org/10.1007/978-3-030-30484-3_18

Sign Based Derivative Filtering for Stochastic Gradient Descent

Abstract

Publication series

Conference

Keywords

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this