Brief Announcement: Gradual Learning of Deep Recurrent Neural Network

Ziv Aharoni; Gal Rattner; Haim Permuter

doi:https://doi.org/10.1007/978-3-319-94147-9_21

Brief Announcement: Gradual Learning of Deep Recurrent Neural Network

Ziv Aharoni, Gal Rattner, Haim Permuter

Department of Electrical & Computer Engineering

Ben-Gurion University of the Negev

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Deep Recurrent Neural Networks (RNNs) achieve state-of-the-art results in many sequence-to-sequence modeling tasks. However, deep RNNs are difficult to train and tend to suffer from overfitting. Motivated by the Data Processing Inequality (DPI) we formulate the multi-layered network as a Markov chain, introducing a training method that comprises training the network gradually and using layer-wise gradient clipping. In total, we have found that applying our methods combined with previously introduced regularization and optimization methods resulted in improvement to the state-of-the-art architectures operating in language modeling tasks.

Original language	American English
Title of host publication	Cyber Security Cryptography and Machine Learning - Second International Symposium, CSCML 2018, Proceedings
Editors	Itai Dinur, Shlomi Dolev, Sachin Lodha
Publisher	Springer Verlag
Pages	274-277
Number of pages	4
ISBN (Electronic)	978-3-319-94147-9
ISBN (Print)	978-3-319-94146-2
DOIs	https://doi.org/10.1007/978-3-319-94147-9_21
State	Published - 17 Jun 2018
Event	2nd International Symposium on Cyber Security Cryptography and Machine Learning, CSCML 2018 - Beer-Sheva, Israel Duration: 21 Jun 2018 → 22 Jun 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10879 LNCS

Conference

Conference	2nd International Symposium on Cyber Security Cryptography and Machine Learning, CSCML 2018
Country/Territory	Israel
City	Beer-Sheva
Period	21/06/18 → 22/06/18

Keywords

Data-processing-inequality
Machine-learning
Recurrent-neural-networks
Regularization
Training-methods

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
General Computer Science

Access to Document

https://doi.org/10.1007/978-3-319-94147-9_21

Cite this

Aharoni, Z., Rattner, G., & Permuter, H. (2018). Brief Announcement: Gradual Learning of Deep Recurrent Neural Network. In I. Dinur, S. Dolev, & S. Lodha (Eds.), Cyber Security Cryptography and Machine Learning - Second International Symposium, CSCML 2018, Proceedings (pp. 274-277). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10879 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-94147-9_21

Aharoni, Ziv ; Rattner, Gal ; Permuter, Haim. / Brief Announcement : Gradual Learning of Deep Recurrent Neural Network. Cyber Security Cryptography and Machine Learning - Second International Symposium, CSCML 2018, Proceedings. editor / Itai Dinur ; Shlomi Dolev ; Sachin Lodha. Springer Verlag, 2018. pp. 274-277 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{a344364700c74acda655bfec119afd77,

title = "Brief Announcement: Gradual Learning of Deep Recurrent Neural Network",

abstract = "Deep Recurrent Neural Networks (RNNs) achieve state-of-the-art results in many sequence-to-sequence modeling tasks. However, deep RNNs are difficult to train and tend to suffer from overfitting. Motivated by the Data Processing Inequality (DPI) we formulate the multi-layered network as a Markov chain, introducing a training method that comprises training the network gradually and using layer-wise gradient clipping. In total, we have found that applying our methods combined with previously introduced regularization and optimization methods resulted in improvement to the state-of-the-art architectures operating in language modeling tasks.",

keywords = "Data-processing-inequality, Machine-learning, Recurrent-neural-networks, Regularization, Training-methods",

author = "Ziv Aharoni and Gal Rattner and Haim Permuter",

note = "Publisher Copyright: {\textcopyright} 2018, Springer International Publishing AG, part of Springer Nature.; 2nd International Symposium on Cyber Security Cryptography and Machine Learning, CSCML 2018 ; Conference date: 21-06-2018 Through 22-06-2018",

year = "2018",

month = jun,

day = "17",

doi = "https://doi.org/10.1007/978-3-319-94147-9_21",

language = "American English",

isbn = "978-3-319-94146-2",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "274--277",

editor = "Itai Dinur and Shlomi Dolev and Sachin Lodha",

booktitle = "Cyber Security Cryptography and Machine Learning - Second International Symposium, CSCML 2018, Proceedings",

address = "Germany",

}

Aharoni, Z, Rattner, G & Permuter, H 2018, Brief Announcement: Gradual Learning of Deep Recurrent Neural Network. in I Dinur, S Dolev & S Lodha (eds), Cyber Security Cryptography and Machine Learning - Second International Symposium, CSCML 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10879 LNCS, Springer Verlag, pp. 274-277, 2nd International Symposium on Cyber Security Cryptography and Machine Learning, CSCML 2018, Beer-Sheva, Israel, 21/06/18. https://doi.org/10.1007/978-3-319-94147-9_21

Brief Announcement: Gradual Learning of Deep Recurrent Neural Network. / Aharoni, Ziv; Rattner, Gal; Permuter, Haim.
Cyber Security Cryptography and Machine Learning - Second International Symposium, CSCML 2018, Proceedings. ed. / Itai Dinur; Shlomi Dolev; Sachin Lodha. Springer Verlag, 2018. p. 274-277 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10879 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Brief Announcement

T2 - 2nd International Symposium on Cyber Security Cryptography and Machine Learning, CSCML 2018

AU - Aharoni, Ziv

AU - Rattner, Gal

AU - Permuter, Haim

PY - 2018/6/17

Y1 - 2018/6/17

N2 - Deep Recurrent Neural Networks (RNNs) achieve state-of-the-art results in many sequence-to-sequence modeling tasks. However, deep RNNs are difficult to train and tend to suffer from overfitting. Motivated by the Data Processing Inequality (DPI) we formulate the multi-layered network as a Markov chain, introducing a training method that comprises training the network gradually and using layer-wise gradient clipping. In total, we have found that applying our methods combined with previously introduced regularization and optimization methods resulted in improvement to the state-of-the-art architectures operating in language modeling tasks.

AB - Deep Recurrent Neural Networks (RNNs) achieve state-of-the-art results in many sequence-to-sequence modeling tasks. However, deep RNNs are difficult to train and tend to suffer from overfitting. Motivated by the Data Processing Inequality (DPI) we formulate the multi-layered network as a Markov chain, introducing a training method that comprises training the network gradually and using layer-wise gradient clipping. In total, we have found that applying our methods combined with previously introduced regularization and optimization methods resulted in improvement to the state-of-the-art architectures operating in language modeling tasks.

KW - Data-processing-inequality

KW - Machine-learning

KW - Recurrent-neural-networks

KW - Regularization

KW - Training-methods

UR - http://www.scopus.com/inward/record.url?scp=85049011453&partnerID=8YFLogxK

U2 - https://doi.org/10.1007/978-3-319-94147-9_21

DO - https://doi.org/10.1007/978-3-319-94147-9_21

M3 - Conference contribution

SN - 978-3-319-94146-2

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 274

EP - 277

BT - Cyber Security Cryptography and Machine Learning - Second International Symposium, CSCML 2018, Proceedings

A2 - Dinur, Itai

A2 - Dolev, Shlomi

A2 - Lodha, Sachin

PB - Springer Verlag

Y2 - 21 June 2018 through 22 June 2018

ER -

Aharoni Z, Rattner G, Permuter H. Brief Announcement: Gradual Learning of Deep Recurrent Neural Network. In Dinur I, Dolev S, Lodha S, editors, Cyber Security Cryptography and Machine Learning - Second International Symposium, CSCML 2018, Proceedings. Springer Verlag. 2018. p. 274-277. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: https://doi.org/10.1007/978-3-319-94147-9_21

Brief Announcement: Gradual Learning of Deep Recurrent Neural Network

Abstract

Publication series

Conference

Keywords

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this