Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training

Daniel Soudry, Dotan Di Castro, Asaf Gal, Avinoam Kolodny, Shahar Kvatinsky

Research output: Contribution to journalArticlepeer-review

Abstract

Learning in multilayer neural networks (MNNs) relies on continuous updating of large matrices of synaptic weights by local rules. Such locality can be exploited for massive parallelism when implementing MNNs in hardware. However, these update rules require a multiply and accumulate operation for each synaptic weight, which is challenging to implement compactly using CMOS. In this paper, a method for performing these update operations simultaneously (incremental outer products) using memristor-based arrays is proposed. The method is based on the fact that, approximately, given a voltage pulse, the conductivity of a memristor will increment proportionally to the pulse duration multiplied by the pulse magnitude if the increment is sufficiently small. The proposed method uses a synaptic circuit composed of a small number of components per synapse: one memristor and two CMOS transistors. This circuit is expected to consume between 2% and 8% of the area and static power of previous CMOS-only hardware alternatives. Such a circuit can compactly implement hardware MNNs trainable by scalable algorithms based on online gradient descent (e.g., backpropagation). The utility and robustness of the proposed memristor-based circuit are demonstrated on standard supervised learning tasks.

Original languageEnglish
Article number7010034
Pages (from-to)2408-2421
Number of pages14
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume26
Issue number10
DOIs
StatePublished - 1 Oct 2015

Keywords

  • Backpropagation
  • hardware
  • memristive systems
  • memristor
  • multilayer neural networks (MNNs)
  • stochastic gradient descent
  • synapse

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training'. Together they form a unique fingerprint.

Cite this