The power of depth for feedforward neural networks

Research output: Contribution to journalConference article

Abstract

We show that there is a simple (approximately radial) function on Rd, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension. The result holds for virtually all known activation functions, including rectified linear units, sigmoids and thresholds, and formally demonstrates that depth - even if increased by 1 - can be exponentially more valuable than width for standard feedforward neural networks. Moreover, compared to related results in the context of Boolean functions, our result requires fewer assumptions, and the proof techniques and construction are very different.

Original languageEnglish
Article number1909.00860
Number of pages22
JournalarXiv
StatePublished - 12 Dec 2015
Event29th Conference on Learning Theory, COLT 2016 - New York, United States
Duration: 23 Jun 201626 Jun 2016

All Science Journal Classification (ASJC) codes

  • Software
  • Control and Systems Engineering
  • Statistics and Probability
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'The power of depth for feedforward neural networks'. Together they form a unique fingerprint.

Cite this