Abstract
Existing depth separation results for constant-depth networks essentially show that certain radial functions in Rd, which can be easily approximated with depth 3 networks, cannot be approximated by depth 2 networks, even up to constant accuracy, unless their size is exponential in d. However, the functions used to demonstrate this are rapidly oscillating, with a Lipschitz parameter scaling polynomially with the dimension d (or equivalently, by scaling the function, the hardness result applies to O(1) -Lipschitz functions only when the target accuracy ϵ is at most poly (1 / d)). In this paper, we study whether such depth separations might still hold in the natural setting of O(1) -Lipschitz radial functions, when ϵ does not scale with d. Perhaps surprisingly, we show that the answer is negative: In contrast with the intuition suggested by previous work, it is possible to approximate O(1) -Lipschitz radial functions with depth 2, size poly (d) networks, for every constant ϵ. We complement it by showing that approximating such functions is also possible with depth 2, size poly (1 / ϵ) networks, for every constant d. Finally, we show that it is not possible to have polynomial dependence in both d, 1 / ϵ simultaneously. Overall, our results indicate that in order to show depth separations for expressing O(1) -Lipschitz functions with constant accuracy—if at all possible—one would need fundamentally different techniques than existing ones in the literature.
Original language | American English |
---|---|
Pages (from-to) | 225-257 |
Number of pages | 33 |
Journal | Constructive Approximation |
Volume | 55 |
Issue number | 1 |
Early online date | 2 Jun 2021 |
DOIs | |
State | Published - 1 Feb 2022 |
Keywords
- Approximation theory
- Deep learning
- Depth separation
- Neural network
All Science Journal Classification (ASJC) codes
- Analysis
- General Mathematics
- Computational Mathematics