IMPLICIT BIAS IN LEAKY RELU NETWORKS TRAINED ON HIGH-DIMENSIONAL DATA

The implicit biases of gradient-based optimization algorithms are conjectured to be a major factor in the success of modern deep learning.In this work, we investigate the implicit bias of gradient flow and gradient descent in two-layer fully-connected neural networks with leaky ReLU activations when the training data are nearly-orthogonal, a common property of high-dimensional data.For gradient flow, we leverage recent work on the implicit bias for homogeneous neural networks to show that asymptotically, gradient flow produces a neural network with rank at most two.Moreover, this network is an ℓ₂-max-margin solution (in parameter space), and has a linear decision boundary that corresponds to an approximate-max-margin linear predictor.For gradient descent, provided the random initialization variance is small enough, we show that a single step of gradient descent suffices to drastically reduce the rank of the network, and that the rank remains small throughout training.We provide experiments which suggest that a small initialization scale is important for finding low-rank neural networks with gradient descent.