Why the starting values of a neural network determine its ultimate success or failure
In short: Before a neural network learns, it must be initialized. Choosing the right starting values for weights and biases is not merely a technical detail; it dictates the speed of convergence, prevents gradient signals from vanishing or exploding, and ultimately determines whether the final model is useful or fundamentally broken.
In deep learning, weight initialization is the critical pre-training step of assigning initial values to a network's trainable parameters. If weights are initialized to zero, the network suffers from symmetry, causing every neuron to learn identical features. To avoid this, researchers use random sampling from normal or uniform distributions. However, naive randomness is insufficient. Methods like LeCun initialization, popularized in 1998, were designed to preserve the variance of neural activations during forward passes, ensuring signals do not die out or saturate as they propagate through layers.
As architectures evolved, so did initialization strategies. Xavier Glorot and Yoshua Bengio introduced Glorot initialization, a compromise aimed at preserving both activation variance in the forward pass and gradient variance during backpropagation. When ReLU activation functions became standard, Glorot initialization performed poorly, leading Kaiming He and colleagues to propose He initialization. This method specifically accounts for the characteristics of ReLU, preventing the 'dying ReLU' problem where neurons stop contributing to the learning process entirely.
More advanced techniques address the challenges of depth and specific structures. Orthogonal initialization, proposed by Saxe et al. in 2013, uses semi-orthogonal matrices to ensure training time remains independent of network depth. Meanwhile, data-dependent approaches like Layer-Sequential Unit-Variance (LSUV) normalize layer outputs to unit variance. With the advent of residual connections in 2015—which allowed networks to exceed the roughly 20-layer limit of models like VGG-19—new strategies like Fixup and T-Fixup emerged. These methods allow for training very deep networks without traditional normalization, demonstrating that the mathematical foundation laid at the very start of training is the primary architect of a model's potential.
The importance of these methods lies in their ability to maintain stable gradient signals. Whether through scaling weights in residual branches or initializing biases to small positive values to keep ReLU gradients active, these techniques are essential for avoiding activation function saturation. By carefully managing the initial state, researchers ensure that the complex optimization process of deep learning can actually begin.
Opens the Intellect+ feed with this card first. Swipe up for the next one.
Sources
- Summary by Intellect · source: Weight initialization
To cite this card, link to https://intellectplus.co.uk/app/science-weight-initialization.
How we source, check and correct cards: Editorial standards.
General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.
Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?
Related
More in Mathematicians · More in Science · Latest · A–Z · All topics
