Learn · Intermediate

Weight initialization: why neural networks start with carefully scaled randomness

Weight initialization sets a neural network’s parameters before it learns from data. Carefully scaled random values let different neurons begin differently while keeping signals and gradients from becoming too large or too small. The starting point influences whether training makes useful progress, but it is not itself evidence that a model has learned anything.

A network contains adjustable weights that transform one layer’s outputs into the next layer’s inputs. Training changes those weights using backpropagation and an optimizer. Before that process begins, every weight needs a value. Choosing those values is a mathematical design decision rather than a decorative source of noise.

Why identical starts can fail

Imagine hiring ten apprentices, giving them identical tools and instructions, and ensuring they see exactly the same examples. If every correction is also identical, they can remain ten copies of the same apprentice. Random differences allow them to specialize.

In a simple hidden layer, identical weights and biases can make neurons compute the same function and receive identical updates. Starting every hidden weight at zero preserves that symmetry instead of encouraging distinct features. Random initialization breaks the tie. This does not mean every parameter must be random: zero biases and fixed initial normalization parameters can be appropriate within an otherwise asymmetric network.

The scale matters as much as the differences. Suppose each neuron sums many incoming signals. If each contribution has modest variance, adding more independent contributions increases the sum’s variance. Repeating an unsuitable scale across many layers can make signals explode or fade. Gradients traveling backward face a related problem, leaving early layers with unstable or uninformative updates.

Xavier and He initialization

Xavier Glorot and Yoshua Bengio studied this problem in their 2010 paper on training deep feedforward networks. Their initialization balances the number of incoming and outgoing connections. For a common zero-mean formulation, the weight variance is 2 divided by the sum of those connection counts. The aim is to keep the scale of information reasonably stable in both directions.

Those counts are called fan-in and fan-out. A layer taking 100 inputs and producing 100 outputs has 100 of each; the common Xavier formula gives a variance of 0.01. This is a variance, not a standard deviation. Confusing the two changes the scale substantially and defeats the purpose of the rule.

Kaiming He and colleagues adapted the analysis to rectified activations in Delving Deep into Rectifiers. A rectified linear activation clips negative values to zero, changing how signal variance propagates. A common He initialization uses variance 2 divided by fan-in. It compensates for the effect of rectification under simplifying assumptions about inputs and weights.

These rules are starting approximations, not universal guarantees. Nonlinearities, correlations, very deep residual paths and architectural details affect the calculation. Modern networks combine initialization with normalization and residual connections. Those techniques address related stability problems without making the initial scale irrelevant.

Random does not mean unreproducible

Most software draws initial values from a pseudorandom generator. Given the same generator, starting state, shapes and sequence of operations, it produces the same numbers. A seed identifies a starting state; it does not certify the source or quality of a checkpoint.

A hash of a tensor’s name can also supply a reproducible seed for that tensor. This can make each tensor independently regenerable without drawing all earlier tensors first. Changing the name changes the generated sequence. Reproduction still depends on the precise algorithm, transformations and numerical rounding, so recording only a seed is insufficient.

That distinction surfaced in the numbat tensor audit. Investigators reported that selected values in an alleged trained checkpoint exactly matched a name-derived generation recipe. The Hum’s separate check reproduced one additional tensor. The package’s index lists about 7.72 TB of weight data; a verified running-memory requirement is unstated in the reviewed evidence. Such a match is evidence about how checked values can be generated, rather than authentication of the claimed training run.

What a match can and cannot establish

Learning normally changes trainable parameters. If values advertised as learned remain exactly reproducible from a simple initialization recipe, the training claim deserves scrutiny. But legitimate training can freeze some parameters, and not every parameter type needs to change. One unchanged tensor does not prove that the rest of a network is untrained or that its files are malicious.

A stronger investigation separates three questions: what generated the starting values, which parameters training changed, and who produced the resulting files. Initialization addresses the first. Update history and experiments inform the second. Provenance requires separate evidence. Even an unchanged model can appear to improve under a noisy evaluation, as Ground Truth’s frozen-model report explains.

The practical lesson is to treat initialization as controlled starting structure. Use an architecture-appropriate scale, record the full reproducibility recipe, and distinguish deterministic beginnings from learned changes. A network needs both a trainable starting point and evidence that training produced the behavior being claimed.

Ground Truth. (2026, November 2). Weight initialization: why neural networks start with carefully scaled randomness. Ground Truth. https://groundtruth.day/learn/weight-initialization-scaled-randomness.html

@misc{groundtruth:weight-initialization-scaled-randomness,
  title  = {Weight initialization: why neural networks start with carefully scaled randomness},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/weight-initialization-scaled-randomness.html}
}

Topics: fundamentals · training · initialization · optimization · reproducibility