neural-networks
Everything on Ground Truth tagged “neural-networks” — 3 items.
Catastrophic forgetting: why teaching an AI something new can erase what it knew Lesson
Catastrophic forgetting is the tendency of a neural network to lose an old skill when trained on a new one, because both skills are stored in the same weights and nothing protects the old settings.
Activation functions: the tiny nonlinearity that makes deep learning possible Lesson
An activation function is a simple nonlinear operation applied to each number flowing through a neural network, and without one, stacking a hundred layers would be mathematically identical to using a single layer. Modern transformers mostly use gated variants like SwiGLU, which Noam Shazeer showed outperform plain ReLU.
Backpropagation: how a neural network learns from its mistakes Lesson
The single algorithm behind nearly all AI training - assigning blame for an error backward through millions of dials, so each one knows which way to turn.