Learn · Intermediate

Catastrophic forgetting: why teaching an AI something new can erase what it knew

Catastrophic forgetting is what happens when a neural network learns a new task and loses an old one in the process. Because all of a network's skills live in the same shared set of weights, the gradient updates that install a new capability move numbers the old capability depended on -- and nothing in ordinary training marks those numbers as protected. The term comes from work by Michael McCloskey and Neal Cohen in 1989, and the problem has never been fully solved; it is one of the sharpest differences between how machines learn and how people do.

Why the weights are the problem

Start with what training actually does. A network's knowledge is not filed in separate drawers. It is spread across millions or billions of weights, and any single weight participates in many different behaviors at once -- a phenomenon closely related to how features are represented inside a model. When you train on new data, gradient descent asks a simple question at each step: which small change to each weight would reduce the error on this batch? It has no memory of the batches that came before and no term in its objective for preserving them.

So the update is locally correct and globally destructive. Imagine a mixing desk where every fader affects several instruments at once. You are asked to make the vocals louder, so you push the faders that help the vocals -- and the drums go quiet, because two of those faders were also carrying the drums. Nobody told you which faders the drums needed. That is catastrophic forgetting, and it is why a model fine-tuned hard on legal documents can get noticeably worse at arithmetic it used to handle.

Humans do not work this way, and the contrast is instructive. Learning to drive does not erase your ability to ride a bicycle, partly because biological memory appears to use both a fast-learning system and a slow consolidating one, and partly because you keep rehearsing old skills incidentally. A network trained on task B and never shown task A again has neither protection.

The four families of fixes

Research on this problem falls into a few recognizable strategies, and each buys something at a price.

Rehearsal. Keep some old data around and mix it into the new training. This works well and is what most practitioners actually do, but it requires storing the old data, which may be impossible for licensing, privacy or sheer volume reasons. iCaRL formalized the idea by keeping a small, carefully chosen set of exemplars per class rather than everything.

Regularization. Estimate which weights the old task depended on and make them harder to move. Elastic Weight Consolidation, from Kirkpatrick and colleagues at DeepMind, does exactly this: it measures each weight's importance to the previous task and adds a penalty for changing important weights, effectively attaching stiff springs to the faders the drums need. It is elegant and cheap, and it degrades as tasks accumulate, since everything eventually becomes important.

Distillation from your past self. Instead of preserving weights, preserve behavior. Learning without Forgetting keeps a frozen copy of the old model and trains the new one to match its outputs on the new data, so the old input-output relationship survives even as the weights change. This is the same idea as distillation, pointed at yourself.

Isolation. Give different tasks different parameters. Adapters and low-rank fine-tuning are the practical version -- freeze the base model and train a small side module, so the original capability is mathematically untouchable. This is the most reliable option available today, and it is why almost nobody fine-tunes all the weights of a large model when they can avoid it. Gradient-based routing takes the idea further, deliberately steering categories of training data into separate modules so they can later be detached, an approach Anthropic and AE Studio explored this month for controlling dangerous knowledge.

Where it shows up in 2026

Forgetting is not a historical curiosity; it is a live constraint on how models get improved. Every post-training run risks trading general ability for specialized ability, which is why labs mix broad data back into narrow fine-tunes and measure regressions across unrelated benchmarks. It limits continual learning, the goal of updating a deployed model with new information rather than retraining from scratch -- the reason most systems bolt on retrieval or external memory instead of writing new facts into weights.

It also has a subtle and underappreciated form. Recent work on on-policy distillation found that when a student learns from several specialist teachers in sequence, it converges on the behavior the teachers share and drops what only one of them knew. Forgetting there is not caused by a new task overwriting an old one, but by the training objective quietly preferring the common ground. The general lesson holds either way: in a network with shared weights, every act of learning is also an act of overwriting, and anything you want to keep has to be protected on purpose.

Ground Truth. (2026, July 29). Catastrophic forgetting: why teaching an AI something new can erase what it knew. Ground Truth. https://groundtruth.day/learn/catastrophic-forgetting.html

@misc{groundtruth:catastrophic-forgetting,
  title  = {Catastrophic forgetting: why teaching an AI something new can erase what it knew},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/catastrophic-forgetting.html}
}

Topics: training · continual-learning · fine-tuning · fundamentals · neural-networks