Learn · Intermediate

RNNs and LSTMs: How Neural Networks Learned to Remember, Before Transformers

Recurrent neural networks, or RNNs, are neural networks built to handle sequences by processing them one step at a time while carrying a running memory called a hidden state. LSTMs are a special kind of RNN that added a system of gates to fix the memory problems of the plain version. For most of the 2010s these were how machines read text, transcribed speech, and translated languages, and understanding them is the clearest way to see exactly what problem transformers were invented to solve.

Start with why sequences are hard. A regular neural network takes a fixed-size input and produces an output in one shot. But language, audio, and time series have no fixed length and, more importantly, meaning depends on order and context: "the dog bit the man" and "the man bit the dog" use the same words. You need a network that can read things in order and remember what it has seen.

An RNN does this with a simple, powerful trick: it reuses the same small network at every step. It reads the first word, produces a hidden state (a vector summarizing "what I know so far"), then reads the second word together with that hidden state to produce a new one, and so on. The hidden state is passed forward like a note the network writes to itself and updates each step. Because the same weights are applied at every position, one modest network can process a sequence of any length. Think of reading a sentence while keeping a running mental summary, revising it with each new word.

The problem is memory. To train an RNN, the error signal has to travel backward through every step of the sequence, a process called backpropagation through time, built on ordinary backpropagation. When a sequence is long, that signal gets multiplied by the same factors over and over, and it tends to either shrink toward zero or blow up, the vanishing and exploding gradient problem. In practice this meant plain RNNs were forgetful: by the time they reached the end of a long paragraph, the influence of the first sentence had faded to nothing. They could remember a few steps back, not a few hundred.

The Long Short-Term Memory network, introduced by Sepp Hochreiter and Jurgen Schmidhuber in their 1997 paper, was the fix, and it is one of the most influential ideas in deep learning. An LSTM keeps a separate "cell state" that runs down the sequence like a conveyor belt with only gentle, controlled changes, so information can travel a long way without being scrambled. Three learned gates manage it: a forget gate decides what to erase from memory, an input gate decides what new information to write, and an output gate decides what to expose to the rest of the network. Because the cell state can pass through mostly unchanged, gradients can flow across many steps without vanishing. In plain terms, the LSTM learns when to hold on to something and when to let it go. A streamlined cousin, the Gated Recurrent Unit from Cho et al. in 2014, merged some of these gates for similar results with less machinery.

These architectures powered real breakthroughs. The sequence-to-sequence framework from Sutskever, Vinyals, and Le used one LSTM to read a sentence into a single summary vector and another to generate a translation from it, launching modern neural machine translation. LSTMs also drove early speech recognition and text generation. The lineage even continues today: state space models revisit the recurrent idea with modern math to get long-memory sequence processing that scales better.

Why did transformers take over? Two reasons. First, an RNN is fundamentally sequential, it must finish step ten before starting step eleven, which makes it slow to train on today's parallel hardware. Second, even LSTMs struggle to connect information across very long distances, because everything still has to squeeze through that one running memory. Transformers replaced the running memory with attention, letting every position look directly at every other position at once, in parallel. That solved both the speed and the long-range problem, which is why the field pivoted.

The takeaway: RNNs taught networks to process sequences by carrying a memory forward, and LSTMs made that memory durable with gates. They were the state of the art for years and remain the cleanest illustration of the core challenge in all sequence modeling, holding on to the right information over time, which is precisely the challenge every architecture since, including the transformer, is still trying to solve better.

Ground Truth. (2026, July 20). RNNs and LSTMs: How Neural Networks Learned to Remember, Before Transformers. Ground Truth. https://groundtruth.day/learn/recurrent-neural-networks-and-lstms.html

@misc{groundtruth:recurrent-neural-networks-and-lstms,
  title  = {RNNs and LSTMs: How Neural Networks Learned to Remember, Before Transformers},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/recurrent-neural-networks-and-lstms.html}
}

Topics: fundamentals · sequence-models · architectures · history