Learn · Intermediate

Looped transformers: reusing the same layers to think deeper

A looped transformer runs its input through the same stack of layers more than once, so it gains the processing depth of a deep network while storing the weights of a shallow one. It is a memory-for-compute trade: you pay to run the layers again, but you do not pay to store a second set of them. This idea is why some small open-weight models, like the recently released Nanbeige4.2-3B, can post benchmark numbers that rival models several times their parameter count.

To see why it works, recall what depth buys you. A standard transformer stacks many distinct layers, each with its own weights, and passes information up through them exactly once. More layers generally means more capacity to build up abstract features: early layers catch surface patterns, deeper ones assemble meaning. But every distinct layer is more weights to store and load, which is the main cost of a large model. The question a looped transformer asks is simple: what if the layers you already have could do double duty?

So instead of building layers 1 through 44, you build layers 1 through 22, run your data through them, then feed the result back into layer 1 and run through all 22 again. You have applied 44 layers of computation using one 22-layer set of weights. The technical name for reusing the same parameters in more than one place is weight tying or parameter sharing, and the loop is just parameter sharing along the depth of the network. Think of it as reading a hard paragraph twice with the same brain, rather than needing two brains to read it once each.

The crucial thing to understand, and the point most easily missed, is that looping saves memory, not compute. Every token still runs attention and the feed-forward network on each pass, so a model that loops twice does roughly the arithmetic of a 44-layer network even though it has the footprint of a 22-layer one. That can be a great deal when memory is your bottleneck, for instance running a capable model on a laptop or a phone, but nobody should expect a looped 3-billion-parameter model to be as fast as an ordinary one. It is fixed recurrent depth, not free intelligence.

The idea has a long lineage. Google's Universal Transformer applied the same transformer block repeatedly with a mechanism to decide, per token, how many times to loop, blending recurrence into the transformer. ALBERT shared parameters across all layers to shrink BERT dramatically without a large quality loss, showing that a big model's layers are more redundant than they look. And Looped Transformers as Programmable Computers showed, more theoretically, that a looped transformer can emulate a small general-purpose computer, running iterative algorithms by going around the loop, which is a clue about why extra passes help on reasoning-flavored tasks.

There is an important design subtlety: naively reapplying the exact same layer can make things worse, not better, because the network can fall into a repetitive fixed point or wash out its own signal. Recent research finds that useful gains usually require training the model with the loop from the start and adding a scheme that keeps each pass a genuine refinement of the last, rather than bolting looping onto a model that never expected it. In other words, recurrence must be trained and evaluated as a compute-quality trade, not treated as automatic capacity. When a small looped model beats a bigger one on paper, it is fair to ask how much came from the loop and how much from a longer, richer training run, because the two are easy to confuse.

Where does this leave a practitioner? Looping is one of a family of tricks for getting more out of fewer stored weights, alongside mixture of experts, which activates only a slice of a large network per token, and quantization, which shrinks the weights themselves. They attack different bottlenecks: mixture of experts and quantization cut the memory and compute per pass, while looping spends more compute to reclaim depth from a smaller weight budget. The honest summary is that a looped transformer is a clever way to trade time for space, and that its benchmark claims deserve the same scrutiny as any other, checked at an equal compute budget before you believe the model is truly smarter rather than merely deeper for its size.

Ground Truth. (2026, July 21). Looped transformers: reusing the same layers to think deeper. Ground Truth. https://groundtruth.day/learn/looped-transformers-and-weight-tying.html

@misc{groundtruth:looped-transformers-and-weight-tying,
  title  = {Looped transformers: reusing the same layers to think deeper},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/looped-transformers-and-weight-tying.html}
}

Topics: architecture · transformers · efficiency · weight-tying · open-weight-models