weight-tying

Looped transformers: reusing the same layers to think deeper Lesson

A looped transformer runs its input through the same stack of layers more than once, so it gets the processing depth of a deep network while storing the weights of a shallow one. It trades extra compute for a smaller memory footprint, and it is the trick behind small models that punch above their parameter count.