transformers
Someone compiled a working computer into transformer weights by hand News
A team constructed transformer weights analytically rather than training them, producing a model that runs arbitrary C programs through a WebAssembly interpreter encoded entirely in its attention layers at about 30,000 tokens per second.
Encoder, decoder, or both: the three ways to build a transformer Lesson
The same transformer block can be assembled three ways -- encoder-only, decoder-only, or encoder-decoder -- and the choice determines whether a model reads text, writes text, or converts one into the other.
Learning rate schedules and warmup Lesson
A learning rate schedule changes how big each training step is over the course of a run, and warmup is the near-universal practice of starting tiny and ramping up -- without it, large transformer training runs frequently diverge in the first few hundred steps.
In-context learning Lesson
In-context learning is a language model's ability to pick up a new task from examples placed in its prompt, without any change to its weights, and it is the reason prompting works at all.
Vision Transformers: what happens when you feed a picture to a language architecture Lesson
A Vision Transformer chops an image into a grid of small patches, treats each patch as a word, and runs the exact same Transformer machinery that powers language models over the resulting sequence. Google Research showed in 2020 that this beats purpose-built image networks once you train it on enough data, and it is why today's image, video and robot models all share one architecture.
torchwright builds working transformer weights from Python - no training involved News
A project called torchwright constructs concrete weights for a standard transformer architecture directly from a typed computation graph written in Python, producing models that load through ordinary Hugging Face APIs with no gradient training at any point.
Activation functions: the tiny nonlinearity that makes deep learning possible Lesson
An activation function is a simple nonlinear operation applied to each number flowing through a neural network, and without one, stacking a hundred layers would be mathematically identical to using a single layer. Modern transformers mostly use gated variants like SwiGLU, which Noam Shazeer showed outperform plain ReLU.
Looped transformers: reusing the same layers to think deeper Lesson
A looped transformer runs its input through the same stack of layers more than once, so it gets the processing depth of a deep network while storing the weights of a shallow one. It trades extra compute for a smaller memory footprint, and it is the trick behind small models that punch above their parameter count.
Multi-Token Prediction: Teaching Models to Look Several Words Ahead Lesson
Multi-token prediction trains a language model to predict several upcoming tokens at once instead of just the next one, giving a richer training signal and a built-in path to faster generation.
Layer normalization: the rescaling that keeps deep networks trainable Lesson
Layer normalization rescales the numbers flowing through each layer of a network to a consistent range, which stops signals from exploding or vanishing as they pass through dozens of layers -- and it is what makes training a transformer at all possible.
FlashAttention: making attention fast by respecting the memory hierarchy Lesson
FlashAttention is an exact attention algorithm that runs much faster and uses far less memory by tiling the computation to keep it in the GPU's fast on-chip memory - never writing the giant attention matrix to slow memory - which is why it became the default attention kernel in modern transformers.
The Logit Lens: Reading a Model's Guesses Before It Finishes Thinking Lesson
The logit lens is an interpretability technique that decodes a language model's partial, mid-computation representations into vocabulary words, letting researchers watch the model's best guess evolve layer by layer before it produces a final answer.
Residual Connections: The Shortcut That Made Deep Networks Possible Lesson
A residual connection is a shortcut that adds a layer's input directly to its output, so the layer only has to learn the change rather than rebuild everything from scratch — a simple trick that lets networks be hundreds of layers deep without collapsing, and the reason modern transformers can be stacked as deep as they are.
Positional encoding: how transformers know word order Lesson
Positional encoding is how a transformer knows the order of words, since its attention mechanism sees the input as an unordered set. Modern models use rotary position embeddings (RoPE), which encode a token's position by rotating its vector by an angle proportional to where it sits in the sequence.
Sparse Attention Lesson
Sparse attention lets a transformer skip most of the pairwise comparisons between tokens, so instead of every token attending to every other token, each one attends to a chosen subset -- which is what makes million-token context windows affordable.
The KV cache: why AI gets slower and hungrier the longer it talks Lesson
The hidden notebook that lets a model avoid re-reading every previous word - and the single biggest reason long context is expensive.
Speculative Decoding: How AI Types Faster Without Changing a Word Lesson
A small, fast model guesses the next few words and a big, slow model checks them all in one pass - producing the exact same output, just quicker. The trick behind a lot of modern AI speedups.
Transformers: the engine inside almost every modern AI Lesson
The neural-network design behind GPT, Claude, and nearly every modern AI model, and the one idea, attention, that made it work.
Tokenization: how an AI chops your words into pieces it can read Lesson
A language model never sees letters or words. It sees tokens, chunks of text turned into numbers. How that chopping works quietly shapes cost, context limits, multilingual fairness, and why models stumble on spelling and arithmetic.
transformer-vm Tool
Compiles C programs to WebAssembly and then into analytically constructed transformer weights, with a C++ engine that executes them inside the model at about 30,000 tokens per second.