architecture
Someone compiled a working computer into transformer weights by hand News
A team constructed transformer weights analytically rather than training them, producing a model that runs arbitrary C programs through a WebAssembly interpreter encoded entirely in its attention layers at about 30,000 tokens per second.
Encoder, decoder, or both: the three ways to build a transformer Lesson
The same transformer block can be assembled three ways -- encoder-only, decoder-only, or encoder-decoder -- and the choice determines whether a model reads text, writes text, or converts one into the other.
JEPA: teaching a model to predict its own understanding Lesson
A joint-embedding predictive architecture trains a network to predict its own internal representation of a missing part of the input, rather than predicting the missing pixels or tokens themselves. Skipping the surface detail is what makes it dramatically more data-efficient than generative self-supervised learning.
A 150M model set an ARC-AGI record for cost, not score News
Pathway's BDH-CQ solves reasoning puzzles by iterating in a latent workspace instead of writing out its thinking, reaching 29.5% on public ARC-AGI-1 at seven hundredths of a cent per task and setting a record in cost efficiency rather than accuracy.
Full-Duplex Speech Models: Listening and Talking at the Same Time Lesson
A full-duplex speech model processes incoming audio while it is generating outgoing audio, which removes the turn detector that decides when you have stopped speaking and makes interruption, backchannels, and overlap possible.
Metis puts an agent's memory inside the model instead of in a database beside it News
A team spanning several Chinese universities released the first prototype of what it calls a memory foundation model - a backbone carrying a memory state that updates on every interaction through a plain forward pass, with no gradients and no external retrieval system.
Vision Transformers: what happens when you feed a picture to a language architecture Lesson
A Vision Transformer chops an image into a grid of small patches, treats each patch as a word, and runs the exact same Transformer machinery that powers language models over the resulting sequence. Google Research showed in 2020 that this beats purpose-built image networks once you train it on enough data, and it is why today's image, video and robot models all share one architecture.
Convolutional neural networks: how machines learned to see Lesson
A convolutional neural network learns small reusable filters that slide across an image, so the same edge or texture detector works anywhere in the frame - the idea that made computer vision practical and still runs inside modern image, audio and video systems.
Activation functions: the tiny nonlinearity that makes deep learning possible Lesson
An activation function is a simple nonlinear operation applied to each number flowing through a neural network, and without one, stacking a hundred layers would be mathematically identical to using a single layer. Modern transformers mostly use gated variants like SwiGLU, which Noam Shazeer showed outperform plain ReLU.
Nanbeige4.2-3B reuses one 22-layer stack twice to punch above its size News
A Chinese lab released Nanbeige4.2-3B, a small open-weight model that runs its 22 transformer layers twice in sequence to get 44 layers of depth from one set of weights, posting benchmark numbers rivaling models three times its size, though the results are vendor-reported and a widely repeated 'beats 4x its size' claim does not survive clean accounting.
Looped transformers: reusing the same layers to think deeper Lesson
A looped transformer runs its input through the same stack of layers more than once, so it gets the processing depth of a deep network while storing the weights of a shallow one. It trades extra compute for a smaller memory footprint, and it is the trick behind small models that punch above their parameter count.
Layer normalization: the rescaling that keeps deep networks trainable Lesson
Layer normalization rescales the numbers flowing through each layer of a network to a consistent range, which stops signals from exploding or vanishing as they pass through dozens of layers -- and it is what makes training a transformer at all possible.
State Space Models and Mamba Lesson
State space models are a transformer alternative that processes a sequence by carrying a fixed-size running summary forward one step at a time, giving them linear cost with length and constant memory per token -- which is why models like Mamba can handle very long inputs cheaply.
Residual Connections: The Shortcut That Made Deep Networks Possible Lesson
A residual connection is a shortcut that adds a layer's input directly to its output, so the layer only has to learn the change rather than rebuild everything from scratch — a simple trick that lets networks be hundreds of layers deep without collapsing, and the reason modern transformers can be stacked as deep as they are.
Positional encoding: how transformers know word order Lesson
Positional encoding is how a transformer knows the order of words, since its attention mechanism sees the input as an unordered set. Modern models use rotary position embeddings (RoPE), which encode a token's position by rotating its vector by an angle proportional to where it sits in the sequence.
Transformers: the engine inside almost every modern AI Lesson
The neural-network design behind GPT, Claude, and nearly every modern AI model, and the one idea, attention, that made it work.
A language model that writes by erasing, and now keeps up with the classics News
Almost every chatbot writes one word at a time, left to right. A newly released model of real size writes the way image AIs paint, refining a whole passage at once, and finally holds its own.
A language model that doesn't write left to right News
iLLaDA is an 8-billion-parameter model that generates text by refining a blurry whole rather than one word at a time, and it's catching up to the mainstream.
Mixture of Experts: The Committee Inside a Giant Model Lesson
Why the biggest AI models are not really one big brain but a large team of specialists, only a few of whom wake up for any given word -- the trick that lets a model be huge and fast at the same time.
A small but elegant idea: putting 'experts' inside the attention layer News
Grouped Query Experts brings the mixture-of-experts trick into attention, activating only half a model's query heads per token while matching the full version -- at least at small scale.
A Classic Efficiency Trick Just Moved Into a New Part of the AI News
For years, the committee-of-specialists design that keeps big models fast lived in one layer of the network. A clean new result shows it works in the attention layer too, halving some of the work for free.
What is a context window? Lesson
A model's context window is how much text it can hold in mind at once — its working memory. Bigger is useful, but a long window isn't the same as a good memory. Here's how it works and where it breaks.
Scaling laws — does bigger always mean better? Lesson
For years, AI progress ran on a simple recipe: make the model bigger, feed it more data, get a better model. That pattern is real and predictable — but it has limits and surprises. Here's what scaling laws actually say.
An openly-released text model that writes by refining, not word-by-word News
Most language models write one word after another, left to right. A new openly-released model of real size generates text the way image AIs make pictures — refining a whole draft at once.
A world model that thinks in loops instead of stacking layers News
Instead of building an ever-deeper neural network to simulate the future, a new design re-runs one small block over and over — doing comparable work with a fraction of the size.
What if a word were a rotation? A more mathematical way to build AI News
A fresh, abstract idea: treat what a model attends to not as plain lists of numbers but as geometric moves like rotations — so useful symmetries come 'for free.' Elegant and early. (A deeper, technical read.)
What are diffusion language models? Lesson
Most AI writes one word at a time and can never go back. Diffusion language models start from noise and clarify it iteratively — and some versions can revise any word at any step. A growing alternative to the standard left-to-right approach.
Nanbeige4.2-3B Tool
An Apache-2.0 4B model that reuses one 22-layer transformer stack twice for 44 layers of depth from a single set of weights, shipping BF16 weights with SGLang, vLLM, llama.cpp, and Ollama paths for local use.