News · 2026-07-24

torchwright builds working transformer weights from Python - no training involved

A project called torchwright constructs concrete weights for a Phi-3-shaped decoder-only transformer directly from a typed computation graph assembled in Python. The resulting models load through ordinary Hugging Face APIs without custom model code. No gradients are computed and no training occurs at any point - the weights are built, not learned.

Key facts

• The core claim: transformer weights constructed from a computation graph, loading through standard Hugging Face APIs with zero gradient training.
• When: posted publicly on July 24, 2026.
• Who: the author, writing as physicsrob, with a full write-up at ood.dev.
• Primary source: the torchwright repository and the author's introduction.

The mechanism is the fun part. torchwright treats the residual stream - the running vector of numbers that flows through a transformer from layer to layer - as allocated scratch memory. Graph values occupy specific columns of that vector. New values write into cleared columns. Dead values get cancelled by adding their own negation. Linear nodes, attention nodes and nonlinear nodes are then scheduled into layers. RMSNorm is deliberately neutralised so it does not interfere, position handling is rebuilt around rotary embeddings, and SwiGLU serves as the nonlinear building block.

If that sounds like writing assembly for a very strange processor, that is roughly right. The residual stream becomes a register file, the layers become instruction slots, and the compiler's job is allocation and scheduling. What comes out is a genuine construction of vanilla transformer weights - not a custom interpreter wearing a transformer costume. You can load it, run it, and inspect it with the same tools you would use on any model.

The asterisk matters and the author is upfront about it. "Ordinary Python" here means Python is the language you use to wire supported operations into a graph. It does not compile arbitrary host control flow, arbitrary PyTorch graphs, or unconstrained algorithms. The documented examples are small symbolic programs: addition, binary increment, Caesar shifting, sorting digits, Fibonacci, range printing, calculators. One of the calculators is explicitly memorisation rather than computation. This is not a route to compiling your codebase into a model.

Loops have exactly two escape hatches, and the second one is where this gets conceptually interesting. Loops with a compile-time bound are unrolled into extra transformer depth - more iterations means a deeper network. Genuine runtime iteration has to go through autoregressive generation, where the tokens the model emits become the external scratchpad that the next forward pass reads. The fixed key-value cache bounds that tape.

That trade - depth for generated tokens - is not an implementation quirk. It is a concrete instance of a formal result: The Expressive Power of Transformers with Chain of Thought established that decoding steps materially expand what a decoder-only transformer can compute, while a fixed-depth model answering immediately is far more limited. Most people meet that idea as a prompting technique. torchwright lets you see it as an architectural fact, because you can build the same program both ways and watch one of them run out of depth. It is the clearest hands-on demonstration of why chain-of-thought reasoning is not merely a stylistic trick.

The limits are documented and real: fp32 only, constrained hidden widths under its RMSNorm construction, a static exported context length, and approximation error. That last one deserves emphasis. The project measures per-operation approximation error over declared ranges and ships assertions plus a node-level probe, but it explicitly says per-operation error bounds do not automatically compose through a whole graph. Building exact arithmetic out of continuous matrix operations is inherently approximate, and the approximations stack in ways that are hard to bound analytically.

This is also not unprecedented, and pretending otherwise would undersell the lineage rather than the work. Thinking Like Transformers introduced RASP, a programming abstraction for what transformers can express. DeepMind's Tracr then compiled restricted RASP expression graphs directly into decoder-only transformer weights, explicitly as a laboratory for interpretability research - if you know exactly what a model computes because you built it, you have ground truth for testing whether interpretability methods recover the right answer. torchwright's delta is packaging a related idea in a standard-looking Phi-3 and Hugging Face container, with normalisation and rotary encoding handled, and Python rather than a bespoke language as the construction surface.

That packaging is the practical value. Mechanistic interpretability work needs models whose internals are known by construction, and it needs them to run in the same tooling as real models so the same probes apply. A compiled transformer that loads like any other checkpoint is a better test fixture than one that requires a special runtime.

The honest caveat: everything above is verified from the project's own documentation, and independent execution is pending. Nobody outside the author has yet cloned it, built a model and confirmed the outputs. The Reddit post that surfaced it carried no comments or votes when captured, so there is no community or expert verdict either - only an early signal that people who care about this class of work noticed. Take the mechanism as documented and the demonstration as unreplicated.

Primary source, verified: read the paper →

Ground Truth. (2026, July 24). torchwright builds working transformer weights from Python - no training involved. Ground Truth. https://groundtruth.day/news/torchwright-compiles-python-to-transformer-weights.html

@misc{groundtruth:torchwright-compiles-python-to-transformer-weights,
  title  = {torchwright builds working transformer weights from Python - no training involved},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/torchwright-compiles-python-to-transformer-weights.html}
}

Topics: interpretability · transformers · open-source · research-tools · compilers