News · 2026-08-26

The case against using transformers for physics

Anima Anandkumar, the Caltech professor who invented neural operators and formerly led AI research at NVIDIA, argued in a new Latent Space interview that transformers are structurally the wrong architecture for physical simulation -- not too small, but categorically mismatched. Her reasoning is arithmetic: industrial-scale simulation runs at roughly a thousand grid points in each of three dimensions plus time, which implies hundreds of billions to a trillion tokens of context. Her verdict, verbatim: "forget ever having a transformer for anything of this scale. All of the world's compute will not be enough."

Key facts

• The weather model her group built trained on about 50,000 global weather maps -- a tiny dataset by language-model standards -- and ran on a consumer-grade GPU.
• FourCastNet 3 produces 60-day forecasts in under four minutes on a single GPU by treating the Earth as a sphere rather than a rectangle.
• Ai2's ACE2 climate emulator runs roughly 1,500 simulated years per wall-clock day while conserving dry air mass and moisture.
• Published August 26, 2026 on Latent Space.

The hook. In 2021, when her group proposed learning weather prediction from data, working meteorologists told them not to bother. "A lot of weather scientists did caution us back then," she recounts. "They said no no no, this is so difficult, there have been decades of development in traditional weather forecasting."

Background. Conventional weather prediction solves the equations of fluid dynamics on a grid, step by step, on a supercomputer. It is careful, bottom-up physics, and it is enormously expensive. The claim that a neural network could match it was, reasonably, not taken seriously.

What happened. They trained it anyway. FourCastNet reached accuracy close to the traditional models while running, in her words, "tens of thousands of times faster. So what would take a big supercomputer to run can now be run and we only needed a consumer-grade GPU." The distribution decision mattered as much as the result: "we were the first to actually open source our weather model, FourCastNet, and do it permissively," she says, which let weather agencies build on it directly. Her framing of the consequence: small agencies in the global south can now reach fidelity that was previously available only to institutions with supercomputers. "It's democratizing weather modeling." We covered a related result when Google's WeatherNext called Melissa's Category 5 landfall five days out.

How it works. A transformer relates every token to every other token, which is powerful and quadratically expensive. A neural operator learns a map between whole functions -- from an initial state of the atmosphere to the atmosphere six hours later -- rather than between fixed-size arrays of numbers. Because it operates on functions, it is discretization-invariant: train at one grid resolution, evaluate at another. And because it works in the Fourier domain, it gets global connectivity at quasi-linear cost. The analogy that fits: a transformer is a room where everyone shouts at everyone else, and the noise grows with the square of the crowd. A Fourier operator is a room where everyone contributes to a handful of shared frequencies, and each person listens to the mix. Far cheaper, and for waves and fluids, far more natural.

The other structural insight is geometry. FourCastNet 3 assumes the Earth is a sphere. Earlier models flattened it into a rectangle, and Anandkumar's point is that the rectangle assumption is what blows up over long rollouts -- the failure mode is geometric mismatch, not insufficient model size.

Why it matters. The pattern generalizes past weather. Ai2's ACE and ACE2 emulators run climate at subseasonal-to-decadal scales while conserving physical quantities, at roughly 1,500 simulated years per day of wall clock. In fusion, Fourier neural operator surrogates for plasma dynamics report a six-orders-of-magnitude speedup over traditional solvers while also handling real camera data from the MAST tokamak. In chip manufacturing, the same operators are used for inverse design of lithography masks that nobody could tune by hand. And TorchLean (repo) formalizes neural networks in Lean 4 so robustness bounds can be machine-checked -- a project she is candid is still CPU-bound and does not yet scale.

Underneath all of it is a claim about data. "When it comes to the physical world and physical data, it's never going to be as plentiful as we see with language models," she says. Fifty thousand samples is nothing next to a text corpus, and it works because nature has latent structure that language does not -- a hurricane has a specific physical signature. The lesson she draws is that when data is scarce, architecture has to carry more of the load: "we have to think about the inductive biases more, we have to add in the physics constraints, cannot be just reliant on data."

The honest caveat. Some of the interview overstates what the papers claim. She describes the Ai2 work as effectively the only working AI climate emulator; the papers make the narrower claim of first-of-its-kind accuracy across variability and forced response. Her "million times faster" for the fusion digital twin is a rounded reading of the plasma paper's six-orders-of-magnitude figure. The 50,000-sample count and the consumer GPU are interview claims not stated in the paper pages. Directionally right, imprecisely stated -- treat the papers as the record. Her closing policy ask is worth repeating regardless: regulation that treats "AI" as synonymous with language models catches AI-for-science in the same net, and the two are not the same thing.

Primary source, verified: read the paper →

Ground Truth. (2026, August 26). The case against using transformers for physics. Ground Truth. https://groundtruth.day/news/the-case-against-using-transformers-for-physics.html

@misc{groundtruth:the-case-against-using-transformers-for-physics,
  title  = {The case against using transformers for physics},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/the-case-against-using-transformers-for-physics.html}
}

Topics: ai-for-science · neural-operators · weather · climate · architecture · research