News · 2026-09-10

One engineer trained a 3.8-billion-parameter model for $998

Hugo Vergnes trained a 3.848-billion-parameter language model from random initialisation for a total cloud bill of $998, and published the complete recipe. According to his writeup, dated 4 September 2026, the run consumed 65.3 billion tokens over 43 hours on eight NVIDIA B200 GPUs and produced a model that scores above GPT-2 on a standard aggregate evaluation. The interesting part is not the score. It is that the whole thing fits on a credit card.

Key facts

• Total cost $998, for 43 hours on 8 B200 GPUs
• 3.848 billion parameters trained on 65.3 billion tokens
• Published 4 September 2026 by Hugo Vergnes, working independently
• Primary source: Training a 3.8B LLM to 0.384 CORE for $998

What a thousand dollars buys

Some perspective on the number. Frontier training runs are discussed in units of hundreds of millions of dollars, and the assumption that follows is that pretraining is closed to anyone without a data centre. That assumption is correct about the frontier and increasingly wrong about everything below it.

What $998 does not buy is a competitor to anything shipping commercially. The resulting model generates competent text on geography, science and history. It is not a coding assistant, an agent, or a research tool. Comparing it to GPT-2 - a model from 2019 - sets the scale honestly.

What it does buy is the thing Vergnes says he wanted: "I wanted to see language and understanding emerge from random weights for myself, and to learn the parts you can only learn by starting from scratch." That is an educational argument, and it is a strong one. There is a category of understanding about pretraining that does not survive being read about.

The choices that made it cheap

The writeup's value is in its specificity, and the choices form a coherent picture of where efficiency now comes from.

The optimiser is a composite: Muon for the matrix-shaped parameters, AdamW for embeddings, the output head and scalars. Splitting optimisers by parameter type is a recent practice, and Vergnes names Muon as a principal source of the gains. The learning rate follows a trapezoidal schedule - warm up, hold flat, decay - rather than the cosine curve that was standard for years.

The data is NVIDIA's Nemotron-ClimbMix, a curated pretraining corpus. This is where a great deal of the leverage sits: at this budget you cannot afford to train on mediocre tokens, and using a filtered public dataset substitutes somebody else's curation work for compute you would otherwise burn. The architecture is broadly Llama-shaped, with a few current refinements including a 3:1 grouped-query attention ratio, QK normalisation, and value embeddings taking up 19% of parameters.

Finally, precision. The run uses FP8 with the vocabulary padded to a friendly size, and a fused cross-entropy implementation. Mixed-precision training is not a detail here; low precision on hardware built for it is a large part of why 43 hours was enough.

Vergnes's own stated lesson is not about any of these individually: "Good infrastructure pays for itself almost immediately." Every run is fully specified by a configuration file, so experiments are configuration changes rather than code changes. That is unglamorous engineering discipline, and he credits it with the ability to iterate at all.

Why this is worth attention

Cheap training runs are the counterweight to a story the industry tells relentlessly, which is that capability follows capital. Some of it does. But the gap between what a lab spent two years ago and what an individual can now reproduce in a weekend keeps closing, and it closes through exactly the sort of accumulated technique on display here - better optimisers, better data, lower precision - rather than through any single breakthrough.

It also sits oddly against the same day's other releases. DeepSeek's V4.1 Flash is a 510 GB download, and Cognition's SWE-2 is post-trained from a 2.8-trillion-parameter base. The distance between the top and the bottom of this field has never been larger in absolute terms, and the bottom has never been more accessible. Both things are true.

The honest caveat

This is a personal blog post, self-reported, with no independent verification of either the cost or the evaluation score. The writeup publishes no weights and no code repository, so nobody can rerun it and check - the recipe is documented but not executable, which is a meaningful limit on what can be concluded.

The cost figure also deserves an asterisk that Vergnes does not hide but readers often drop: $998 is the compute bill for the run that worked. It does not include the failed runs, the debugging, the time, or the expertise required to make those choices correctly - and the choices are what the post is actually about. "Anyone can train a model for $998" is not the claim being made. "One competent engineer who knows what they are doing can, once they know it" is closer, and the gap between those is most of the work.

Primary source, verified: read the paper →

Ground Truth. (2026, September 10). One engineer trained a 3.8-billion-parameter model for $998. Ground Truth. https://groundtruth.day/news/one-engineer-trained-a-3-8-billion-parameter-model-for-998-dollars.html

@misc{groundtruth:one-engineer-trained-a-3-8-billion-parameter-model-for-998-dollars,
  title  = {One engineer trained a 3.8-billion-parameter model for $998},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/news/one-engineer-trained-a-3-8-billion-parameter-model-for-998-dollars.html}
}

Topics: training · open-source · efficiency · cost · independent-research