News · 2026-07-23

A Huawei-chip training report shows what leaving CUDA actually costs

A technical report published July 22 documents full-parameter post-training of a DeepSeek-V4-family model entirely on Huawei Ascend hardware -- and its most valuable content is the list of things that broke. arXiv:2607.20145, SLAI T-Rex, with a companion repository, describes rebuilding distributed layout, rescheduling communication, and hand-writing replacement compute kernels. That work list, not the hardware brand, is the actual measure of what independence from Nvidia's software stack costs.

Key facts

• What: full-parameter post-training -- continued pre-training plus supervised fine-tuning -- of a DeepSeek-V4-family mixture-of-experts model on Ascend CloudMatrix384 and Ascend 910C NPUs.
• The performance claim: roughly one-third of theoretical model compute, described as nearly tripling the team's stated open-source Ascend baseline. Self-reported.
• When: submitted July 22, 2026; a technical report, not peer-reviewed.
• Primary source: the paper and the repository.

The scope correction comes first, because it is the difference between an interesting engineering report and a headline that is not true. This is post-training from an existing checkpoint, not pre-training from scratch. The public workflow converts 8-bit Hugging Face weights to 16-bit, then into the MindSpeed/Megatron-Core format, trains, and exports again. Nothing here shows DeepSeek-V4 was originally built on Ascend.

Nor is "no Nvidia chips" verified. The sources establish Ascend as the training runtime for this run. They do not provide an auditable hardware inventory for every component of the project, and the data-construction toolkit can call a generic external model endpoint. The safe claim is that an Ascend post-training run is reported. The unsafe claim is that the entire project used zero Nvidia.

With that settled, the substance is genuinely worth reading, because it makes concrete something usually discussed in the abstract. Three things had to be rebuilt.

Distributed layout. The team used pipeline parallelism plus very wide expert parallelism, apparently keeping tensor parallelism at one. Spreading expert computation broadly across devices changes which data transfers dominate -- the all-to-all shuffles that route tokens to experts, and the handoffs between pipeline stages. This is the layer our lesson on distributed training parallelism covers, and it is not portable: the right split depends on the interconnect topology of the specific machine.

Communication scheduling. Rather than treating the collective operations as blocking gaps where the chips sit idle, the report describes orchestrating computation and communication together, including coordination between CPU and NPU. Huawei's HCCL library supplies the primitives -- AllReduce, AllGather, ReduceScatter, AlltoAll, point-to-point -- with algorithms that adapt to topology and message size. Having the primitives is not the same as having them scheduled well.

Kernel replacement. This is the sovereignty point in one sentence. The repository's AscendC workflow, called AuraKernel, targets sparse attention, RMS normalization, lightning-indexer gradients, rotary positional encoding, limited SwiGLU, and related chains. These are the operations that consume nearly all the training time, and on Nvidia hardware they exist as decades of accumulated hand-tuned CUDA. Moving to another vendor means someone rewrites them, by hand, and tunes the memory movement schedule for a different chip.

The analogy that fits: switching chip vendors is often described as swapping engines in a car. It is closer to swapping engines and then discovering the transmission, driveshaft, and fuel injection were all custom-machined for the old block.

The team reports reaching roughly one-third of theoretical model compute utilization, nearly tripling their stated open-source Ascend baseline. That is a substantial internal improvement -- and it is self-reported, with the custom kernels and performance traces not public.

The model result is much narrower than the infrastructure story. SLAI specializes DeepSeek-V4-Flash for Operations Research, generating parameterized optimization instances, checking them with a solver, rendering them as business problems, and reconstructing executable formulations (pipeline code). It reports beating selected comparison models on four operations-research benchmarks -- a self-reported task evaluation, not a general ranking.

There is also a telling detail in Huawei's own materials: MindSpeed-LLM lists DeepSeekV4-Flash support as a "Prototype", its label for features that are not fully validated. The vendor is more cautious about this configuration than the report's framing suggests.

Reception is early-stage curiosity rather than consensus. The paper ranked first on Hugging Face Daily Papers with 45 upvotes on the day, its only visible comment coming from the submitter, while the GitHub repository had under 20 stars and no issues or pull requests when checked.

The honest caveat is about reproducibility. The repository explicitly withholds the production continued-pre-training engine, private corpora, solver logs, generated documents, custom operators, cluster configuration, logs, and proprietary evaluation artifacts. That makes this a credible systems report and an unreproducible one. No independent replication, benchmark rebuttal, or expert critique has appeared.

Primary source, verified: read the paper → (arXiv 2607.20145)

Ground Truth. (2026, July 23). A Huawei-chip training report shows what leaving CUDA actually costs. Ground Truth. https://groundtruth.day/news/ascend-post-training-shows-the-cuda-exit-bill.html

@misc{groundtruth:ascend-post-training-shows-the-cuda-exit-bill,
  title  = {A Huawei-chip training report shows what leaving CUDA actually costs},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/ascend-post-training-shows-the-cuda-exit-bill.html}
}

Topics: hardware · china · training · open-weight-models · research