Learn · Intermediate

Machine-learning compilers: turning tensor operations into efficient device programs

A machine-learning compiler transforms a model’s tensor computations into a program suited to a particular device. It can combine operations, choose efficient memory layouts and specialize execution for known input properties. This matters because the same mathematical model can run at very different speeds depending on how its operations become machine work.

Imagine a restaurant receiving a recipe as a long list of tiny instructions. Following each instruction separately means repeatedly fetching ingredients, moving bowls and waiting for the next direction. An experienced kitchen planner groups compatible steps, keeps useful ingredients nearby and assigns tasks to suitable equipment. The recipe remains recognizable, but its execution becomes more efficient.

A model framework is a convenient place to express operations such as matrix multiplication, addition and activation functions. A device executes lower-level programs called kernels. Running every operation as an isolated kernel can create launch overhead and intermediate data transfers. A compiler looks across operations to find a more efficient execution plan.

Tianqi Chen and colleagues’ TVM paper describes an optimizing compiler approach spanning high-level computation graphs and low-level tensor programs. Martín Abadi and colleagues’ TensorFlow systems paper provides a foundational account of representing computation as a graph and placing it across heterogeneous devices. A graph-based system and an optimizing compiler are not identical, but graphs make many cross-operation transformations possible.

Capture the computation

A graph represents operations and the values flowing between them. Capturing a region of a model gives an optimizer more context than executing one framework instruction at a time. It can see that an intermediate result has only one consumer, that two operations can share a layout, or that a constant expression need not be recomputed.

Capture has boundaries. A program may depend on a Python branch, an unsupported operation or data that changes its structure. The system may compile separate regions with ordinary execution between them. Those interruptions can reduce the benefit, especially when a workload contains many small operations rather than a few large, efficient kernels.

Fuse work and reduce movement

Fusion combines compatible operations into a kernel. For example, matrix multiplication followed by a bias and an activation may avoid writing every intermediate value to device memory and reading it back. The benefit is often less data movement and fewer launches, rather than fewer mathematical operations.

Fusion is not always better. A larger kernel can require too many registers, limit parallelism or interfere with a highly tuned library implementation. The compiler must choose among imperfect alternatives. Ground Truth’s memory-bound inference lesson explains why moving data can dominate arithmetic; compiler optimization is one way to reduce that movement.

Layout is another choice. The same logical tensor can be arranged differently in memory. A layout convenient for one operation may be inconvenient for the next. An optimizer tries to reduce conversion costs across a sequence. This is why judging an isolated operation can miss the best whole-program arrangement.

Specialize without assuming too much

A compiler can generate better code when it knows dimensions, data types or a restricted range of shapes. Specialization turns those facts into optimization opportunities. A program prepared for one batch size may be especially efficient, while another size needs a different version.

Runtime guards check whether the assumptions still hold. A new shape can trigger recompilation or a fallback. Supporting flexible inputs and exploiting fixed shapes therefore pull in different directions. A benchmark using one repeated input shape may understate the costs of a service whose requests vary widely.

Compilation itself takes time. Cold-start measurements include work that a warm request may reuse. Caches can avoid repeating compatible compilations, but a high rate of new shapes can reduce their value. A complete performance report separates initial preparation, steady-state execution and behavior on unfamiliar inputs.

Read backend announcements precisely

Google’s TorchTPU repository describes integration with PyTorch’s compilation interface and a cache for XLA computations. That means a more familiar frontend does not imply the underlying compiler disappears. Framework interface, graph capture, optimization and device execution are distinct layers that can change independently.

Likewise, DeepSeek’s DeepInfer release combines kernels with scheduling and distributed execution. A compiler can improve a local computation without solving routing, persistent cache management or service reliability. Ground Truth’s Amdahl’s law lesson explains why a faster compiled region may produce only a modest reduction in total waiting time.

Correctness remains part of the comparison. Operation reordering can change floating-point rounding, and some optimizations deliberately change precision. Evaluate outputs at the level the application needs, then measure representative shapes, concurrency and warmup. A compiler’s value is an efficient implementation of the required computation under real workload conditions, rather than an impressive speedup detached from the program that users actually run.

Ground Truth. (2026, October 21). Machine-learning compilers: turning tensor operations into efficient device programs. Ground Truth. https://groundtruth.day/learn/machine-learning-compilers-graphs-fusion-and-specialization.html

@misc{groundtruth:machine-learning-compilers-graphs-fusion-and-specialization,
  title  = {Machine-learning compilers: turning tensor operations into efficient device programs},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/machine-learning-compilers-graphs-fusion-and-specialization.html}
}

Topics: fundamentals · compilers · inference · systems · optimization