ONNX Explained: Moving AI Models Between Tools and Devices
Learn what an ONNX model contains, how trained models move from a framework to a phone or browser, and how runtimes, compilers, optimization and quantization affect the result. Eris and Vestra explain conversion failures and the checks that make a deployment more trustworthy. This audio was created with Boson AI's Higgs Audio -- https://www.boson.ai/higgs-audio
Listen on Spotify · Watch on YouTube · Pocket Casts
The model has to travel
Eris: <|prosody:expressive_high|>ONNX is a shared format for describing AI models so different software can run them. For example, you might train an image recognition model on one computer and then want it to recognize pictures on a phone. The handoff is useful, but <|prosody:pause|>it doesn't guarantee the phone will produce the same answers. What actually has to make that journey?
Vestra: <|prosody:expressive_high|>A trained model has two essential parts: a recipe of computations and learned numbers called weights. During training, software adjusts those numbers using examples. After training, inference means applying the resulting recipe and weights to a new input, such as a photo, to produce a prediction.
Eris: The training framework is the software toolkit used to build and train the model. PyTorch and TensorFlow are examples. Each has its own ways of describing and running calculations. A phone app or a web page may not have that toolkit, its programming environment, or implementations suited to the device's chip. You can sometimes deploy the original framework, but that can bring more software than the application needs. I'd rather not bring that whole environment along just to recognize a picture.
Vestra: A model file is a saved representation of that recipe and its numbers. To export a model is to produce such a representation for use outside the original environment. <|emotion:enthusiasm|>ONNX, short for Open Neural Network Exchange, defines an open format for the model and a shared vocabulary for its calculations. A tool that understands ONNX can read the result without needing the original training framework, provided it supports the particular model.
Eris: Microsoft and Facebook announced the project in 2017 to make models easier to move between tools. It is now a community project under LF AI and Data, with open governance and public specifications and code. But what does open promise here?
Vestra: <|prosody:expressive_high|>Here, open means people can inspect and contribute to the shared format; it doesn't mean that every model saved in it is open or that every device supports every model.
Inside the ONNX model
Eris: <|prosody:expressive_high|>The heart of an ONNX model is a graph: a map of computations and the paths along which their results travel. A point on that map is a node, and each node calls an operator, a named building-block operation with defined behavior. What actually travels along those paths?
Vestra: The values moving through the graph are usually tensors: arrays of values that can have one dimension, like a list, two, like a table, or more. A tensor's shape describes the size of each dimension; its type says what kind of values it contains, such as floating-point numbers. The file declares what inputs the graph expects and what outputs it produces, including their types and shapes, though a dimension can be unspecified.
Eris: <|emotion:enthusiasm|>Take a small prediction model. An input table goes into a matrix multiplication node, which combines it with a stored table of learned weights. An add node then adds another learned value to produce the prediction. Matrix multiplication is a rule for combining rows and columns of numbers. That example is small, but the same graph idea can describe much larger networks.
Vestra: <|emotion:enthusiasm|>I like how concrete that gets: stored numbers, named operations. Another operator is convolution: it applies learned filters across regions of an input, a useful operation for images. ONNX's main operator list in our source snapshot contains roughly two hundred named operations. A model uses whichever ones its recipe needs, not the whole list.
Eris: The graph usually stores its learned weights as constant tensors called initializers. It can also carry metadata, such as the model's author, version, and the tool that produced it. Does that mean all the weights are in the main file?
Vestra: <|prosody:expressive_high|>For a large model, tensor data can live in separate external files, with the main model pointing to their locations. You need those files too; <|prosody:pause|>the small main file alone may not contain the weights.
Eris: <|emotion:surprise|>That would be an easy piece to miss. Can the tools work out the sizes of the tensors between the inputs and outputs?
Vestra: <|emotion:enthusiasm|>Shape inference is a tool's attempt to work out the sizes of intermediate tensors from known input sizes and operator rules. For instance, adding two identically shaped tables produces another table of that shape. Knowing the sizes can help plan memory, although some sizes cannot be determined in advance. And what if the shapes differ?
Eris: <|prosody:expressive_high|>Broadcasting lets certain operators combine compatible shapes. If you add a single number to a table, the operation applies that number to each entry. It doesn't mean any two differently shaped tensors can be combined; the operator's rules determine which shapes fit.
Translating the recipe
Eris: <|prosody:expressive_high|>ONNX’s shared vocabulary has versions. An ONNX model records an IR version and an opset version. IR means intermediate representation: the rules for the model's overall structure and data types. An opset is a particular version of a set of operator definitions. What happens if the software receiving a model supports different versions?
Vestra: Those versions answer different questions. The IR tells a reader how the model is organized. The opset tells it which definitions govern the graph's operations. Operators can gain new versions as their behavior or accepted inputs change, so the model declares the set it was written against. <|prosody:expressive_high|>A runtime that doesn't support the necessary IR, operator versions, types, or custom operations may reject the model or be unable to run a part of it.
Eris: The exporter or converter is the translator from a framework's model into that shared graph. Sometimes one original operation needs several ONNX nodes to express the same result. That translation is a real piece of software, and it can make mistakes. What did researchers find when they examined that handoff?
Vestra: Purvish Jajal and colleagues studied that handoff. They surveyed 92 practitioners and examined 200 selected, closed issue reports about the PyTorch and TensorFlow converters to ONNX. In those reports, the most common symptom was a crash; <|prosody:pause|>about a third described a model that was valid-looking but behaved incorrectly. Nearly three quarters of the investigated failures were located in the stage that translates individual nodes.
Eris: <|emotion:surprise|>A file that loads can still give you the wrong answer. That's more unsettling to me than a crash. <|prosody:expressive_high|>The researchers found incompatibilities and type problems behind many crashes, while wrong models were often associated with type problems or errors in the conversion algorithms. One framework's operation may not have a one-to-one equivalent in ONNX, and details like tensor shapes and data types have to survive the mapping.
Vestra: I share that worry, but we need to keep the scale of it clear. Those percentages describe selected historical bug reports, not the chance that any new export fails. In a separate experiment, the team compared outputs from original models with converted models running in ONNX Runtime. They tested both existing models and artificially generated models designed to exercise unusual combinations. The existing models generally converted well once they reached the converters; the synthetic cases exposed more failures and differences.
Eris: <|sfx:humming|>Hmm, the comparison used a very strict numerical threshold. Crossing it identifies <|prosody:pause|>a difference, not automatically a meaningful loss for every application. The authors saw shared operation sequences in many mismatching synthetic models, but didn't directly establish that those sequences caused the errors. Their practical conclusion is to check the behavior of the model you actually intend to use.
From the graph to a device
Eris: <|prosody:expressive_high|>Once you have the ONNX model, something still has to perform its calculations. A runtime is software that loads a model and executes its operations on a device. ONNX Runtime is one example; ONNX itself is the description, not the engine. So who decides which parts run on which hardware?
Vestra: <|emotion:enthusiasm|>ONNX Runtime can divide the graph among execution providers. An execution provider is a pluggable back end that connects the runtime to a hardware-specific computing library. A provider reports which nodes or parts of the graph it can handle, and the runtime assigns work accordingly. Depending on the configured providers, some operations can fall back to the computer's general-purpose processor, the CPU, if the preferred accelerator can't handle them.
Eris: A graphics processing unit, or GPU, is a chip designed for highly parallel calculations and can accelerate suitable model operations. ONNX Runtime Web can use WebGPU, a browser interface for GPU computation, to run supported work on the user's device. Browser, hardware, operator, and provider support still matter; choosing WebGPU <|prosody:pause|>doesn't guarantee that every model or operation will run there. Does execution always have to go through a runtime?
Vestra: <|emotion:enthusiasm|>There is another route: compilation. A compiler translates a program into executable code ahead of use. The ONNX-MLIR research project takes an ONNX model and translates it through a sequence of representations into native code for a target machine. MLIR is compiler infrastructure that lets the developers work at several levels of detail.
Eris: At the first level the compiler can still reason about model operations. At a lower level it can work on loops: repeated instructions that process tensor elements. That gives it opportunities to rearrange calculations and memory use before producing a library an application can call. That's an appealing place to look for speed.
Vestra: <|prosody:expressive_high|>I'd keep that hope separate from what this paper demonstrated. The ONNX-MLIR paper was a preliminary report. Its authors demonstrated compilation of selected models, including an image classifier, on specific systems; their reference implementation wasn't optimized for a meaningful performance comparison. Compilation is an alternative way to execute a supported model, not a promise that every ONNX graph will compile or run faster.
Making the model smaller and faster
Eris: <|prosody:expressive_high|>Once the model runs, you can try to make it fit or respond faster. One approach changes the graph itself. ONNX Runtime can remove redundant steps, calculate constant-only expressions in advance, and fuse several operations into one. For example, it can fold an addition into a convolution. How much can you change before the answers change too?
Vestra: The basic rewrites are intended to preserve the computation's meaning. <|prosody:expressive_high|>But a changed calculation order can affect numerical rounding, and some optional optimizations explicitly use approximations. The runtime documentation describes an approximation in an activation function, a calculation used inside a network. Its own example found almost the same result on one language task; <|prosody:pause|>that doesn't establish the effect on your model. Compare outputs with the settings you will actually use.
Eris: Another choice changes how numbers are represented. Floating-point numbers store approximations of real values. Switching from 32-bit floating point to 16-bit, often called half precision, can reduce model size and improve speed on some GPUs. But fewer bits mean some numbers are rounded differently or cannot be represented in the same range.
Vestra: <|emotion:enthusiasm|>Mixed precision keeps selected operations at the higher precision while using the smaller representation elsewhere. ONNX's tools can try combinations against sample inputs and an acceptable difference. That is a useful search, but a sample is only as informative as the cases it covers. I'd want to know what those samples missed.
Eris: <|prosody:expressive_high|>Quantization takes a different route to coarser numbers: for example, it maps floating-point values onto a limited set of eight-bit integers, using a scale and a reference for zero. The runtime can recover approximate values for computation. This can reduce storage and sometimes speed up inference, but mapping many possible values onto fewer available values loses detail. Whether it runs faster depends on the model and hardware. When does it choose that scale?
Vestra: With static quantization, a set of representative examples called calibration data helps choose the scale of intermediate values ahead of time. With dynamic quantization, parameters for those values are calculated as the model runs, which adds work. Both methods need evaluation on the target task.
Eris: <|emotion:enthusiasm|>I'd rather keep detail where it matters most. Researchers Nikolaos Louloudakis and Ajitha Rajan explored a more selective choice in their SeQTO tool. They compared the internal results of an original model and a fully quantized version to identify sensitive layers, then tried leaving some layers at higher precision. They compared candidate sizes and predictions from four image classifiers on a laptop CPU and a phone GPU.
Vestra: In their experiments, selective versions reduced disagreement with the original model's top prediction compared with fully quantized versions, while retaining some of the size savings. The exact trade-off varied.
Eris: <|prosody:expressive_high|>That's encouraging. But did they measure whether those predictions were right, or whether they matched the original?
Vestra: Their measure was agreement with the original predictions, <|prosody:pause|>not an independent demonstration of accuracy against ground-truth labels. These were preliminary image-classification tests, and the paper did not systematically evaluate speed, so the result doesn't guarantee a faster or equally accurate model in another application.
Checking the model you deploy
Eris: <|prosody:expressive_high|>How do you check that an ONNX model made the handoff intact? Keep the original trained model as a reference, export it, and check that the ONNX model is structurally valid. Make sure the required external weight files are present, and check the declared inputs, outputs, types, shapes, IR version, and opset against the destination software.
Vestra: <|prosody:expressive_high|>Then give the original and exported models the same inputs and compare their outputs. Use representative examples and difficult cases, including the input sizes you expect to support. Decide in advance how much numerical difference is acceptable for the task, and check the actual task result too. <|prosody:pause|>A small change in a score might matter if it changes the chosen label near a decision boundary.
Eris: That's the part I'd be watching: whether the chosen label changes. Run those checks on the actual runtime, execution providers and hardware you intend to use. Test again after changing the export, runtime version, optimization settings or numerical precision. A successful check on one photo and one laptop tells you about that photo and that setup; it doesn't cover every input or every deployment.
Vestra: Measure speed and memory on the target device as well. A smaller model isn't automatically faster, and a graph optimized for one provider or processor may not suit another.
Eris: <|sfx:humming|>Hmm, suppose the outputs do diverge. Where would you look for the change?
Vestra: Compare intermediate values where tools allow it, or try disabling an optimization or leaving sensitive operations at higher precision to locate the change.
Eris: <|prosody:expressive_high|>ONNX sits between model creation and execution: it gives different tools a common model description. It doesn't itself train your model, supply the missing hardware implementation, guarantee identical answers, or decide whether the original predictions are good.
Vestra: There is a qualification to the training point. The specification does include training extensions, though many deployments use the inference path we've described.
Eris: <|emotion:determination|>The model, the translator, the engine, and the actual task <|prosody:pause|>all need their own checks.
Sources and follow
Vestra: The ONNX documentation and papers are linked with this episode.
Eris: <|prosody:expressive_high|>Follow the daily AI stories at Ground Truth, groundtruth.day, or listen to today’s separate news brief. Follow AI in Detail for more explanations.