Learn · Intermediate

Expectation-maximization: learning when part of the data is hidden

Expectation-maximization is a method for fitting a statistical model when some information needed to describe the data is hidden. It alternates estimating that hidden information under the current model and updating the model using those estimates. This makes problems such as learning overlapping groups from unlabeled observations tractable without pretending that uncertain assignments are already known.

A familiar problem with a hidden ingredient

Imagine receiving a box of temperature readings collected from two rooms, with the room labels removed. One room is usually cooler, the other warmer, and their readings overlap. You want to estimate each room’s usual temperature and variability, but you do not know which room produced each measurement.

If the labels were available, you could calculate a mean and variance for each room directly. If the room distributions were known, you could calculate how likely each room was to have produced a given reading. Each missing piece would be easy if you knew the other. Expectation-maximization turns that circular dependence into an alternating procedure.

Arthur Dempster, Nan Laird and Donald Rubin formalized the general method in their 1977 paper, Maximum Likelihood from Incomplete Data via the EM Algorithm. Their framework covers more than clustering: hidden group labels are one example of incomplete information. The essential ingredient is a probabilistic model linking observed data, hidden variables and parameters.

The expectation step preserves uncertainty

Start with provisional room means, spreads and proportions. For every reading, calculate the probability that it came from each room under that model. A very cold reading might get 0.95 probability for the cooler room; a reading in the overlap might get 0.55 and 0.45.

These numbers are often called responsibilities. They sum to one for each observation. A reading does not get copied into both rooms as a full observation; its influence is divided according to the model’s current uncertainty. The existing lesson on Bayesian updating explains the probability rule used to calculate these conditional assignments.

Technically, the expectation step computes the expected complete-data log likelihood under the current conditional distribution of hidden variables. That phrase means averaging the score you would calculate if the missing labels were available, weighted by how plausible each labeling currently is. It does not usually mean filling in one guessed label and forgetting uncertainty.

The maximization step refits the model

Next, update the room parameters using those responsibilities as weights. The cooler room’s new mean is a weighted average of all readings, with more influence from observations likely to belong to it. Its variance is updated around that mean. Its mixture proportion comes from its total responsibility divided by the number of readings.

Repeat both steps. Updated room distributions change assignment probabilities, which change the next fitted distributions. The process stops when the improvement becomes small or a chosen iteration limit is reached. Gaussian mixture models are a classic case where these updates have convenient formulas.

The distinction from a hard clustering procedure is important. Assigning every reading to its single most likely room discards the overlap information. Standard expectation-maximization carries uncertainty through the update. Neither choice guarantees the inferred rooms correspond to real categories; that still depends on whether the model describes the data reasonably.

Why improvement has a precise, limited meaning

With exact expectation and maximization steps, the observed-data likelihood does not decrease. Likelihood measures how well the current model accounts for the observations. The alternating construction guarantees improvement or equality for that objective, rather than guaranteeing that every scientific interpretation becomes better.

This is a useful contrast with generic self-improvement claims. In Sakana’s MASS paper, improvement depends on workflow search, model judgments and measured task performance. Expectation-maximization’s mathematical guarantee applies to a specified statistical objective under specified updates. Calling both procedures iterative does not give an agent loop the same guarantee.

Where the guarantee stops

The method can converge to a local optimum, so initialization matters. Two different starting guesses can produce different groups. Multiple starts and held-out comparisons help reveal that dependence; the lesson on cross-validation explains why training fit alone is insufficient.

Gaussian mixtures also have a sharp failure case: a component can collapse onto one observation with near-zero variance, driving likelihood upward without producing a useful model. Practical fitting needs constraints or regularization. Even a numerically stable fit can use too many components, mistake a long tail for a separate group or force unsuitable distributions onto the observations.

Approximate expectation steps and incomplete maximization need their own analysis; the exact guarantee should not be casually transferred to arbitrary implementations. The lesson on gradient descent covers a different optimization mechanism that may be used within a harder parameter update.

Expectation-maximization is valuable because it gives uncertainty an operational role in learning. Its discipline is to specify the hidden variables, the observation model and the objective clearly. Better fit then has a concrete meaning, and the remaining question becomes whether that meaning matches the real phenomenon you wanted to understand.

Ground Truth. (2026, October 12). Expectation-maximization: learning when part of the data is hidden. Ground Truth. https://groundtruth.day/learn/expectation-maximization.html

@misc{groundtruth:expectation-maximization,
  title  = {Expectation-maximization: learning when part of the data is hidden},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/expectation-maximization.html}
}

Topics: fundamentals · unsupervised-learning · latent-variables · optimization · statistics