Learn · Beginner

Linear regression and least squares: learning a numerical prediction from weighted evidence

Linear regression predicts a numerical outcome by adding weighted input features, and least squares learns those weights by minimizing squared prediction errors. It is a foundation of machine learning because it makes the relationship between data, model, loss and evaluation unusually visible. A fitted line describes an association; it does not, by itself, establish what caused the outcome.

Suppose a support team wants to predict the minutes needed to resolve a ticket. Inputs might include the number of affected accounts and whether the issue involves billing. The target is a measured duration. Unlike logistic regression, which models a category’s probability, ordinary linear regression predicts a number without restricting it to the interval between zero and one.

What the model learns

With one input, the prediction is an intercept plus a slope times the input: predicted minutes = b + w × affected accounts. The intercept is the prediction when the input is zero; the slope says how much the prediction changes for a one-unit increase. With several inputs, add one weighted term for each feature.

The model is linear in its learned coefficients. You can provide a squared input or an interaction between two inputs and still fit a linear regression, because the coefficients continue to enter as a weighted sum. The resulting curve need not be a straight line in the original input. Choosing useful features is therefore part of choosing the model.

A concrete analogy is a mixing desk. Each input has a volume control, and the model combines their contributions into one output. Training adjusts the controls using past examples. A large coefficient does not automatically mean an input is intrinsically important: its size depends on whether the input is measured in accounts, thousands of accounts, minutes or hours.

Why square the errors?

For each training example, subtract the prediction from the observed outcome. That difference is a residual. Least squares squares each residual and adds the results. Squaring prevents positive and negative errors from cancelling and makes a large error count disproportionately.

A prediction that misses by four minutes contributes sixteen squared minutes, while two predictions that each miss by two contribute eight altogether. This preference is useful when large misses deserve extra penalty. It is less suitable when a few corrupted measurements would dominate the fit. Least squares is a choice of objective, rather than a universal definition of a good prediction.

Trevor Hastie, Robert Tibshirani and Jerome Friedman explain this framework in The Elements of Statistical Learning. The historical least-squares method predates modern AI, but the structure is the same one used in larger learners: choose parameters, define a loss, and find parameters that reduce it.

How fitting works

For a modest dataset, numerical linear-algebra routines can solve the least-squares problem directly. Implementations often use matrix factorizations rather than explicitly inverting a matrix, which can be unstable. For larger datasets, gradient descent repeatedly adjusts coefficients in the direction that reduces the loss.

Léon Bottou’s work on large-scale stochastic gradient descent explains why updates from small batches can make learning practical. For ordinary least squares the objective is convex, so it does not have the many competing local minima associated with some other optimization problems. If inputs are redundant, however, different coefficient settings can produce the same predictions.

What a good fit does not prove

Training error measures agreement with examples already used to fit the model. It can look impressive while predictions on new tickets fail. Use a separate holdout set, and make that set resemble deployment. Splitting near-duplicate tickets across training and testing can make a weak model look strong.

Inspect residuals as well as an average error. A model that systematically underpredicts long tickets may be missing a nonlinear relationship. Error variability may change with predicted duration. Least squares can still produce a fit, but uncertainty estimates based on inappropriate assumptions can be misleading. Useful diagnosis asks what patterns remain unexplained.

Prediction also differs from causation. A coefficient on AI-tool adoption does not establish that introducing AI changes resolution time by that amount. Early adopters may have better documentation, easier tickets or different staffing. Today’s Apollo productivity comparison illustrates why descriptive patterns need a separate causal argument. A more elaborate fitted model does not automatically supply that argument.

Why it remains an AI foundation

Linear regression is a transparent baseline for numerical prediction and a controlled setting for studying learning itself. Ekin Akyurek and colleagues use simple regression tasks in What learning algorithm is in-context learning? to investigate whether transformers can implement learning procedures from examples in a prompt.

Its value is that the moving parts stay visible: features determine what the model can express, coefficients determine the prediction, the loss defines a mistake, and held-out evaluation tests transfer. Those distinctions remain necessary when the model has billions of parameters and no longer fits on a whiteboard.

Ground Truth. (2026, October 25). Linear regression and least squares: learning a numerical prediction from weighted evidence. Ground Truth. https://groundtruth.day/learn/linear-regression-and-least-squares.html

@misc{groundtruth:linear-regression-and-least-squares,
  title  = {Linear regression and least squares: learning a numerical prediction from weighted evidence},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/linear-regression-and-least-squares.html}
}

Topics: fundamentals · regression · supervised-learning · optimization