Learn · Intermediate

Orthogonal Procrustes: aligning embedding spaces without changing their geometry

Orthogonal Procrustes alignment finds the rotation or reflection that best matches two sets of corresponding vectors without changing either set’s internal geometry. In machine learning, it lets separately learned embedding spaces share information when their structure is similar but their coordinate systems differ. A successful fit can connect representations without retraining both models, but cannot manufacture structure that one model never learned.

Two maps, different north arrows

Imagine two accurate maps of the same town. One is printed with north pointing up; the other is rotated sideways. Streets do not need to be rebuilt to compare them. Turn one map until its landmarks match the other’s. If the maps have different distortions rather than different orientations, turning them will leave errors.

Embeddings create a similar problem. Their coordinates are learned numbers, not labels such as latitude and longitude. Two models can arrange related words or images similarly while assigning different directions to the arrangement. Directly comparing coordinate number one across models can therefore be meaningless even when their representations share useful structure.

What the fit actually solves

Suppose each row of a matrix X is one model’s vector for an example, and the corresponding row of Y is the other model’s vector for the same example. Orthogonal Procrustes chooses a matrix Q that minimizes the sum of squared mismatches between XQ and Y, subject to Q-transpose times Q equaling the identity matrix.

That constraint is the essential part. It makes Q preserve lengths and angles, so the fit cannot squash inconvenient directions or inflate others to improve the score. The general orthogonal solution allows reflection as well as rotation. If an application requires a proper rotation only, it adds a determinant constraint and adjusts the solution accordingly.

Peter H. Schönemann’s 1966 generalized solution gives a direct computational route. Form X-transpose times Y, then take its singular value decomposition, written U times Sigma times V-transpose. For the row-vector convention above, Q equals U times V-transpose. The decomposition identifies how the shared directions line up; the orthogonal factors provide the best permitted change of coordinates.

This is a fitted transformation, even though it has a closed-form solution rather than a long neural-network training loop. If the clouds are shifted, center them first. If they differ in scale, decide explicitly whether normalization is appropriate or a scale parameter belongs in a different problem. Orthogonal alignment alone performs neither translation nor arbitrary rescaling.

Why language researchers used it

Tomas Mikolov and colleagues’ 2013 cross-language work demonstrated the usefulness of learning a linear relationship between independently trained word spaces. Their broader linear-map approach helped establish the practical question: can geometric relationships learned in one language make translation easier in another?

Alexis Conneau and colleagues’ Word Translation Without Parallel Data later used an orthogonal mapping and Procrustes refinement within an unsupervised word-translation system. The full system also needs machinery for discovering correspondences and retrieving plausible translations. Procrustes does the fitting once candidate pairs exist; it does not by itself know which French word corresponds to which English word.

For a concrete supervised example, begin with a seed dictionary containing paired translations. Fit the map using those pairs. Transform an unseen source word, then search for a nearby target-language vector. Success depends on the spaces having sufficiently similar local structure and the seed pairs covering the directions that matter.

How to test the claim honestly

Measure alignment on held-out pairs, not only the landmarks used to fit it. A low training error can reflect a small or unrepresentative dictionary. Compare against simpler baselines and inspect performance across categories rather than reporting one favorable average. Our holdout-set lesson explains why the fitting and evaluation examples need separation.

Equal dimensionality is another assumption. A rectangular map between different-size spaces can preserve structure only in a restricted sense; it is not a rigid transformation of two identical-dimensional spaces. Dropping dimensions or allowing a general learned projection changes the question and the guarantees.

The method also fails gracefully when the maps are genuinely different. If one encoder tracks viewpoint and the other ignores viewpoint, a rotation cannot erase that discrepancy while preserving all distances. A shared similarity pattern does not guarantee that every feature, task or input distribution will align.

That is why representation alignment should remain distinct from model equivalence. JEPA describes a family of representation-learning objectives; Procrustes is a way of comparing or connecting the resulting spaces. A strong alignment score is evidence about geometry under a specified test, not proof of equal intelligence, factual reliability or language-generation skill.

Ground Truth. (2026, November 1). Orthogonal Procrustes: aligning embedding spaces without changing their geometry. Ground Truth. https://groundtruth.day/learn/orthogonal-procrustes-embedding-alignment.html

@misc{groundtruth:orthogonal-procrustes-embedding-alignment,
  title  = {Orthogonal Procrustes: aligning embedding spaces without changing their geometry},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/orthogonal-procrustes-embedding-alignment.html}
}

Topics: representation-learning · embeddings · linear-algebra · alignment · multilingual-models