CMSA New Technologies in Mathematics Seminar: On the emergence of representational geometry from the statistics of natural language

---

Large language models organize their internal representations into rich geometric structures that mirror the concepts of natural language. For example, categorical concepts often map to linear directions, while words and phrases having a notion of space or time trace out winding manifolds. Remarkably, these geometries emerge across diverse settings despite differences in neural architecture and training algorithms. Further, this representational structure is key to understanding how models perform computations. What is the origin of this universal geometry? We turn to simpler models of language — classic word embedding methods — to shed light on this question. We develop a theoretical framework for the co-occurrence statistics of natural language whose predictions show striking agreement with empirical findings. Our work provides a unified explanation for prior observations of representational geometry and a rigorous foundation for further theoretical development.

Zoom link:https://harvard.zoom.us/j/95406961810  Passcode: cmsa