CMSA Colloquium: Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks

SEMINARS, CMSA EVENTS

Speaker:

Hidenori Tanaka - Harvard University

In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle, but the role of symmetry breaking is not well understood. Here, we develop a Lagrangian formulation to study the geometry of learning dynamics in neural networks and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. Then, we generalize Noether’s theorem known in physics to describe a unique symmetry breaking mechanism in learning and derive the resulting motion of the Noether charge: Noether’s Learning Dynamics (NLD). Finally, we apply NLD to neural networks with normalization layers and discuss practical insights. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.