CMSA Colloquium: Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks


View Calendar
November 16, 2022 12:30 pm - 1:30 pm
CMSA, 20 Garden St, G10
Address: 20 Garden Street, Cambridge, MA 02138

Hidenori Tanaka - Harvard University

In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle, but the role of symmetry breaking is not well understood. Here, we develop a Lagrangian formulation to study the geometry of learning dynamics in neural networks and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. Then, we generalize Noether’s theorem known in physics to describe a unique symmetry breaking mechanism in learning and derive the resulting motion of the Noether charge: Noether’s Learning Dynamics (NLD). Finally, we apply NLD to neural networks with normalization layers and discuss practical insights. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.