CMSA Computer Science for Mathematicians: Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems
CMSA EVENTS
Speaker:
Vaggos Chatziafratis - Google Research NY
How can we use the theory of dynamical systems in analyzing the capabilities of neural networks? Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory. In a seminal paper, Telgarsky reveals the limitations of shallow neural networks by exploiting the oscillatory behavior of a simple triangle function and states as a tantalizing open question to characterize those functions that cannot be well-approximated by small depths.
In this work, we point to a new connection between DNNs expressivity and dynamical systems, enabling us to get trade-offs for representing functions based on the presence of a generalized notion of fixed points, called periodic points that have played a major role in chaos theory (Li-Yorke chaos and Sharkovskii's theorem). Our main results are general lower bounds for the width needed to represent periodic functions as a function of the depth, generalizing previous constructions relying on specific functions.
Based on two recent works:
with Ioannis Panageas, Sai Ganesh Nagarajan, Xiao Wang from ICLR'20 (spotlight): https://arxiv.org/abs/1912.
with Ioannis Panageas, Sai Ganesh Nagarajan from ICML'20: https://arxiv.
Zoom: https://harvard.zoom.us/j/98231541450