Motivated by the iterative nature of training neural networks, we ask: If the weights of a neural network are updated using the induced gradient on an image of a tiger, how does this update impact the prediction of the neural network at another image (say, an image of another tiger, a cat, or a plane)? To address this question, I will introduce a phenomenon termed local elasticity. Roughly speaking, our experiments show that modern deep neural networks are locally elastic in the sense that the change in prediction is likely to be most significant at another tiger and least significant at a plane, at late stages of the training process. I will illustrate some implications of local elasticity by relating it to the neural tangent kernel and improving on the generalization bound for uniform stability. Moreover, I will introduce a phenomenological model for simulating neural networks, which suggests that local elasticity may result from feature sharing between semantically related images and the hierarchical representations of high-level features. Finally, I will offer a local-elasticity-focused agenda for future research toward a theoretical foundation for deep learning. This talk will be based on the following three papers:



https://arxiv.org/abs/1910.06943



https://arxiv.org/abs/2010.11775



https://arxiv.org/abs/2010.13988

20 Nov 2020
10:30am - 11:50am
Where
https://hkust.zoom.us/j/5616960008 (Passcode: math6380p)
Speakers/Performers
Prof. Weijie SU
University of Pennsylvania
Organizer(S)
Department of Mathematics
Contact/Enquiries
Payment Details
Audience
Alumni, Faculty and staff, PG students, UG students
Language(s)
English
Other Events
26 Apr 2024
Seminar, Lecture, Talk
IAS / School of Science Joint Lecture - Molecular Basis of Wnt Biogenesis, Secretion and Ligand Specific Signaling
Abstract Wnt signaling is essential to regulate embryonic development and adult tissue homeostasis. Aberrant Wnt signaling is associated with cancers. The ER-resident membrane-bound O-acyltransfera...
18 Apr 2024
Seminar, Lecture, Talk
IAS / School of Science Joint Lecture - Understanding the Roles of Transposable Elements in the Human Genome
Abstract Transposable elements (TEs) have expanded the binding repertoire of many transcription factors and, through this process, have been co-opted in different transcriptional networks. In this ...