The theory and practice of stochastic optimization has focused on stochastic gradient descent (SGD) in recent years, retaining the basic first-order stochastic nature of SGD while aiming to improve it via mechanisms such as averaging, momentum, and variance reduction. Improvement can be measured along various dimensions, however, and it has proved difficult to achieve improvements both in terms of nonasymptotic measures of convergence rate and asymptotic measures of distributional tightness. In this work, we consider first-order stochastic optimization from a general statistical point of view, motivating a specific form of recursive averaging of past stochastic gradients. The resulting algorithm, which we refer to as Recursive One-Over-T SGD (ROOT-SGD), matches the state-of-the-art convergence rate among online variance-reduced stochastic approximation methods. Moreover, under slightly stronger distributional assumptions, the rescaled last-iterate of ROOT-SGD converges to a zero-mean Gaussian distribution that achieves near-optimal covariance. This is a joint work with Wenlong Mou, Martin Wainwright, and Michael Jordan.
28 Aug 2020
11am - 12pm
Where
https://hkust.zoom.us/j/5616960008
Speakers/Performers
Dr. Junchi LI
University of California, Berkeley
Organizer(S)
Department of Mathematics
Contact/Enquiries
mathseminar@ust.hk
Payment Details
Audience
Alumni, Faculty and Staff, PG Students, UG Students
Language(s)
English
Other Events
26 Apr 2024
Seminar, Lecture, Talk
IAS / School of Science Joint Lecture - Molecular Basis of Wnt Biogenesis, Secretion and Ligand Specific Signaling
Abstract Wnt signaling is essential to regulate embryonic development and adult tissue homeostasis. Aberrant Wnt signaling is associated with cancers. The ER-resident membrane-bound O-acyltransfera...
18 Apr 2024
Seminar, Lecture, Talk
IAS / School of Science Joint Lecture - Understanding the Roles of Transposable Elements in the Human Genome
Abstract Transposable elements (TEs) have expanded the binding repertoire of many transcription factors and, through this process, have been co-opted in different transcriptional networks. In this ...