Tuesday, September 10th @ 11:00-12:30 PM (1011 Evans Hall)
Towards theoretical understanding of large batch training in stochastic gradient descent
Xiaowu Dai, UC Berkeley
Stochastic gradient descent (SGD) is almost ubiquitously used in training non-convex optimization tasks. Recently, a hypothesis by Keskar et al. (2017) that large batch SGD tends to converge to sharp minima has received increasing attention. We justify this hypothesis by providing new properties of SGD in both finite-time and asymptotic regimes, using tools from Partial Differential Equations. In particular, we give an explicit escaping time of SGD from a local minimum in the finite-time regime. We prove that SGD tends to converge to flatter minima in the asymptotic regime (although it may take exponential time to converge) regardless of the batch size. We also find that SGD with a larger learning rate to batch size ratio tends to converge to a flat minimum faster. However, its generalization performance could be worse than the SGD with a smaller learning rate to batch size ratio.