Loading Events

« All Events

  • This event has passed.

Xiaowu Dai, UC Berkeley: Towards theoretical understanding of large batch training in stochastic gradient descent

September 10 @ 11:00 am - 12:30 pm

Stochastic gradient descent (SGD) is almost ubiquitously used in training non-convex optimization tasks. Recently, a hypothesis by Keskar et al. (2017) that large batch SGD tends to converge to sharp minima has received increasing attention. We justify this hypothesis by providing new properties of SGD in both finite-time and asymptotic regimes, using tools from Partial Differential Equations. In particular, we give an explicit escaping time of SGD from a local minimum in the finite-time regime.  We prove that SGD tends to converge to flatter minima in the asymptotic regime (although it may take exponential time to converge) regardless of the batch size. We also find that SGD with a larger learning rate to batch size ratio tends to converge to a flat minimum faster. However, its generalization performance could be worse than the SGD with a smaller learning rate to batch size ratio.

Related paper
Related paper

Details

Date:
September 10
Time:
11:00 am - 12:30 pm
Event Category:

Venue

1011 Evans Hall
1011 Evans Hall
Berkeley, CA 94720 United States