Adaptive Step Sizes for Stochastic Gradient Descent

Abstract

In this thesis, we first lay some theoretical groundwork before motivating and discussing the stochastic gradient descent method along with its variations. We then analyze some popular step size strategies with a focus on the stochastic Polyak step size, a step size strategy requiring very little fine-tuning of parameters. At the end of this theoretical part, we prove the convergence of stochastic gradient descent with stochastic Polyak step sizes. In the practical part, we first implement and compare the different step size strategies numerically using a small test problem to gain a better understanding about their characteristics. Finally, we use stochastic gradient descent with Polyak’s step size to solve a parameter identification problem of an ordinary diffential equation with uncertain initial conditions.