Convergence of stochastic gradient descent with momentum for non-smooth non-convex optimization

What the research examines

This paper establishes non-asymptotic O(1/epsilon^4) convergence guarantees to epsilon-stationary points for stochastic heavy-ball momentum methods optimizing weakly convex, non-smooth objective functions.

Why it may matter

Supplies the first rigorous theoretical convergence guarantees for popular deep learning optimizers operating with momentum under ReLU non-smoothness.

What to examine carefully

Constants in the convergence rate depend on the Lipschitz continuity constant of the stochastic gradient variance.

Try a small, reproducible follow-up

Simulate SGD with momentum on non-smooth Rosenbrock and Rastrigin test functions in Python and track trajectory oscillation damping.

Questions to investigate

1. How was the momentum memory term controlled without assuming global gradient Lipschitz smoothness?
2. What condition on step-size decay ensures avoidance of strict saddle points with probability one?

These are independently proposed educational investigations, not activities that the original research authors claim to have completed.

Publication & researcher information

Multi-author research may involve universities in several countries. Unverified institutions and contact addresses are not inferred from authors' names.

Primary sources