SGD
Stochastic gradient descent, with optional momentum and Nesterov look-ahead.
class SGD : Optimizer— python/vkml/optim.py:146
The update as implemented, in order:
g = grad, plusweight_decay · pwhen weight decay is set.- With momentum:
v ← v·momentum + g, and on the first stepv = grather than a zero-initialised buffer. - Classical momentum then steps along the buffer:
g ← v. - Nesterov steps along the buffer and then one more momentum-step further —
g ← g + v·momentum— which is the look-ahead. It uses the current gradient again rather than replacing it, and thatgis the gradient after weight decay, which is what torch feeds in too. p ← p − lr·g, assigned in place.
State: one velocity tensor per parameter, and only when momentum is non-zero — so plain SGD costs no extra memory.
Construction¶
__init__¶
def __init__self, params, lr: float=0.01, momentum: float=0.0, weight_decay: float=0.0, nesterov: bool=False— python/vkml/optim.py:155