Optimizer
Base class for optimisers: holds a parameter list and clears gradients.
Captures the parameter list at construction. Every optimiser here iterates that captured list, which is why model.to(device) must happen before the optimiser is built — to replaces each parameter with a new tensor, and an optimiser made first would keep updating the old ones while the model used the new.
Every step runs under no_grad and writes through assign_. Both matter: the update is a mutation of the parameters, not part of the function being differentiated, so recording it would keep step N's graph alive into step N+1; and rebinding self.params[i] instead of assigning in place would update the optimiser's view and leave the model untouched.
Construction¶
__init__¶
Interface¶
step¶
Apply one update to every parameter, in three batched passes.
Subclasses implement _plan and do not override this. The batching is
one piece of subtle ordering (see the module docstring), and four copies
of it would be four chances to get it wrong.
zero_grad¶
Clear accumulated gradients.
Gradients accumulate rather than overwrite (PyTorch's rule), so this must be called between steps unless accumulation is intended.