Adam

Adam with bias correction, matching torch.optim.Adam defaults.

The update as implemented:

• g = grad, plus weight_decay · p when set — coupled decay, added to the gradient.
• m ← m·β₁ + g·(1−β₁), and on the first step m = g·(1−β₁) rather than a zero buffer.
• v ← v·β₂ + g²·(1−β₂), likewise.
• p ← p − lr·m̂ / (√v̂ + ε), assigned in place.

Bias correction is applied to the step size, not to m and v individually. Algebraically identical, one fewer tensor operation per parameter, and it is what torch does.

State: two tensors per parameter, so Adam costs 2× the model size in optimiser state — worth knowing before choosing it on a device with 5.75 GiB.

Construction¶

__init__¶

Update¶

See also AdamW, SGD, Optimizer