AdamW
Adam with decoupled weight decay, matching torch.optim.AdamW.
class AdamW : Adam— python/vkml/optim.py:318
The only difference from Adam is where the decay is applied, and it is not cosmetic.
Adam adds wd·p to the gradient, so the decay then passes through the second-moment normalisation and is scaled by 1/√v — meaning parameters with small gradients get decayed far harder than intended.
AdamW subtracts lr·wd·p from the parameter directly, leaving the adaptive step to act on the gradient alone. Everything else — the moments, the bias correction, the state cost — is inherited unchanged.
Construction¶
__init__¶
def __init__self, params, lr: float=0.001, betas: tuple[float, float]=(0.9, 0.999), eps: float=1e-08, weight_decay: float=0.01— python/vkml/optim.py:333