Autograd

Reverse-mode differentiation built entirely from forward operators — and what that buys.

Every backward rule is a forward operation

d(a·b)/da is mul(grad, b) — an ordinary Mul node appended to the graph, not a call into a dedicated mul_backward kernel.

This is ggml's model and tinygrad's, and it is the reason the project needs roughly 64 kernels rather than 120. Three consequences follow for free:

the backward pass reuses the executor, the allocator and every kernel — so a bug fixed in mul is fixed in the gradient of mul;
higher-order derivatives need no new machinery, because the backward graph is an ordinary graph that can itself be differentiated;
gradient checkpointing becomes "re-emit that subgraph".

The cost is that a fused backward kernel would be perhaps 10–20% faster on some operators. That is recorded as the right thing to trade away, and individual operators can be fused later as a pure optimisation with no API change.

Four exceptions, each earned

OpKind contains no *_backward entries — with four deliberate exceptions, which appear as ordinary forward operators because they genuinely cannot be composed from the element-wise and reduction set:

Operator	Adjoint of	Why it cannot be composed
`ScatterAdd`	`index_select`	Several sources may target one destination, so contributions must accumulate.
`Col2Im`	`im2col`	Overlapping windows must sum where they overlap.
`MaxPool2dBackward`	`max_pool2d`	The gradient goes only to the argmax of each window.
`SliceBackward`	`slice`	Scatters into a zero-filled tensor of the original extent — no combination of the element-wise or reduction operators expresses that without an index kernel.

All four are scatters. That is the shape of thing this design cannot express, and naming them as forward operators rather than as backward special cases keeps the rule intact: there is one kind of node, and gradients are built from it.

How backward() walks the graph

Checks first. The root must be defined, must require grad, and must be a floating dtype — an integer root raises DTypeError rather than producing zeros.
Autograd order. The subgraph reachable from the root is ordered.
Seed. The root's gradient is the supplied seed; the no-argument form uses ones and requires a scalar, matching torch.Tensor.backward().
Reverse iteration. A node's gradient is complete only once every consumer has contributed, and consumers all appear later in the order — so walking it backwards is what makes each gradient final when it is used. Nodes with no gradient entry are unreachable from the root and skipped; leaves receive but do not propagate.
Deposit into leaves, accumulating rather than replacing.

ⓘ Note

Accumulation is PyTorch's rule and is what makes gradient accumulation across micro-batches work. It also means a training loop must call zero_grad() between steps — vkML will not do it, because it cannot tell an intentional accumulation from a forgotten reset.

Which tensors keep a gradient

Only leaves marked as parameters retain one. Intermediate .grad is dropped, matching PyTorch, where it is discarded unless explicitly retained. The gradient is realized as it is deposited, so .grad holds a value rather than an unevaluated graph — which matters because an optimiser reads every gradient immediately afterwards.

47 rules of 66 operators

19 operators have no gradient rule. Calling backward through one raises NotImplementedError naming the operator rather than silently producing a zero. They fall into four groups, and the distinction matters — only one of them is a gap:

Group	Operators	Status
Leaves and creation	`Input` `Const` `Full` `Arange` `Rand`	Nothing to propagate to — they have no inputs.
Boolean results	`Equal` `NotEqual` `Less` `LessEqual` `Greater` `GreaterEqual`	Not differentiable — the output is not a float.
Discontinuous or integral	`ArgMax` `ArgMin` `Sign`	Derivative is zero almost everywhere and undefined at the steps.
Already a backward	`MaxPool2dBackward` `SliceBackward`	Second-order through them is not implemented.
Genuinely missing	`Prod` `Erf` `Erfc`	Differentiable, simply not written.

Only the last row is a gap, and a small one: erfc exists so that gelu's gradient can be built on it, and the gradient of erfc itself has no caller yet. prod has no Vulkan kernel either, for the separate numerical reason described on its own page.

Each operator's page states which case it is in, extracted from the dispatch rather than listed by hand.

Turning recording off

no_grad() suppresses graph recording for its scope. Two places rely on it, and for the same reason: an update is a mutation of state, not part of the function being differentiated.

Optimiser steps. Recording the update would keep step N's graph alive into step N+1.
BatchNorm's running statistics. They are bookkeeping about the data seen so far; letting them onto the tape would retain every past batch's graph.

detach is the per-tensor equivalent: same values, no history.