Losses

5 functions — 5 documented.

cross_entropy¶

Cross-entropy between unnormalised logits and integer class targets.

Takes logits, not probabilities. The log-softmax is applied internally; passing an already-softmaxed tensor applies it twice and produces a quietly wrong, still-plausible loss.

Composed rather than kernelled, in one line of ops.cpp:

• log_softmax(logits, -1) — never forms the probability, so it cannot underflow. log(softmax(x)) reaches log(0) = -inf once the logit gap passes about 90 in float32, and that poisons every gradient in the batch.
• multiplied by a one-hot mask built from the labels, rather than gathered. That composes from operators both backends already have, so cross-entropy needed no new kernel on either.
• summed along the class axis, negated, then reduced.

Exactly one term per row survives the mask, so summing the masked row recovers that term exactly — adding zeros is exact in IEEE-754. The row sum is therefore not a source of error, which is why the tolerance is inherited from log_softmax rather than widened for a reduction over C.

Parameters

input (Tensor)

Logits, (N, C) or (C,) for a single sample.

target (Tensor)

Class indices, (N,), int64, each in [0, C).

reduction (Reduction = mean) optional

mean, sum or none.

Returns

A 0-d tensor under mean or sum; (N,) under none.

Example

copy>>> logits = vkml.tensor(np.random.rand(4, 10).astype(np.float32))
>>> target = vkml.tensor(np.array([1, 0, 4, 9], dtype=np.int64))
>>> vkml.cross_entropy(logits, target).shape
()

From the header

Softmax cross-entropy from raw logits and integer class labels, following torch.nn.functional.cross_entropy. logits is (N, C) or (C,); target is I64 holding a class index per sample.

TAKES LOGITS, NOT PROBABILITIES. Passing softmax output would apply the normalisation twice and train against a wrong objective while still producing finite, plausible numbers -- so the distinction is worth stating twice.

Built on log_softmax, which already subtracts the row maximum. The naive log(softmax(x)) underflows to -inf as soon as the model becomes confident -- softmax of a losing logit reaches 0 in fp32 around a 90-logit gap, and log(0) then poisons every gradient in the batch. The stable form costs nothing and is the only reason this trains at all.

The label is selected by multiplying against a one-hot mask rather than gathering, so this composes from operators that already exist on both backends. Exactly one term per row survives, which makes the row sum exact rather than merely well-conditioned. The mask is built by comparing class indices in F32, which is exact for any C below 2^24.

Implementation

See also log_softmax, softmax, binary_cross_entropy_with_logits, kl_div

mse_loss¶

Mean squared error between two tensors.

(input − target)², then reduced. Composed from sub, square and the reduction — no kernel of its own.

Parameters

input (Tensor)

Predictions.

target (Tensor)

Targets, the same shape as input.

reduction (Reduction = mean) optional

mean, sum or none.

Returns

A 0-d tensor under mean or sum; the elementwise shape under none.

Example

copy>>> a = vkml.tensor(np.array([1.0, 2.0], dtype=np.float32))
>>> b = vkml.tensor(np.array([1.5, 2.5], dtype=np.float32))
>>> vkml.mse_loss(a, b).item()
0.25

From the header

Mean squared error between input and target, which must broadcast together. Composed from sub/square and a reduction.

Implementation

See also huber_loss, square

huber_loss¶

Squared error near zero, absolute error beyond delta.

Quadratic inside delta and linear outside it — the two agree in value and slope at the join, which is the whole point of the loss: it is less sensitive to outliers than MSE without the gradient discontinuity of absolute error.

Composed as where(|error| < delta, 0.5·error², delta·(|error| − 0.5·delta)). Both branches of where are evaluated — that is what elementwise selection means — and both are finite here, so nothing is lost by the discarded one.

Parameters

input (Tensor)

Predictions.

target (Tensor)

Targets, the same shape as input.

reduction (Reduction = mean) optional

mean, sum or none.

delta (float = 1.0) optional

Where the quadratic region ends. Must be positive.

Returns

A 0-d tensor under mean or sum; the elementwise shape under none.

Example

copy>>> a = vkml.tensor(np.array([0.0, 5.0], dtype=np.float32))
>>> b = vkml.tensor(np.array([0.5, 0.0], dtype=np.float32))
>>> vkml.huber_loss(a, b, vkml.Reduction.none, 1.0).numpy()
array([0.125, 4.5  ], dtype=float32)

From the header

Huber loss, following torch.nn.functional.huber_loss.

Quadratic within delta of the target and linear beyond it, so a single outlier contributes a bounded gradient instead of dominating the batch the way squared error lets it. The two pieces meet with matching value and slope at |error| = delta, which is what makes it usable as a training objective.

Not torch's smooth_l1_loss: that is this divided by delta (beta there). Both exist in torch and differ by exactly that factor, which is worth knowing before comparing numbers against a reference implementation.

Implementation

See also mse_loss, where, clamp

kl_div¶

Kullback–Leibler divergence between a log-probability input and a target.

Takes log-probabilities as input, matching torch.nn.functional.kl_div. Pair it with log_softmax, not softmax.

The target may be probabilities or log-probabilities, selected by log_target, and the two branches are genuinely different computations:

• log_target=True — exp(t)·(t − input). Both sides are already logs, so there is no log(0) to guard.
• log_target=False — t·(log t − input), which is 0 at t = 0 by the limit of t log t. The arithmetic gets there as 0 · -inf = NaN instead, so the zero is selected rather than computed, with a where on t > 0. log(0) still happens and still gives -inf; it is the multiply that would poison the result, and that product is the one discarded.

Parameters

input (Tensor)

Log-probabilities.

target (Tensor)

Probabilities, or log-probabilities if log_target.

reduction (Reduction = mean) optional

mean, sum or none.

log_target (bool = False) optional

Whether target is already logged.

Returns

A 0-d tensor under mean or sum; the elementwise shape under none.

From the header

Kullback-Leibler divergence, following torch.nn.functional.kl_div.

input HOLDS LOG-PROBABILITIES and target holds probabilities, which is torch's convention and trips people every time. Pass log_softmax output, not softmax output. With log_target true, target is log-probabilities too.

Pointwise value is target * (log(target) - input), defined as 0 wherever target is 0 -- the limit of t log t as t goes to 0, which the arithmetic would otherwise produce as 0 * -inf = NaN.

NOTE ON REDUCTION: Mean averages over every element, matching torch's default, which is not the mathematical definition. The KL divergence of a batch is the SUM over classes averaged over samples, so use Sum and divide by the batch size. Reduction has no BatchMean member because it is shared with every other loss here, and a value only one of them honours would be a trap in the other three.

Implementation

See also log_softmax, cross_entropy, where

binary_cross_entropy_with_logits¶

Binary cross-entropy applied directly to logits.

Takes logits, not probabilities — the sigmoid is folded in. Doing it in one step is what makes it stable: log(sigmoid(x)) underflows for large negative x in exactly the way log(softmax(x)) does, and the fused form never builds the probability.

Parameters

input (Tensor)

Logits, any shape.

target (Tensor)

Targets in [0, 1], the same shape as input.

reduction (Reduction = mean) optional

mean, sum or none.

Returns

A 0-d tensor under mean or sum; the elementwise shape under none.

From the header

Binary cross-entropy from raw logits, following torch.nn.functional.binary_cross_entropy_with_logits. target holds probabilities in [0, 1] -- usually 0 or 1, but soft labels are allowed.

TAKES LOGITS, NOT PROBABILITIES, for the same reason cross_entropy does. There is deliberately no variant taking probabilities: computing log(p) for a confident model underflows, and torch's version of that function exists only with a clamp bolted on to hide it. Apply this to the value you would have passed to sigmoid.

Evaluated as max(x, 0) - x*y + log(1 + exp(-|x|)), which is the standard rearrangement that never evaluates exp on a positive argument: exp(-|x|) lies in (0, 1], so the logarithm's argument stays in (1, 2] whatever the logit magnitude. The naive -[y log s(x) + (1-y) log(1-s(x))] loses the losing term to underflow well before x reaches 100.

Implementation

See also sigmoid, cross_entropy