`PositionalEncoding`

Fixed sinusoidal position signal, added to an embedded sequence.

class PositionalEncoding : Module— python/vkml/nn.py:974

Attention is permutation equivariant: without a position signal a sequence model cannot tell "dog bites man" from "man bites dog". This is the last piece that made a transformer assemblable end to end from the modules here.

The formulation is Vaswani et al. 2017 §3.5 unchanged — PE[pos, 2i] = sin(pos / 10000**(2i/d)) and PE[pos, 2i+1] = cos(...).

Sinusoidal and not learned, deliberately. Learned positions already compose: they are Embedding(max_len, d_model) indexed by position, which is what GPT does and which needs nothing new. Sinusoidal does not compose from anything — it is a closed-form table. Adding the one that cannot be built from the parts, and leaving the one that can, is the rule the backward rules follow too.

ⓘ Note

The table is a buffer, not a parameter: it never changes, carries no gradient, appears in state_dict() and moves with .to(). It is computed once at construction.

⚠ Warning

The trigonometry runs in float64 and is narrowed once. pos reaches 5000 while the smallest frequency divisor is 1, so the argument spans ten orders of magnitude and float32 loses low-order bits of it before the sine runs. Doing it in double costs nothing — it happens once — and makes the table correct to float32's last bit rather than approximately correct.

d_model must be even, so every frequency has both a sine and a cosine; an odd width raises rather than silently truncating.

Construction¶

`init`¶

def __init__self, d_model: int, max_len: int=5000, base: float=10000.0, batch_first: bool=True— python/vkml/nn.py:1019

Forward¶

`forward`¶

def forwardself, x: V.Tensor -> V.Tensor— python/vkml/nn.py:1051

Adds the first `seq_len rows of the table to x`.

Broadcasting does the rest: the table is (S, E) and the input is (B, S, E) or (S, B, E), and in both layouts the trailing two axes line up with a leading axis of extent 1.

Internals¶

`repr`¶

def __repr__self -> str— python/vkml/nn.py:1088

PositionalEncoding