`TransformerEncoderLayer`

Self-attention followed by a feed-forward block, each residual.

class TransformerEncoderLayer : Module— python/vkml/nn.py:1093

Parameter names match torch.nn.TransformerEncoderLayer — self_attn, linear1, linear2, norm1, norm2 — so a state_dict loads unchanged and the comparison is against torch's own layer.

norm_first selects pre- or post-normalisation. Post (torch's default) is the original formulation; pre is what deep stacks use, because normalising inside the residual branch keeps the gradient path to the input clean.

ⓘ Note

activation accepts "relu" or "gelu"; anything else raises at construction rather than at the first forward pass.

Construction¶

`init`¶

def __init__self, d_model: int, nhead: int, dim_feedforward: int=2048, dropout: float=0.1, activation: str='relu', layer_norm_eps: float=1e-05, batch_first: bool=True, norm_first: bool=False, seed: int=0— python/vkml/nn.py:1105

Forward¶

`forward`¶

def forwardself, src: V.Tensor, src_mask: V.Tensor=None, is_causal: bool=False -> V.Tensor— python/vkml/nn.py:1139

Internals¶

`repr`¶

def __repr__self -> str— python/vkml/nn.py:1150

TransformerEncoderLayer