Layer Norm

Paper: Layer Normalization

Layer normalization is different from batch normalization. In layer normalization each token is first normalized $$ x_{\text{normalized}} = \frac{x - \text{mean}(x)}{\text{std}(x)} $$ and then it is shifted and scaled by learnable parameters $\gamma$ (scale) and $\beta$ (shift) $$ \gamma x_{\text{normalized}} + \beta. $$

The original Transformer paper used LayerNorm after the residual connection and the MSHA. However, nowadays people use LayerNorm before the MHSA and FF layers to stabilize gradients during training.