Post-LN

The original Transformer paper used LayerNorm after the residual connection and the Multi-Head Self-Attention (MHSA). Suppose $x$ was our input sequence of tokens, then the sequence would go through MHSA, then added to itself and then through LayerNorm $$ y = \text{LayerNorm}(\text{MHSA}(x) + x). $$ This would be fed through a Feed-Forward Neural Network (FFNN) following the same proceedure $$ z = \text{LayerNorm}(\text{FFNN}(y) + y) $$ This is known as Post-LN, since the LayerNorm is done after the layer of interest (MHSA/FFNN) and importantly it is done after the residual connection.