Relative

Relative positional encoding was introduced in Self-Attention with Relative Position Representations and then improved in Music Transformer.

Self-Attention with Relative Position Representations

They modify the self-attention mechanism to include relative positional information. Suppose the input sequence is $x=(x_1,\dots ,x_n)$, meaning that $n$ is the context length, and $x_i\in {\mathbb{R}}^{d_x}$. In a single head, the standard self-attention mechanism, given query, key and value matrices $W^Q$, $W^K$ and $W^V$ computes $$\begin{array}{rlrl}{e}_{ij}& =\frac{(x_i{W}^Q)(x_j{W}^K{)}^{\mathrm{\top }}}{\sqrt{d_z}}& & \text{similarities}\\ {\alpha }_{ij}& =\frac{\exp (e_{ij})}{\sum _{k=1}^n\exp (e_{ik})}& & \text{softmax normalization}\\ z_i& =\sum _{j=1}^n{\alpha }_{ij}(x_j{W}^V)& & \text{attention and value multiplications}\end{array}$$ They additionally learn two vectors for each pair of tokens in the context, $a_{ij}^V$ and $a_{ij}^K$. These vectors are $d_z$ dimensional and are shared across heads. Then, their new attention mechanism is $$\begin{array}{rlrl}{e}_{ij}& =\frac{(x_i{W}^Q)(x_j{W}^K+a_{ij}^K{)}^{\mathrm{\top }}}{\sqrt{d_z}}& & \text{relative similarities}\\ {\alpha }_{ij}& =\frac{\exp (e_{ij})}{\sum _{k=1}^n\exp (e_{ik})}& & \text{softmax normalization}\\ z_i& =\sum _{j=1}^n{\alpha }_{ij}(x_j{W}^V+a_{ij}^V)& & \text{attention with relative value}\end{array}$$ In practice, they don’t learn two vectors for all possible pairs of tokens in the context. Instead, they choose a maximum “relative distance” $k$ and then clip it so that they only learn $a_{ij}^V$ and $a_{ij}^K$ from $-k$ to $k$.