attention

Unsupervised Topic Modeling on BBC articles.

LLM trained on Dante Alighieri's Divina Commedia using Pytorch.

Input to the Transformer The input to the transformer is a sequence $X^{(0)}\in {\mathbb{R}}^{D\times N}$ where $N$ is the length of the sequence and $D$ is the dimensionality of each item in the sequence, which are known as tokens and denoted as ${\mathbf{x}}_n^{(0)}\in {\mathbb{R}}^{D\times 1}$. $$X=[{\mathbf{x}}_0^{(0)},\dots ,{\mathbf{x}}_N^{(0)}]$$ The items in the sequence are representations of objects of interest. For instance, in language tasks, a token is usually a unique vector representation of a word, whereas for an image it would be a vector representation of a patch.

Input to the Transformer The input to the transformer is a sequence $X^{(0)}\in {\mathbb{R}}^{D\times N}$ where $N$ is the length of the sequence and $D$ is the dimensionality of each item in the sequence, which are known as tokens and denoted as ${\mathbf{x}}_n^{(0)}\in {\mathbb{R}}^{D\times 1}$. $$X=[{\mathbf{x}}_0^{(0)},\dots ,{\mathbf{x}}_N^{(0)}]$$ The items in the sequence are representations of objects of interest. For instance, in language tasks, a token is usually a unique vector representation of a word, whereas for an image it would be a vector representation of a patch.