Transformer (machine learning model)

Main article: Wikipedia: Transformer (machine learning model)

Attention is all you need paper
A neural network architecture by Google.
It is currently the best at NLP tasks and has mostly replaced RNNs for these tasks.

Architecture

The Transformer uses an encoder-decoder architecture. Both the encoder and decoder are comprised of multiple identical layers which have attention and feedforward sublayers.

Attention

Attention is the main contribution of the transformer architecture.

The attention block outputs a weighted average of values in a dictionary of key-value pairs.
In the image above:

• \(\displaystyle Q\) represents queries (each query is a vector)
• \(\displaystyle K\) represents keys
• \(\displaystyle V\) represents values

The attention block can be represented as the following equation:

• \(\displaystyle \operatorname{SoftMax}(\frac{QK^T}{\sqrt{d_k}})V\)

Encoder

The receives as input the input embedding added to a positional encoding.
The encoder is comprised of N=6 layers, each with 2 sublayers.
Each layer contains a multi-headed attention sublayer followed by a feed-forward sublayer.
Both sublayers are residual blocks.

Resources

Guides and explanations

• The Annotated Transformer
• Youtube Video by Yannic Kilcher