LLMs · Transformers · Paper notes
Attention Is All You Need: a short walkthrough
This is a sample post to show what the template supports. Replace it with your own writing.
The core idea
Scaled dot-product attention is defined as
where , and are the query, key and value matrices, and is the key dimension.
In code
import torch
def attention(q, k, v):
d_k = q.size(-1)
scores = q @ k.transpose(-2, -1) / d_k ** 0.5
return torch.softmax(scores, dim=-1) @ v
Results at a glance
| Model | Task | Note |
|---|---|---|
| Transformer (base) | WMT14 En-De | Strong BLEU at lower training cost |
| Transformer (big) | WMT14 En-De | State of the art at publication |
Tip: set
draft: truein a post’s front matter to hide it until it’s ready.