Rao Rizwan Iqbal

LLMs · Transformers · Paper notes

Attention Is All You Need: a short walkthrough

November 22, 2026 · Read the paper

This is a sample post to show what the template supports. Replace it with your own writing.

The core idea

Scaled dot-product attention is defined as

Attention(Q,K,V)=softmax ⁣(QK⊤dk)V\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

where QQ, KK and VV are the query, key and value matrices, and dkd_k is the key dimension.

In code

import torch

def attention(q, k, v):
    d_k = q.size(-1)
    scores = q @ k.transpose(-2, -1) / d_k ** 0.5
    return torch.softmax(scores, dim=-1) @ v

Results at a glance

ModelTaskNote
Transformer (base)WMT14 En-DeStrong BLEU at lower training cost
Transformer (big)WMT14 En-DeState of the art at publication

Tip: set draft: true in a post’s front matter to hide it until it’s ready.

New paper breakdowns, every other week

Short, careful, no hype. Unsubscribe anytime.

Or follow by RSS.