Why Sine & Cosine for Transformer Neural Networks

3-Second Blueprint (English)

1. Transformers add a same-sized positional encoding vector to every token embedding before passing inputs into encoder-decoder layers.
2. Sine and cosine functions bound positional values strictly within [-1, 1], helping tokens attend stably across distant positions.
3. Because sinusoidal encodings use a continuous closed-form formula, models can extrapolate position vectors to unseen sequence lengths.