How might LLMs store facts | Deep Learning Chapter 7

3-Second Blueprint (English)

1. Transformer MLP blocks process each token vector independently through two learned matrix projections and a non-linear clip.
2. Rows of the up-projection matrix act like soft AND-gates that fire when input vectors encode combined entity features.
3. Activated neurons add factual attribute directions back into the residual stream, packing extra concepts via superposition.