← The Transformer Abides: the companion site · All working papers
Attention Sinks at Long Context
One Log-N Law for Retrieval and Abstention
Abstract
Softmax attention cannot abstain on its own: its weights sum to one, so a head with nothing relevant to read must still attend somewhere. Trained models improvise such a place, the attention sink. Practice now builds one in: a null, an option that absorbs attention and delivers nothing, such as a learned sink logit, a register with zero value or an output gate. We show that these nulls are one mechanism, a gate on an ordinary softmax, and ask what happens to them as the context grows. The answer is one dilution law for two abilities. To retrieve, a matching key must outweigh the N other keys; to abstain when nothing matches, the null must. With bounded scores both leads must grow like log N, and because a query with a match and one without share the same null, no rule that sums over the keys keeps both at long context: a fixed null loses abstention, and a null that grows with the length loses retrieval. Tempering, which scales every score, the null included, by log N / log N_train, provably keeps both under margin conditions, and a head that decides by its largest score escapes the law. Trained heads in a controlled pilot follow the law closely, and tempering keeps both sides there, though not in every harder setting. In 125M-parameter language models, every rule with a null removed the first-token sink and most massive activations; past the training length, keeping positions in range mattered more than the null rule, and adding tempering to YaRN, which already includes a temperature, hurt. In small vision transformers trained at one resolution, tempering at evaluation raised accuracy at higher resolutions. A test on gpt-oss was inconclusive.
These are working drafts, not final versions. They are unpolished in places, incomplete in others, and keep exploratory material a polished paper would leave out, because it helps decide what comes next and shows the program's big picture more fully. Please read them with that in mind: the work will be refined further.
To cite: Neil D. B. Bruce. Attention Sinks at Long Context: One Log-N Law for Retrieval and Abstention. Zenodo, version 1.0, 2026. https://doi.org/10.5281/zenodo.23541963. Open access under a CC BY 4.0 licence.