← The Transformer Abides: the companion site · All working papers
The Transformer as a Constrained Equilibrium
A Lattice of Protections for Adaptive Sequence Mixing, and Where the Room Is
Abstract
A sequence-mixing layer that is iterated through depth and rebuilt from its own input faces six failure modes. Each is met by a protection—anchoring, alignment, gain control, abstention, a routing budget, or an arrow—with an operator-level statement proved within a declared class, a signature when it is removed, and a price. We organize attention designs as a lattice of kernel, dynamical, and block constraints in which relaxing a constraint buys a capability and obliges a re-supply. We read softmax attention inside a pre-normalized residual block as the lattice point where every protection is either supplied or cheaply re-supplied under parallel-hardware and optimization pricing: of the deficits tested, all but the anchor's were compensated by capacity elsewhere in the deeper models, and explicit protection knobs did not beat the block, both measured at small scale, while corner structure, by which designs would jump rather than drift, remains a hypothesis. The lattice reads the selected adopted changes to the mixing layer, its normalization and its residual path over the last decade as cheap re-supplies of protections, and it locates the room for change. The clearest room is abstention, governed by a two-sided log-N law: with a bounded logit scale, no null keeps both abstention and single-key retrieval as context grows, and tempering the entire logit budget with log N keeps both when its margins are strictly separated. A controlled pilot follows the law, while a test of its sink prediction on an open production model was inconclusive; descriptively its slopes leaned toward compensation through position, in which position encoding holds the background's growth down. In a trained harness of 125M-parameter language models, tempering every logit by log N / log N_train on top of YaRN, which already carries a log-length temperature, did not keep abstention and raised long-context perplexity. In these language models every rule that gives attention a null removed the first-token sink and cut massive activations by 67% to 82%, as earlier reports for gates and attention biases found, and, sigmoid aside, lowered perplexity slightly at the training length; the comparison made for two of these rules held in every seed, while the same comparison failed in the harness's vision transformers. We close with falsifiable predictions for future architectures, most of them not yet tested.
These are working drafts, not final versions. They are unpolished in places, incomplete in others, and keep exploratory material a polished paper would leave out, because it helps decide what comes next and shows the program's big picture more fully. Please read them with that in mind: the work will be refined further.
To cite: Neil D. B. Bruce. The Transformer as a Constrained Equilibrium: A Lattice of Protections for Adaptive Sequence Mixing, and Where the Room Is. Zenodo, version 1.0, 2026. https://doi.org/10.5281/zenodo.23542051. Open access under a CC BY 4.0 licence.