← The Transformer Abides: the companion site · All working papers
Beyond Transformers
Sums, Maxima and the Price of Order
Abstract
The properties that make a transformer work were read from the transformer, but they describe a design space, not a design. Any machine in which content decides who talks to whom, something else decides what is said, and the process repeats, faces seven conditions. Each of these may be stated within a declared class of machines: a tether to its own input (the anchor), a routing budget, a null (abstention), separate address and cargo (alignment), gain below its tether (gain control), an arrow and a writer. We price six of the conditions with results proved within declared classes and show for the seventh that transport alone cannot create features, and two of the conditions organize the space. Against an exchangeable background, a decision that sums over all N items needs a ln N schedule, while one that reads a maximum needs none against backgrounds bounded below its threshold and stops only at an evidence wall that the keys set and that, under a condition on the joint likelihood of the scores, no router can move. A hard machine orders writes without spending any of the router's margin, while a soft router with K ln r nats of positional range orders at most a fraction of pairs that tends to K/(K+1), where two identical writes count as ordered when the later gets at least r times the earlier's weight. In mean field, if every position must stay retrievable, the positional terms are paid from the router's margin Δ above the gain wall: their exponential mean over positions is at most e^Δ, so at the gain wall nothing can be ordered. Beneath both, routing information is conserved in the sense of a family of lower bounds: naming one of N items costs about ln N nats in any machine, and only the currency differs. Tests on constructed keys and codes measure the walls and the exchange rates, and a catalog places 43 families, from attention variants to physical and social analogies, on the conditions. From the law of sums we build a max-statistic head that should abstain and retrieve at every length with no schedule, up to its evidence wall. In trained 125M language models with positions interpolated by YaRN, it is the only rule tested whose null and match weights both stay within half a nat of log-odds out to eight times the training length, on seed means though not in every seed, and it fails without interpolation: the matching key falls far down the ranking, and the background maximum grows with length. These are attention weights, not behaviour. Seven small machine families train end to end as the token mixer of otherwise identical small models, but a log-N scale learned in training does not remove the length penalty of the machines that normalize competition and were trained with it. We read the transformer as the cheapest point under one price list, not the only point.
These are working drafts, not final versions. They are unpolished in places, incomplete in others, and keep exploratory material a polished paper would leave out, because it helps decide what comes next and shows the program's big picture more fully. Please read them with that in mind: the work will be refined further.
To cite: Neil D. B. Bruce. Beyond Transformers: Sums, Maxima and the Price of Order. Zenodo, version 1.0, 2026. https://doi.org/10.5281/zenodo.23542133. Open access under a CC BY 4.0 licence.