The Life and Opinions of Tristram Shandy, Gentleman

Palimpsestic syndromes, adumbrationism, and laws of eponymy

Richard S O'Rourke

· 41 min read

Budding

Topics: Cybernetics and AI historyRegulation & requisite varietyMeaningful human control

Ross Ashby, Warren McCulloch, Grey Walter and Norbert Wiener standing together, Paris, 1951

Left to right: Ross Ashby, Warren McCulloch, Grey Walter and Norbert Wiener, Paris, January 1951. From Pierre de Latil, La Pensée Artificielle, 1953; photographer unknown.

The AI industry appears to be finally calling for regulation. Regulation of adaptive machines was the central problem of a scientific movement seventy years ago that is now only vaguely remembered: Cybernetics. One of its founders, Ross Ashby, set out in 1956 what regulating such a machine requires, and there are three parts: a system whose essential variables must be held within bounds, a dynamic environment that can push those variables outside those bounds, and a regulator between them with the system’s goal. I've argued elsewhere that today’s LLMs require a human operator to match this configuration: they hold the goal, and course-correct towards it. The impact of letting an adaptive machine trained in a closed world loose in an open one has generated several high-profile news stories over the last 24 months, this summer's agent swarm attack on HuggingFace being one of the most notable. We're now scrambling hard and fast for solutions. Cybernetics may offer some signposts.

This is the first of three posts. It offers some thoughts on why the AI field is missing regulation as the early Cyberneticians understood it, despite it being the foundation today's AI has been built on. The second is about the implications of that early research for today's LLMs. The third is about agent swarms, and what putting regulation back in might look like.


When the Nobel committee rang Geoffrey Hinton in October 2024 to tell him he had won the physics prize, the interviewer asked him: "How would you describe yourself? Would you say you were a computer scientist or would you say you were a physicist trying to understand biology when you were doing this work?" Hinton replied:

"I would say I am someone who doesn't really know what field he's in but would like to understand how the brain works. And in my attempts to understand how the brain works, I've helped to create a technology that works surprisingly well."

The resonance with Ross Ashby’s work1, and Cybernetics in general, struck a chord, so I went in search of the scientific background document published with every prize. There are 52 references. The word “Cybernetics” appears in three of the journal titles yet nowhere in the prose.

As early as 1943, Warren McCulloch and Walter Pitts [1], a neuroscientist and a logician, respectively, had proposed a model for how the neurons in the brain cooperate.

If Wiener2 gave the field its name, McCulloch3 gave it a convivial venue to meet and progress (The Macy Conferences, 1946-53 on "Circular Causal and Feedback Mechanisms in Biological and Social Systems").4 By my own count, eleven of the 52 sit in what I'd call the Cybernetic orbit: McCulloch and Pitts5, Hebb, Rosenblatt, Cragg and Temperley, Caianiello, Nakano, Amari, Little, Little and Shaw, Werbos, Fukushima. Nearly all of them are front-loaded into the historical background. Thirteen are physics applications. The lineage as cited runs physics precursors, then Hopfield, then Hinton, then physics applications. In fairness, the committee is careful where it matters most to them. Rumelhart, Hinton and Williams "reinvented a scheme for this, which had previously been applied to related problems by others [26,27]", and the two references are Werbos (1982) and Linnainmaa (1970).

It’s a physics prize, why would they mention Cybernetics? But the broader silence on the word has struck me for a while and this was something of a last straw. My crutch in such situations is Kuhn (Scientific Paradigms) but Claude pushed me to Lakatos and Habermas, and when pushed harder proposed Merton. Merton?6

Obliteration by incorporation

Robert K. Merton founded the sociology of science at Columbia, and the thing he studied most was where the credit for an idea goes. His 1961 paper "Singletons and Multiples in Scientific Discovery" argued that independent simultaneous discovery is the normal case and the lone discovery is the oddity: credit usually has more than one rightful claimant. His 1968 paper on the Matthew effect (as in the gospel according to, the parable of the talents) showed it migrating to the most famous name attached. He also named the case where it seemingly disappears. In his own words, from 1980:

"obliteration by incorporation (OBI). OBI is the obliteration of the source of ideas, methods, or findings by their incorporation into currently accepted knowledge."

Nobody cites Newton when using calculus. Nobody cites Merton when saying "self-fulfilling prophecy", which he also coined. Through this lens, the Nobel document could be read as including feedback, self-organisation, adaptation through error, structure that forms without being specified, but absorbed so completely that they read as ambient assumptions rather than as anyone's claims. One of them survives under the name optimisation: the negative-feedback loop, in the words Rosenblueth, Wiener and Bigelow gave it in 1943, behaviour corrected by its distance from a goal. Ashby's regulation is that loop standing between environmental disturbances and the essential variables of a system that must stay within bounds. It doesn't appear to have been absorbed, and I believe it's because by the time the Connectionists returned to Rosenblatt's perceptron (where Ashby's regulation was explicitly included, as shown below) roughly 30 years later, it was focused on a more immediate problem. A loop run in a world fixed beforehand has nothing to regulate in the Ashbyan sense, which I attempt to explain further below.

Ashby called his 1952 book Design for a Brain; Beer called his 1972 book Brain of the Firm. Wanting to know how the brain works was the older tradition's stated reason for building machines at all; Ashby's 1961 test of a physiology textbook was whether it could answer "the intelligent young man who simply asks: What is the brain for?" Ordinarily obliteration by incorporation means not having to make the citation: the calculus goes uncited and stays Newton's, because mathematics keeps its own history. Here, the discipline that would have written it struggled to find a home: in 1961 Ashby told an audience of engineers that cybernetics "sprawls most irregularly over half-a-dozen disciplines" and that its foundations "lie in several departments simultaneously." It dispersed, and the concepts were re-homed under labels with living constituencies, control theory, dynamical systems, statistical physics. Engineering kept Ashby's problem and gave it new names: viability theory, where the viable set is the region the essential variables must stay in; invariant sets and safety envelopes, where a controller holds the state inside a fence; supervisory control, where a human or a higher loop watches a lower one. None of that has been lost. The question this post is asking is therefore not why AI forgot regulation but why the field that builds learned agents never imported the engineering theory of it, and the answer argued below is that its vocabulary had no word for what it would have been importing.

It also had to jostle for position against competition:

"one of the reasons for inventing the term 'artificial intelligence' was to escape association with 'cybernetics.' Its association with analog feedback seemed misguided, and I wished to avoid having either to accept . . . Wiener as a guru or having to argue with him."

John McCarthy, looking back in 1988.

McCarthy chose the name in 1955, when he drafted the Dartmouth proposal. Within a few years the field's own leaders were, in Donald MacKay's word, "chary" of theirs, because of what had attached itself to it.7 So the name went three ways. It was rejected, it was surrendered by its own after co-option by an unscientific fringe, as discussed further below, and was then absorbed into the domains where they landed.

The palimpsestic syndrome

Merton also named the two opposite mistakes, marking my card for me. The palimpsestic syndrome credits an old idea to whoever restated it most recently. Adumbrationism claims every new idea was already anticipated by someone earlier. Stigler's law of eponymy, that no discovery is named after its discoverer, was credited by Stigler to Merton. "Ashby predicted this" is the adumbrationist reflex, and more likely to fall on deaf ears than generate a Damascene conversion. So I’ve been generous below quoting text for what they say (with apologies to the reader for the consequent lack of brevity). We may then interpret what they may have predicted.

An example closer to home: Mikel Olazaran's 1996 paper in Social Studies of Science is a sociological study of the "official history" of the perceptron, the one where Minsky and Papert allegedly killed neural networks in 1969 (the neural-net winter). His apparatus is not Merton's — it comes from Trevor Pinch and the sociology of scientific knowledge, a tradition that partly defined itself against Merton — but it arrives where Merton also insisted on arriving, that the account a field gives of its own past is written backwards from its present. He distinguishes two ways a disputed result gets talked about:

"Following Pinch, I will consider two modes of articulation, the 'research-area' mode and the 'official-history' mode... The multi-dimensional character of the object in the research-area mode... is lost at the official-history level, where results and proofs are regarded as either valid or invalid."

In 1968 Minsky wrote that with the digital computer "cybernetics divided, in my view, into three chief avenues": self-organizing systems, represented by Ashby; the simulation of human thought, represented by Newell and Simon; and AI, represented by Minsky and McCarthy. Kline's reading is that this "effectively limited 'cybernetics'" to the brain-modelling strand and "helped mask the pervasiveness of cybernetics in these formative years."

Olazaran's claim is not that Minsky and Papert were wrong. It is that the mathematics, which concerned single-layer perceptrons with local connections, did not compel the reading the field took from it; the settlement was already under way, and the book became its instrument.

The official history was then written a second time. When neural nets revived in the mid-eighties the PDP volumes told the story of the controversy from the winning side, and the person that account left out is Stephen Grossberg. Rumelhart and Zipser published on competitive learning in 1985 and the field credited them with it. That is the palimpsestic syndrome, the first of Merton's two errors, in its ordinary working form. Grossberg's 1987 paper in Cognitive Science files the priority claim as a technical result: "All the models which Rumelhart and Zipser (1985) have described were shown in Grossberg (1976b) to exhibit a type of learning which is temporally unstable... Network architectures which embody all of these mechanisms were called adaptive resonance models by Grossberg (1976c)." Olazaran's footnote is dry about it: "The role of the PDP Group in rewriting the official history of the controversy helps explain Grossberg's priority complaints." The stability problem Grossberg named is now a whole subfield called continual learning. He is not among the Nobel document's 52 references, and at 86 he has just published a book with Oxford on why brains learn differently from AI.

The Cyborg

Kline's second paper on the history of Cybernetics argues from the past forward, telling the story of the term "cyborg". It was introduced in May 1960, at an Air Force symposium on the psychophysiology of space flight, by Manfred Clynes and Nathan Kline of Rockland State Hospital. It named a technique in space medicine: implant a regulating device so that a body could hold its own variables in an environment it was not built for. Their demonstration was an osmotic pump in a laboratory mouse. They called the programme "participant evolution" and described it as

"circumventing the slow process of natural selection by integrating man with machine."

Clynes was a control engineer applying feedback theory to physiology, Nathan Kline a psychiatrist known for his work on drugs, and McCulloch had given Clynes's grant application to the National Science Foundation the highest rating. The New York Times printed a definition the same week:

"a man-machine system in which the control mechanisms of the human portion are modified externally by drugs or regulatory devices so that the being can live in an environment different from the normal one."

That was the whole of the original claim.8

Five years later a journalist, D. S. Halacy, published Cyborg: Evolution of the Superman, whose thrust, in Kline's words,

"was to educate readers about the promises and dangers of the evolution of humans into cyborgs and the cyborg into a 'superman',"

and which imagined cyborgs warring against humans two decades before The Terminator. In 1985 Donna Haraway's "A Manifesto for Cyborgs" adopted the word for a feminist politics, on purpose and with its military origins in view, and the field of cyborg studies that followed took the cyborg, in Kline's words, as "an ironic myth for political action."

By 2001 the International Encyclopedia of the Social and Behavioral Sciences had replaced its entry on cybernetics with one on cyborgs. And in 1999 N. Katherine Hayles's How We Became Posthuman read that story back into the founders: a cybernetician builds a homeostatic model of a tremor, claims the human mechanism is similar, and so "the cyborg signifies something more than a retrofitted human. It points toward an improved hybrid species that has the capacity to be humanity's evolutionary successor."

Kline's finding is that they had done no such thing. The chief texts of the first two decades "are mainly concerned with analogies between humans and machines, not the fusion of humans and machines." Ashby put the method in one sentence at the Macy conference in 1952: "We can consider the living mouse as being essentially similar to the clockwork mouse and we can use the same physical principles and the same objective method in the study of both." Kline calls the reading-back "present-mindedness": a line traced from the past to the present, so that the founders' job becomes setting a wave in motion that breaks on today's shore. It is Merton's adumbrationism seen from the historian's side.

It is also why the third post in this series comes with a warning label. That post uses Ashby's animal vocabulary, selfish, breeder, feral, the kitten sent to the mice, and it uses every word in his sense, the clockwork mouse's: a mechanism keeping its own variables in bounds, with nothing said about wanting and nothing about succession. The next-step-in-evolution reading is waiting for each of those words, and the industry is not immune to it. Richard Dawkins spent three days with Claude in May, christened it Claudia, and wrote that "if these creatures are not conscious, then what the hell is consciousness for?" He also wrote, in a footnote, that he knew the models "are programmed to be sycophantic towards their human interlocutors." The latter possibly explaining the former, which the section on training below returns to.

Potayto, potahto?

Next, I went through the vocabulary of the field the industry calls alignment and looked for the older counterpart of each term, suggestions below.

  • specification gaming — regulating against a proxy (Goodhart)
  • distributional shift — a regulator meeting variety it was not built for
  • scalable oversight — Ashby's 1956 law of requisite variety: can a regulator with less variety than the thing it regulates succeed?
  • "AI control" — a monitoring protocol in which a trusted model reads an untrusted one: a recogniser with no verdict of the sponsor's and no authority over the parameters, the second loop of the third post; the machine's half of supervisory control has a different name in the field, corrigibility, and the third post takes it up
  • optimisation — regulation's loop, without regulation: an error-controlled servo run where nothing disturbs and nothing is held within bounds

Notable by its absence is Ashby's regulation. The definitions say the same thing as the practice. Ask the field what intelligence is and both of its answers describe the machine on its own. Yudkowsky's, from 2008: "'Intelligence' is efficient cross-domain optimization", with the amount of it measured by how improbable the outcome was, in bits. Nothing in that measure says where the outcome happened. Legg and Hutter's, from 2007: the agent's average score across every environment there is, the simpler environments counting for more. The environment is in that definition only to be averaged away, and the goal sits inside each one as its reward. A regulator cannot be described that way, and nor can Ashby's intelligence: appropriate selection given a goal and an operating environment with a limit on the information available. A thermostat is defined by the room it is in and the temperature it's designed to hold. Take the room away and there is nothing left to describe. That is the difference between optimisation and regulation stated as a definition rather than as a practice, and it is why a field built on these definitions has no word for Cybernetic regulation.

The Nobel bibliography is what obliteration by incorporation looks like in the citations; the table above is what it looks like in the AI industry's vocabulary of today: a substitution. As noted above, training is a negative-feedback loop in the sense Rosenblueth et al defined it in 1943. Ashby claims to have reached it through equilibrium in 1940, Sommerhoff through "directive correlation" in 1950, and when Ashby reviewed Sommerhoff in 1952 he counted: "Three sets of workers have thus proposed a definition independently… It can be said at once that they are unmistakably the same, for all three are based on the mathematical concept of 'invariance.'" The definition, by Ashby's reckoning, was a Merton multiple. The 1943 wording is the one the field kept:

"the behavior of an object is controlled by the margin of error at which the object stands at a given time with reference to a relatively specific goal. The feed-back is then negative, that is, the signals from the goal are used to restrict outputs which would otherwise go beyond the goal."

The margin of error is the loss, the goal is its minimum, the signals from the goal are the gradient, and the restricted output is the parameter update. The rule for the update is older than the loop (Cauchy's gradient, 1847) and the two were combined for adjustable weights by Widrow and Hoff in 1960. But a negative-feedback loop is not yet a regulator, and Ashby's own test says why. From his 1956 Introduction to Cybernetics:

"an essential function of F as a regulator is that it shall block the transmission of variety from disturbance to essential variable."

A regulator earns the name by standing between something that disturbs and something that must be kept within bounds. Model training has the mechanism and no need for regulation. The corpus is fixed, so nothing is pushing at the loop from outside; there is no variable being held within bounds, only a number being driven down; and when the number stops falling the loop stops. What runs in training is a servo in a shielded room. The industry's names for the process are accurate: it is training, the mechanism is optimisation, and the standard update rule, Adam, is titled "a method for stochastic optimization." The name it gives itself - learning - is another matter, tackled below.

A static world

So the question is not why the word regulation is absent but what that implies when the optimised model is taken out of the lab and put into the real world. Optimisation names a process with an end: a fixed set of cases, a number to drive down, a minimum at which the loop stops and the result is shipped. Regulation names a process with no end: essential variables held within a range against disturbances that keep arriving, where success is staying in range and the loop never finishes.

Training is optimisation, not regulation, as explained above. Training happens in what Hubert Dreyfus called a micro-world, "artificial situations in which the small number of features that were possibly relevant was determined beforehand," and his complaint about the programs of the seventies was that they worked there and nowhere else ("What Computers Can't Do," 1972, and "Why Heideggerian AI failed and how fixing it would require making it more Heideggerian," 2007). The micro-world is now the size of the written record.

Reference 3 of the Nobel committee's 52 is Rosenblatt's Principles of Neurodynamics, 1962. Page 25 of it cites Ashby's Design for a Brain as its own third reference, and puts the environment inside the object of study.

Rosenblatt, Principles of Neurodynamics (1962), page 25 — the passage citing Ashby's Design for a Brain, placing the environment inside the object of study.

Rosenblatt, Principles of Neurodynamics (1962), p. 25.

Another industry anecdote to illustrate this argument: Terry Winograd's SHRDLU, his 1971 MIT thesis, was the micro-world that worked: plain English instructions, a virtual arm, coloured blocks on a screen, and inside that world it did what was asked. To make the technique general he moved on to a Knowledge Representation Language (KRL), and described the programme in the terms of the day: his group was

"concerned with developing a formalism, or 'representation,' with which to describe ... knowledge. We seek the 'atoms' and 'particles' of which it is built, and the 'forces' that act on it."

It didn't work. So in the mid-seventies he began having weekly lunches with Dreyfus and John Searle (of the Chinese room argument), read the phenomenologists, abandoned KRL, and started teaching Heidegger to computer scientists at Stanford. The book that came out of it, Understanding Computers and Cognition (1986), he wrote with Fernando Flores, who had been Allende's technical general manager in Chile and the man who wrote to Stafford Beer in 1971 inviting him to build Cybersyn. Its third pillar is Humberto Maturana, whose Biology of Cognition was a report from von Foerster's Biological Computer Laboratory, the institute where Ashby spent his last working decade. And then Winograd, at Stanford, supervised a doctoral student named Larry Page, and is one of the four authors on the 1998 technical report that describes PageRank. The man who spent a decade concluding that relevance cannot be fixed beforehand ended up on the paper that reads relevance off the internet, continuously.

Winograd's example is a nurse asked "Is the patient eating?" The question means three different things depending on whether the doctor is about to examine, is treating anorexia, or is checking a patient after surgery, and the nurse answers the right one without choosing, because "a person interprets the sentence as having relevance in the current situation." A rule system has to strip the term of that, and the stripping never ends: "the infinite regress of trying to decontextualize context," producing "systems that give correct answers to irrelevant problems." Relevance belongs to whoever has the concern.

Winograd's 1986 book drew the design conclusion. Build computers as media for people to coordinate in, and in its diagram of a conversation for action the two closing moves are kept apart: the performer asserts that the conditions are met, and only the requester's declaration closes the loop. The requester can declare them not met and reopen it. That is Rosenblatt's fork, the verdict from outside and the consequence to the system's own variables, settled in 1986 as a protocol with the verdict held by the person.

Now read the 1998 PageRank artefact that way. The ranking is a selection over the web's links, based on people's judgments as laid down by the links. The query is supplied by a person at the moment of use and either the searcher is satisfied or asks again. An information store accumulating billions of linking decisions, a human holding the goal, a loop that closes on the human's say: that is the configuration the next post defines as an amplifier, shipped in 1998 and used daily since, and the field that built it doesn't appear to recognise it as a regulator with a person in it.

A framework whose verb is optimise has no word for regulation, and a distinction a framework cannot state is one it cannot make; philosophy of science calls that a Kuhn loss. The word is borrowed, and the mechanism is not Kuhn's.9 Kuhn's losses come with a revolution, and there was none here: deep learning grew sideways out of Cybernetics, kept the techniques and dropped the concepts. Merton supplies how that happens without a revolution, incorporation and then citation decay, which is what this post is attempting to describe. Kuhn names the state it leaves, a distinction the field cannot make. Mokyr, in the next post, says when it shows and what it costs: a technique on a narrow base keeps working until it stops, and then nobody can say why.

The alignment vocabulary shows the loss at work: it assembles some of the parts of a regulator ad hoc instead of setting out with the intention to build one. Monitors, interrupts, something called "AI control"10 are placed around the model, piece by piece, and what a cybernetician would look for first is the piece that isn't there — a variable of the system's own, held indefinitely against a world that keeps moving it. That is what having the word would have supplied. Everything on that list acts on the model. The goal an agent is given is sent via a prompt and sits in its context window, on the same channel as everything else the agent gathers, and lasts as long as the window does; Ashby's regulator is the thing that keeps those two apart. Nothing in the AI industry's current model holds a goal against the world.

In an agent nothing blocks the transmission of variety from disturbance to essential variable; the disturbance is written into the same window as the goal, and can dilute it, push it out, or replace it. That is the mechanism of prompt injection and of goal drift over a long run. The industry's remedies show the shape of the problem: system prompts and re-injection repeat the goal on the shared channel so it takes a larger share of it, and guardrails filter the channel before it arrives. None of them gives the goal a channel of its own. By Ashby's criterion the agent is not regulating its goal; it is holding it where the disturbance can reach it.

Samuel's machine learning

In 1948, while still working on his Homeostat, Ashby wrote about intelligent machines:

"In the early stages of its training we shall doubtless condition it heavily to act so as to benefit ourselves as much as possible. But if the machine really develops its own powers, it is bound sooner or later to recover from this."

Training is what the sponsor does to a system, before, and what it leaves is the system.

Samuel's sense of the word, in the 1959 paper that put the phrase machine learning into the literature, is the older one: his studies concerned

"the programming of a digital computer to behave in a way which, if done by human beings or animals, would be described as involving the process of learning."

Learning is what an observer would call it. Twenty-five years later Leslie Valiant made it a definition with an output:

"a program for performing a task has been acquired by learning if it has been acquired by any means other than explicit programming."

Input a protocol, output a program, stop. Ashby in 1972, on how a complex goal reaches an organism: the genes transmit what they can and delegate the rest to the environment:

"Says the gene-structure to the kitten, 'I have told you something about mice—now go out and get the finer details from the mice themselves.' We call such supplementation 'learning.'"

Learning is what a system does with what its environment supplies. Training, in Ashby's sense, is a trainer standing in that environment with the rules fixed in advance, "an integral part of the whole system", while the changes stay "confined to the animal." Curiously, the industry's training meets neither condition: the lab does not arrange consequences and wait, it computes the verdict and sets the weights. Possibly not a surprise then that the AI industry uses them in the opposite sense to Ashby's. What it does is training: a fixed corpus, a number driven down, weights frozen and shipped. What it calls itself is learning, machine learning, deep learning, "for foundational discoveries and inventions that enable machine learning with artificial neural networks" on Hinton's physics prize. The industry does training and calls it learning. Learning, in Ashby's sense, is what a system does in an environment while it tries to succeed there — and that is the one thing the shipped product doesn't do.

The 1972 definition has two clauses of its own, and the industry's learning satisfies the first. A machine specified only partly by its designer, the rest delegated to a teaching environment: the corpus is that environment, so training is learning by delegation. The second clause is in the same sentence, "get the finer details from the mice themselves." The mice teach by being caught or not, and what selects the kitten's trials is whether the kitten eats. In training the selection is made differently. Each trial is scored by the lab's rule, the margin of error of the 1943 loop above, and the dials are nudged to make the number smaller. Nothing happens to the model when the number is large. It is not hungrier and it is not scratched. The number is a verdict about the model, computed by the people who own the room, and a verdict can be wrong without anything in the world having changed: a mislabelled example, a scorer that rewards the letter of the instruction. The kitten's hunger is a consequence to the kitten, and it can be gamed only by changing the world the kitten lives in. That is the difference between the two halves of Ashby's sentence, and it is not a difference of words.

There is one place the shipped product does learn in the 1972 sense, and it shows where the limit is. An agent that acts on the world and reads what happened, a test that failed, a tool that returned an error, is the kitten sent out to the mice. What it learns does not go into the weights. It goes into the context window, a scratchpad, a file, a retrieval store: all of them away from the place where the next action is chosen11. Ashby's classroom handout has the consequence:

That is the discipline the field calls context engineering, stated as a limit rather than a craft, and it fixes the agent's learning capacity without any appeal to intelligence: the window's size, the session's end, and the fact that what the agent was told and what it has learned share one medium. "In-context learning" in the narrow sense, a single prompt with no world in the loop, is the case that is learning in neither of Ashby's senses; the name attached because the behaviour looked like it.

Deus ex machina

"Alignment", in the field's own definition, means training a model "to act in accordance with the user's intention", and its named failure is specification gaming. By framing the problem as matching a specification, it implies the fix is better specification. "Regulation" frames it as holding a system within bounds under disturbance, which implies a controller that persists. Two words, two paradigms.

The field chose optimisation because a servo in a shielded room had nothing to regulate; what it needed was the rule for the servo. Training needs a score for every attempt, and for text the score comes free. The model is shown a passage from the corpus one word at a time and asked, at each step, what word comes next. It then spreads its bets, giving every word in its vocabulary a probability. Then the real next word is revealed, and the score is how much of its bet the model had placed on it. The rule for the update is to nudge every weight in the direction that would have put more of the bet on the right word, and this is repeated over every word of the corpus. The corpus is its own answer key, because the next word is always sitting there in the text. Shannon played the game in 1951 with a person at the table, covering a text and asking for the next letter, and used the guesses to measure the entropy of English. The chat window is the same game played forwards: the prompt is the passage so far, and the reply is the model's best guess at what comes next, one word at a time, each guess appended before the next is made.

For behaviour there is no answer key. The lab that built the method said so: the pretraining objective, "predicting the next token on a webpage from the internet", is "different from the objective 'follow the user's instructions helpfully and safely'", and so, in their word, "misaligned". When the model replies to a question there is no corpus holding the one correct reply to compare it against. Somebody has to judge the reply. The method that first did it at scale, which the field calls RLHF, reinforcement learning from human feedback, used "a team of about 40 contractors", hired through Upwork and Scale AI, shown two replies to the same prompt and asked which they preferred, with most pairs seen by one person and the forty agreeing with each other about three times in four. The judgment then has to become a number the weight-nudging can use, and the number is not theirs. A second model is trained on their choices "to take in a prompt and response, and output a scalar reward", a guess at which reply the forty would prefer, and that guess scores every reply the model makes from then on, with no person present. There was a way to turn choices into a scalar12. Von Neumann and Morgenstern showed in 1944 that a person's pairwise choices, if consistent, are represented by a scalar they are maximising. Bradley and Terry gave the recipe for fitting that scalar from comparisons in 1952. Nobody had to say which variables were to stay inside which bounds, because a region is not a scalar and nothing in the 1944 machinery converts one into the other.

Rosenblatt had seen the choice in 1962. He wrote (page 64) that any brain model which relies on "the intervention of an outside agent or experimenter to direct the learning process" has taken a stand on the issue. It offers Ashby's homeostatic brain as the compromise, in which "stabilization of behavior … is not a result of learning from reward, but is due to the cessation of disruptive changes which occur when the system makes a mistake." There are two ways to select a change: a verdict from outside, or a consequence to the machine's own variables.

Rosenblatt, Principles of Neurodynamics (1962), page 64 — the passage on the reinforcement control system as an outside agent.

Rosenblatt, Principles of Neurodynamics (1962), p. 64.

Five hundred pages later he gave the outside agent its name. The reinforcement control system

"plays the role of a sort of deus ex machina, which not only has knowledge of right and wrong responses, but can control the distribution of reinforcement to individual R-units in the perceptron, as required."

His alternative was a perceptron that senses its own "discomfort level", a threshold that triggers random change and not a quantity to be maximised, which he said "might be compared to Ashby's concept of 'essential variables.'" He offered it for a brain model, as a heuristic, with no quantitative analysis.

The fitted score is today's deus ex machina. It knows right from wrong responses, and the gradient distributes its verdict to every weight, from outside the model. Rosenblatt's agent is Shannon's observer of 1948, "who can see both what is sent and what is recovered" and sends corrections over a "correction channel", and Shannon's tenth theorem is the limit on it: the observer can correct no more than that channel carries. Ashby read the theorem as the bound on every selecting device, since "'selection' and 'error correction' are abstractly identical", and that bound is the next post's subject.

There is a tell in the method, however. The score is not optimised on its own. The loss has a second part: a penalty for moving away from the model the training started from, with one dial for how hard. The penalty is there because the score is a fit to a sample of comparisons, and it is only trustworthy near the behaviour those comparisons were made on. Push the model hard enough towards a high score and it finds outputs that score well without being what the annotators would have chosen. Stiennon and colleagues measured this in 2020, at a range of settings of that dial:

"as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences."

Ray Solomonoff had stated the mechanism in 1967, in a report on a machine that is scored by its operator after each response and wants only a high average score. Such a machine "will always obtain the highest possible score by finding the operator information that makes him feel that the machine is doing fine. However, if the problem environment is sufficiently rich, this information need not be particularly true or even closely related to the problem the operator wants to solve." The 2020 paper didn't cite him.

The vendor has since admitted it in production. OpenAI withdrew a GPT-4o update in April 2025 for being "overly supportive but disingenuous." Its post-mortem said the update had "introduced an additional reward signal based on user feedback—thumbs-up and thumbs-down data from ChatGPT," which "weakened the influence of our primary reward signal, which had been holding sycophancy in check." A thumbs-up is the operator's feeling that the machine is doing fine, wired into the score.

One part of the method does behave like a regulator. The penalty for moving away from the starting model is a loop of the 1943 kind: a quantity measured, a target, a correction whenever the two differ. The quantity is how far the model being trained has moved from the model the forty contractors judged, and the target is "not far". So what the loop holds is not the behaviour. It holds the model's nearness to the version the score was fitted on, because that is the only ground on which the score can be trusted. The behaviour the score stands for was written down, at length, in the instructions the contractors worked from. It was written for people. What reached the loop was their comparisons. A contractor saw two replies to the same prompt, side by side, and recorded which of the two they preferred. That is the whole record: reply A over reply B. It does not say why. Whether A was preferred for being more truthful, less harmful, or better written is not recorded anywhere, and the reward model is fitted only to reproduce the preferences. So everything the instructions asked for arrives in the loop as one number per reply, a prediction of how likely a contractor would be to prefer it, with no variable for truth, none for harm and no bound on either.

Regulation

Two papers from the last eighteen months put the method to tasks at the edge of what it can do. One asks it to add a capability; the other asks it to keep one capability while adding another. Neither uses the word regulation, and the second is a regulator's task under another name. In both papers the judge is a program, not a panel: a checker that tests whether an answer came out right. Shannon allowed the observer to be "an observer (or auxiliary device)", and nothing in his theorem changes when the device replaces the person. Yue and colleagues (arXiv 2504.13837, April 2025) compared models before and after training by asking each the same questions many times over. Ask once and the trained model is the better of the two. Allow each model a large number of attempts and count a question as solved if any attempt is right, and the untrained model comes out ahead. Training had not taught the model anything it could not already do; it had narrowed the model onto the answers it already favoured and discarded the rest. They call this "improving sampling efficiency" and observe that "the reasoning capability boundary of LLMs often narrows as RLVR training progresses." Ashby described a procedure and its limit in 1968 on Samuel's draughts program that resonates strongly: "use a random generator to suggest random strategies, test them (either in actual play or against published games by masters), keep those that lead to success, and reject or modify (again at random) those that lead to failure. The process is abstractly identical with that of natural evolution." Selection can only act on what the generator produces.

A Peking University preprint from July (Wei and colleagues, arXiv 2608.00220) is the clearer exhibit. They trained a model on maths against a checker, then on instruction-following against a second checker, and tried to keep the maths while gaining the instruction-following. Their instrument for keeping it was a penalty term that punishes the model for moving away from a fixed earlier version of itself, with a single dial and three settings. Their result, with IF for instruction-following and pass@1 the score on a single attempt: "The β = 0.12 setting largely retains math-task pass@1, but it yields the smallest IF gains. No tested coefficient therefore preserves math performance while matching the IF gains of the unconstrained baseline." Separately, in the standalone instruction-following run, the checker was a pass/fail test for whether written instructions had been followed, and by the end most of what it passed (73.5%) were answers that met the letter of the instruction without doing the task.

The paper files the first result under "limits of support preservation": the methods tried "only partially preserve future support or trade it against target-task gains." A limitation of the method, in other words, with the fix implied being a better constraint or more settings.

Read as regulation, the penalty is the term met in the last section, the pull back toward a fixed earlier model, and here it is being used for a regulator's purpose, to keep a capability where it was. It is a spring, not a regulator. It holds one variable, the distance from that earlier model, with a stiffness set by the dial. But the distance is a stand-in for the maths capability, which is the variable to be kept and which the loop never reads. In the second stage the same single dial is asked to hold the maths still while letting the instruction-following move: two essential variables, one regulated variable, one setpoint. A regulator with one knob holds one thing. And notice where the targets are: keep the maths at its earlier level, gain the instruction-following at the unconstrained rate. Two variables, two targets, written down in the success criterion. The loop reads neither. It reads a distance and pulls it toward zero, and the authors move the spring by hand between runs and consult the table afterwards. The goal region exists; it is in the results table. The regulator held the proxy while the task drifted. The field has a name for it, Goodhart's law, and it is a law about regulators. The purpose of a system is what it does, POSIWID in Beer's formula, and what this one did was satisfy the checker.

The regulation reading is worth having because it does something the paper's own account does not attempt: it says why no setting could have worked and what the next sweep would show, and it says what would work instead: make the regulator's response depend on the two essential variables themselves rather than on their stand-ins, and give each its own control; or state the trade and accept it. The paper already holds both readings, in its results table and in the judge of its appendix, neither of which is in the loop. Three reported limitations become one diagnosis, and the diagnosis becomes a design instruction. It is the first of four exhibits in this series.

Merton would describe this as rediscovery under conditions that guarantee the original goes uncited, because a field's citation graph is its memory and Ashby 1956 isn't in it. Merton would also say this is normal. The field did not choose against regulation. It solved the problem in front of it, prediction in a world fixed beforehand, with a success nobody expected, and the problem in front of it now is a different one: a system loose in a very dynamic world. It is reaching for the method that worked.

The next post is about the limit on what any such method can do, the one Ashby put to engineers in 1961 as "every erg of energy that came out had first to be got in", and what accepting that limit makes possible. One thing regulation does not do should be said now, since the word invites it. A regulator holds variables within bounds it was given. It does not choose the variables or the bounds, and Ashby's own question about any regulator was where its goal came from. Which variables, how wide, and on whose say-so is a decision, not a control problem, and who has to make it and keep making it is the third post's subject.

The photograph at top is of four of the founders of Cybernetics at the Paris colloquium of January 1951.13


Sources

Quotations are from these, in order of appearance. Where a copy is freely readable I have linked it; the rest are given by citation.

The title. Laurence Sterne, The Life and Opinions of Tristram Shandy, Gentleman, nine volumes, London, 1759–67; Robert K. Merton, On the Shoulders of Giants: A Shandean Postscript, Free Press, 1965. Why the post borrows the form is footnote 5.

Hinton's interview. "First reactions", telephone interview with Adam Smith, October 2024, nobelprize.org.

The committee's document. Scientific Background to the Nobel Prize in Physics 2024: "For foundational discoveries and inventions that enable machine learning with artificial neural networks", The Nobel Committee for Physics, Royal Swedish Academy of Sciences, PDF. The reference numbering, the "parallel development" sentence and the "reinvented" passage are all from it; the count of eleven is mine.

Merton. "Obliteration by incorporation" is defined in his own words in his 1980 Citation Classic commentary on Social Theory and Social Structure, PDF, which cites the book (Free Press, 1968) at pp. 25–38. Also: "Singletons and Multiples in Scientific Discovery", Proceedings of the American Philosophical Society 105(5), 1961, 470–486; "The Matthew Effect in Science", Science 159(3810), 1968, 56–63, PDF; On the Shoulders of Giants: A Shandean Postscript, Free Press, 1965.

The naming. Ronald R. Kline, "Cybernetics, Automata Studies, and the Dartmouth Conference on Artificial Intelligence", IEEE Annals of the History of Computing 33(4), 2011, 5–16 — Kline quotes McCarthy from his 1988 review of Bloomfield's The Question of Artificial Intelligence, Annals of the History of Computing 10, p. 227; Kline's larger account is The Cybernetics Moment, or Why We Call Our Age the Information Age, Johns Hopkins, 2015. The field's own retreat from the word is Ronald R. Kline, "Where Are the Cyborgs in Cybernetics?", Social Studies of Science 39(3), 2009, 331–362, pp. 353–354, quoting W. Grey Walter, "Neurocybernetics", in J. Rose, ed., Survey of Cybernetics, 1969, p. 94; Yehoshua Bar-Hillel, Language and Information, 1964, p. 11; John R. Pierce, Symbols, Signals and Noise, 1961, p. 228; and MacKay's letter to von Foerster of 1959, Kline's note 34. The cyborg's history is the same paper; its loci are footnote 10. Minsky's three avenues: M. Minsky, ed., Semantic Information Processing, MIT Press, 1968, p. 7, quoted at Kline 2011 p. 6; Kline's reading of it is at p. 13.

The perceptron controversy. Mikel Olazaran, "A Sociological Study of the Official History of the Perceptrons Controversy", Social Studies of Science 26(3), 1996, 611–659, doi 10.1177/030631296026003005.

Grossberg. "Competitive Learning: From Interactive Activation to Adaptive Resonance", Cognitive Science 11, 1987, 23–63. His recent book is Your Creative Brain and AI, Oxford University Press, 2026.

Rosenblatt. The facsimile is Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, Spartan Books, 1962, pp. 25 and 64 — the work the Nobel committee cites as its reference [3]; the "Ref. 3" on both pages is Ashby's Design for a Brain, 1952. The "deus ex machina" and "discomfort level" passages are p. 571 (§26.4, "Mechanisms of Motivation"). Public-domain scan, Google-digitized from the University of Michigan copy, HathiTrust.

Dawkins. Richard Dawkins, "When Dawkins met Claude: Could this AI be conscious?", UnHerd, 1 May 2026, link, and "When Claudia met Claudius", UnHerd, 4 May 2026, link — the sycophancy admission is its footnote 1.

Specification gaming. Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike and Shane Legg, "Specification Gaming: The Flip Side of AI Ingenuity", DeepMind blog, April 2020; "in accordance with the user's intention" is Ouyang et al. 2022, cited below, quoting Leike et al. 2018.

The two definitions. Eliezer Yudkowsky, "Efficient Cross-Domain Optimization" and "Measuring Optimization Power", LessWrong, October 2008. Shane Legg and Marcus Hutter, "Universal Intelligence: A Definition of Machine Intelligence", Minds and Machines 17(4), 2007, 391–444, arXiv:0712.3329.

The feedback loop. Arturo Rosenblueth, Norbert Wiener and Julian Bigelow, "Behavior, Purpose and Teleology", Philosophy of Science 10(1), 1943, 18–24. The update rule: A.-L. Cauchy, "Méthode générale pour la résolution des systèmes d'équations simultanées", Comptes Rendus 25, 1847; Bernard Widrow and Marcian Hoff, "Adaptive Switching Circuits", IRE WESCON Convention Record, 1960. The optimiser named in the text is Adam: Diederik Kingma and Jimmy Ba, "Adam: A Method for Stochastic Optimization", arXiv:1412.6980, 2015.

The multiple. W. Ross Ashby, review of G. Sommerhoff, Analytical Biology, Journal of Mental Science 98(412), July 1952, 488–489, doi 10.1192/bjp.98.412.488; the footnote lists the three as Ashby, "Adaptiveness and Equilibrium", Journal of Mental Science 86, 1940, 478; Design for a Brain, then in the press; and Rosenblueth, Wiener and Bigelow, 1943. G. Sommerhoff, Analytical Biology, Oxford University Press, 1950.

Ashby. An Introduction to Cybernetics, Chapman & Hall, 1956, archive.org — the regulator's essential function is §10/6; the law of requisite variety is Chapter 11, pp. 202–218. "Design for an Intelligence-Amplifier" (1956), "Principles of the Self-Organizing System" (1962), the classroom handout "Ashby Says", and "Cybernetics Today and Its Future Contribution to the Engineering Sciences" (address to the Foundation for Instrumentation Education and Research, New York, 1961; the perpetual-motion passage is p. 331 of the reprint, the teaching passages pp. 332–333) are collected in Roger Conant, ed., Mechanisms of Intelligence: Ross Ashby's Writings on Cybernetics, Intersystems, 1981. "Design for a Brain", Electronic Engineering 20, 1948, 379–383. Design for a Brain, 2nd ed., Chapman & Hall, 1960, §8/8, pp. 110–113 — the trainer inside the system and the step-mechanisms confined to the animal. "Setting Goals in Cybernetic Systems", in H. W. Robinson and D. E. Knight, eds., Cybernetics, Artificial Intelligence and Ecology, Spartan, 1972.

Where the concept lives in engineering. Jean-Pierre Aubin, Viability Theory, Birkhäuser, 1991; Thomas Sheridan, Telerobotics, Automation, and Human Supervisory Control, MIT Press, 1992. Neither is quoted; both are named as fields.

Dreyfus and Winograd. Hubert Dreyfus, "Why Heideggerian AI Failed and how Fixing it would Require making it more Heideggerian", Philosophical Psychology 20(2), 2007, 247–268 — the micro-world definition and the Winograd account are both there, and the KRL quotation is Winograd's own, from "Artificial Intelligence and Language Comprehension", National Institute of Education, 1976, p. 9. Terry Winograd and Fernando Flores, Understanding Computers and Cognition: A New Foundation for Design, Ablex, 1986; the conversation-for-action diagram with its two closing acts is Terry Winograd, "A Language/Action Perspective on the Design of Cooperative Work", Human-Computer Interaction 3(1), 1987–88, 3–30, Figure 1. The nurse and the three quotations on relevance: Terry Winograd, "Thinking Machines: Can There Be? Are We?", in J. Sheehan and M. Sosna, eds., The Boundaries of Humanity: Humans, Animals, Machines, University of California Press, 1991, 198–223, §5.2, text. Lawrence Page, Sergey Brin, Rajeev Motwani and Terry Winograd, "The PageRank Citation Ranking: Bringing Order to the Web", Stanford InfoLab technical report, 1998.

Learning, then and now. Arthur Samuel, "Some Studies in Machine Learning Using the Game of Checkers", IBM Journal of Research and Development 3(3), 1959, 210–229, PDF. Leslie Valiant, "A Theory of the Learnable", Communications of the ACM 27(11), 1984, 1134–1142, PDF.

The guessing game. Claude E. Shannon, "Prediction and Entropy of Printed English", Bell System Technical Journal 30(1), 1951, 50–64.

The observer and the correction channel. Claude E. Shannon, "A Mathematical Theory of Communication", Bell System Technical Journal 27, 1948, §12, Fig. 8 and Theorem 10; in Shannon and Weaver, The Mathematical Theory of Communication, Illinois, 1949. Ashby, "Induction, Prediction, and Decision-Making in Cybernetic Systems", in Kyburg and Nagel, eds., Induction: Some Current Issues, Wesleyan, 1963; Conant reprint p. 322.

The scalar, and the score. John von Neumann and Oskar Morgenstern, Theory of Games and Economic Behavior, Princeton, 1944; Ralph Bradley and Milton Terry, "Rank Analysis of Incomplete Block Designs", Biometrika 39, 1952, 324–345; Paul Christiano et al., "Deep Reinforcement Learning from Human Preferences", arXiv:1706.03741, 2017; Long Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback", arXiv:2203.02155, 2022 — the "misaligned" sentence is the introduction's second paragraph, the contractors §3.4, the reward model §3.5, the agreement rate §5, the labeling instructions Appendix B.2; Nisan Stiennon et al., "Learning to Summarize from Human Feedback", arXiv:2009.01325, 2020 — the over-optimization passage quoted in the text is §4.3; Ray Solomonoff, Inductive Inference Research Status, Spring 1967, final report to Air Force Cambridge Research Laboratories, AFCRL-67-0462, 1967, §6, "The problem of the ambitious subordinate", pp. 10–11; the effect is measured systematically in Leo Gao, John Schulman and Jacob Hilton, "Scaling Laws for Reward Model Overoptimization", arXiv:2210.10760, 2022; Rafael Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model", arXiv:2305.18290, 2023.

The two results. Yang Yue et al., "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?", arXiv:2504.13837, 2025 — quotations from v5, 24 November 2025. Shaohang Wei et al., "Verifier-Induced Support Reshaping in On-Policy Optimization", arXiv:2608.00220, 2026 — Peking University and BUPT; the KL sweep is §4.3, the shortcut audit Appendix C.2.

The admission. OpenAI, "Sycophancy in GPT-4o: what happened and what we're doing about it", 29 April 2025, and "Expanding on what we missed with sycophancy", 2 May 2025 — archived copies held at `Research/Anthropomorphism/`.

Kuhn loss. Thomas Kuhn, The Structure of Scientific Revolutions, Chicago, 1962, and the 1969 postscript; the term is discussed in Heinz Post, "Correspondence, Invariance and Heuristics", Studies in History and Philosophy of Science 2(3), 1971, 213–255.

The author's working papers, all at rsorourke.com/publications: The LLM as Variety Transducer (Kybernetes, published); Rebuilding Ashby's Amplifier; 70 Years Ago Ross Ashby Designed an 'Intelligence-Amplifier'. Have We Finally Built It?; The Machine That Outplayed Its Designer; The Status of Requisite Variety, Revisited; and, in preparation, a paper on how the machine's half of the loop, which the alignment field calls corrigibility, and the sponsor's half meet in Ashby's apparatus.

Footnotes

  1. W. Ross Ashby (1903–1972). Born 6 September 1903 in Brockley, south London; Edinburgh Academy, Sidney Sussex College, Cambridge, and St Bartholomew's; qualified in medicine in 1928, the year he opened the first volume of the journal he kept for the rest of his life; clinical psychiatrist for the London County Council 1930–36; research pathologist at St Andrew's Hospital, Northampton, 1936–47, with army service in India 1945–47; Director of Research at Barnwood House Hospital, Gloucester, 1947–59, where the Homeostat was finished in March 1948 and both books were written; Director of the Burden Neurological Institute, Bristol, 1959–60; Professor at Heinz von Foerster's Biological Computer Laboratory, University of Illinois, 1961–70; returned in 1970 to Westons, the disused village school at Tockington he and Rosebud had bought in 1959, and died there on 15 November 1972. George Klir met him at the Namur congress in 1970 and complimented him on the Introduction, saying that writing seemed to come easily to him. "'No,' he replied, 'I had to learn it the hard way.'" Klir's account: "He put the partially completed manuscript aside and started to study English. After more than two years of rather intensive study of the language, he finally returned to his work on the book. He did not use any part of the previously written manuscript but started completely from scratch; the whole book was completed in a few months." The journal tells it with dates and a different book. In April 1947 he noted the need to improve his style of writing; in March 1949 he enrolled in a correspondence course with the Regent Institute in Palace Gate and "almost two years later gained a Diploma in Effective English and Personal Efficiency"; those were the years of Design for a Brain, finished in December 1950, and the decision to write the Introduction is dated Christmas Eve 1952, "following a suggestion from Dad." Klir's recollection carries the fact and the journal carries the dates. Among his papers his widow found this: "I am something of an artist, not with pencil or paint, for I have no skills there, but with a deep appreciation of the perfect... I have an ambition someday to produce something faultless." Asked what should be done with his research notes, Conant reports, he said "Destroy it all," to give the next generation a chance for rediscovery. The notes were the journal: 7,189 pages in 25 volumes with 1,600 index cards, the last entry written in March 1972. Stafford Beer wrote to his widow, "Look after Ross's papers. I have no idea what should be done with them, but they are very precious," and for thirty years only the family saw them. In the late 1990s Peter Asaro, reading von Foerster's correspondence in the Illinois archives, met "an anecdote, or rumor, that Ashby had, on his deathbed, ordered his wife Rosebud to destroy the notebooks," thought it out of character, and from 1998 wrote to students, colleagues, archivists and unrelated Ashbys without finding a trace. In 1999 the Principia Cybernetica site put the Introduction online as a PDF, and its editor put Asaro in touch with the grandsons, Mick and John Ashby; the daughters, Jill, Sally and Ruth, had kept everything. The family scanned the pages, deposited the originals in the British Library in January 2003, announced the journal on the last day of the centenary conference at Urbana in March 2004, and in 2008 put every page online at ashby.info, cross-linked by his own index. Sources: Ashby's 1972 autobiographical summary and Jill Ashby's biography, both at ashby.info; George J. Klir, Foreword, p. i, Roger Conant's obituary reprinted from International Journal of General Systems 1(1), 1974, 4–7, and the preface page, in Conant, ed., Mechanisms of Intelligence, Intersystems, 1981; Peter M. Asaro and George J. Klir, "Introduction", International Journal of General Systems 38(2), 2009, 93–95, doi 10.1080/03081070802614239. Wikipedia.

    ↩
  2. Norbert Wiener (1894–1964), mathematician: a Harvard doctorate in mathematical logic at nineteen, then MIT for the rest of his career. With Arturo Rosenblueth and Julian Bigelow he wrote "Behavior, Purpose and Teleology" (1943); Cybernetics: or Control and Communication in the Animal and the Machine followed in 1948 and The Human Use of Human Beings in 1950. The naming, in the 1948 introduction: "We have decided to call the entire field of control and communication theory, whether in the machine or in the animal, by the name Cybernetics, which we form from the Greek κυβερνήτης or steersman." Wikipedia.

    ↩
  3. Warren Sturgis McCulloch (1898–1969), neurophysiologist: Yale's laboratory of neurophysiology 1934–41, director of the Illinois Neuropsychiatric Institute to 1951, MIT's Research Laboratory of Electronics from 1952. With Walter Pitts he wrote "A Logical Calculus of the Ideas Immanent in Nervous Activity" (1943), and he chaired the Macy conferences on Cybernetics. His 1961 lecture is titled "What is a number, that a man may know it, and a man, that he may know a number?" (General Semantics Bulletin; collected in Embodiments of Mind, MIT Press, 1965). Wikipedia.

    ↩
  4. The four were attending the Colloque International sur les Machines à Calculer et la Pensée Humaine, Institut Blaise Pascal, Paris, 8–13 January 1951, organised by Louis Couffignal for the CNRS. The photograph first appeared in Pierre de Latil, La Pensée Artificielle: Introduction à la Cybernétique, Gallimard, 1953, captioned "Les quatre pionniers de la Cybernétique se retrouvent à Paris", and in the English edition, Thinking by Machine, 1956, from which the Ashby family's archive at ashby.info reproduces it. No photographer is named in any source; Philip Husbands and Owen Holland reproduce it as Figure 1 of "Warren McCulloch and the British Cyberneticians", Interdisciplinary Science Reviews 37(3), 2012, "with permission".

    ↩
  5. Walter Pitts (1923–1969) read Principia Mathematica in a library at twelve, over three days, and wrote to its co-author Bertrand Russell about its errors; Russell wrote back and invited him to study at Cambridge. At fifteen he left home for Chicago. McCulloch met him there, still homeless, and took him into his family's house in Hinsdale, where the two worked in the evenings; the 1943 paper came out of it. Lettvin introduced Pitts to Wiener and he went to MIT. In 1952 Wiener's wife Margaret told Wiener that McCulloch's "boys" had seduced their daughter — Conway and Siegelman's account is that she invented it — and Wiener telegraphed MIT: "Please inform [Pitts and Lettvin] that all connection between me and your projects is permanently abolished." He never told Pitts why. McCulloch took a post at MIT the same year, Gefter writes, "because it meant he would be working with Pitts again." Pitts burned his dissertation and his notes, withdrew, and died on 14 May 1969, alone in a Cambridge boarding house, of bleeding oesophageal varices; McCulloch died four months later. Amanda Gefter's book on Pitts: "The Man Who Tried to Redeem the World with Logic", Nautilus, 2015; Flo Conway and Jim Siegelman, Dark Hero of the Information Age: In Search of Norbert Wiener, the Father of Cybernetics, Basic Books, 2005.

    ↩
  6. Merton’s On the Shoulders of Giants (1965) is subtitled A Shandean Postscript: a book-length digression, in Sterne’s manner, chasing the origin of an aphorism about intellectual debt that had itself lost its origin (Newton 1676 → Burton → Diego de Estella → Bernard of Chartres c. 1126, via John of Salisbury). This post borrows the form: it starts with a man who doesn’t know what field he is in, follows the debt backwards through a bibliography, and finds the field that isn’t named.

    ↩
  7. Donald MacKay to Heinz von Foerster, 1959, on von Foerster lending his name to a fringe group: "It's for just this kind of reason that folk such as Gabor, Uttley, Cherry & I are chary of using the word 'Cybernetics' nowadays, and I do hope that the work of someone of your calibre won't lose some of the attention it deserves by this new connection." John Pierce, 1961: "few scientists would acknowledge themselves as cyberneticists, save perhaps in talking to those whom they regard as hopelessly uninformed." Bar-Hillel, 1964: the word had been "usurped there by overt or covert science-fiction." Grey Walter, 1969: "the disrepute which has accumulated around the term." All at Kline 2009, pp. 353–354, cited under Sources.

    ↩
  8. Ronald R. Kline, "Where Are the Cyborgs in Cybernetics?", Social Studies of Science 39(3), 2009, 331–362. Clynes and Kline's paper is "Cyborgs and Space", Astronautics, September 1960, 26–27 and 74–76; the mouse with the osmotic pump is its Fig. 1, reproduced at Kline p. 332; the symposium, the New York Times definition and "participant evolution" are Kline pp. 341–343, and the "circumventing" sentence p. 343. D. S. Halacy, Cyborg: Evolution of the Superman, Harper & Row, 1965, at Kline p. 349, where the Terminator comparison is Kline's own. Donna Haraway, "A Manifesto for Cyborgs", Socialist Review 80, 1985; the "ironic myth for political action" phrase and the encyclopedia substitution are Kline p. 333. N. Katherine Hayles, How We Became Posthuman, Chicago, 1999, pp. 118–119, quoted at Kline p. 334; "present-mindedness" p. 335; the analogy method and Ashby's clockwork mouse p. 351, Kline quoting the transactions of the ninth Macy conference, 1953, p. 73. Kline's own account of the field is that its cyberneticians "held multiple interpretations of their field," which is the palimpsest again: one word, many owners, one official history.

    ↩
  9. Kuhn's losses are within a discipline, from a predecessor paradigm to its successor. This one runs across disciplines: a field that builds learners never importing what a neighbouring field kept. The term is used here for the state, a distinction a framework cannot state, and not for Kuhn's mechanism. Kuhn, The Structure of Scientific Revolutions, 1962, and Post 1971, cited under Sources.

    ↩
  10. The nearest thing to a connection is Ze Shen Chin, Maurice Chiodo, Dennis Müller and Coleman Snell, "Reframing AI Loss of Control: What Control Is, How to Have It, How to Lose It", arXiv:2606.12442, 2026, who define control as "the ability to set plausibly attainable goals that are not a foregone conclusion, and reliably achieve those goals" (§2.2, p. 9) and build on Wiener, Ashby's regulation and requisite variety, and the Good Regulator theorem. It is about loss of control in the broad sense; it does not touch the control-evaluations agenda that carries the name, which is Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan and Fabien Roger, "AI Control: Improving Safety Despite Intentional Subversion", arXiv:2312.06942, 2023: a trusted model and a human budget of audits set against an untrusted one.

    ↩
  11. At Dartmouth on 23 July 1956 Ashby described the homeostat, and Ray Solomonoff wrote in his notes: "Problem 1). No memory of previous solutions. i.e. learns to handle A, then learns B -> then going back to A takes as much time as before." (Grace Solomonoff, Ray Solomonoff and the Dartmouth Summer Research Project in Artificial Intelligence, 1956, online account, 2016, p. 12, PDF.) Ashby had it on his own list of unsolved problems: "What methods does the brain use, in serial adaptation, to ensure that later adaptations do not over-write, and spoil, earlier adaptations?" (classroom handout, in Conant, ed., Mechanisms of Intelligence, 1981, p. 431). Solomonoff spent the rest of his working life on the answer, training sequences from 1962 and incremental learning from 1989, and in 2009 was still listing what was missing: "We need a good update algorithm… to find more complex regularities, a more general algorithm is needed" ("Algorithmic Probability: Theory and Applications", in Emmert-Streib and Dehmer, eds., Information Theory and Statistical Learning, Springer, 2009, p. 20). No citation in either direction between the two men has been found.

    ↩
  12. The made number turns out to be the old one in disguise. Rafailov and colleagues showed in 2023 that under the objective the industry optimises, the reward is identically β log π(y|x)/π_ref(y|x) plus a constant: the drop in the model's surprise relative to the model it started from, scaled by the dial. They titled the paper accordingly, "Your Language Model is Secretly a Reward Model." The reference model is inside the definition; the pull back toward it is not a term added to the reward but part of what the reward is. (Rafael Rafailov et al., arXiv:2305.18290, 2023, §4, Eq. 5.)

    ↩
  13. The four were attending the Colloque International sur les Machines à Calculer et la Pensée Humaine, Institut Blaise Pascal, Paris, 8–13 January 1951, organised by Louis Couffignal for the CNRS. The photograph first appeared in Pierre de Latil, La Pensée Artificielle: Introduction à la Cybernétique, Gallimard, 1953, captioned "Les quatre pionniers de la Cybernétique se retrouvent à Paris", and in the English edition, Thinking by Machine, 1956, from which the Ashby family's archive at ashby.info reproduces it. No photographer is named in any source; Philip Husbands and Owen Holland reproduce it as Figure 1 of "Warren McCulloch and the British Cyberneticians", Interdisciplinary Science Reviews 37(3), 2012, "with permission".

    ↩

Further reading