Ashby's Variety & Shannon's Entropy

Ross Ashby ends his 1958 paper on requisite variety by naming two things together:

"The law of requisite variety, and Shannon's theorem 10, in setting a limit to what can be done, may mark this era as the law of conservation of energy marked its era a century ago."

That is a large claim to hang on a theorem and on a word, variety, that he uses in a sense very much his own. I wanted to know what the two have to do with each other. The answer in An Introduction to Cybernetics is that they are two limits, proved separately, which he sets side by side and calls homologous. Shannon's theorem comes first below, since it is the shorter of the two to state. Then the book's order: the four chapters that take the reader from counting to regulation, the law, and the pairing of the two.

Theorem 10¶

Claude Shannon's "A Mathematical Theory of Communication" appeared in the Bell System Technical Journal in 1948. Part II treats the channel with noise, and its section 12 introduces the equivocation, the uncertainty that remains about what was sent once you know what was received. To give that quantity a meaning Shannon adds a second channel:

"We consider a communication system and an observer (or auxiliary device) who can see both what is sent and what is recovered (with errors due to noise). This observer notes the errors in the recovered message and transmits data to the receiving point over a 'correction channel' to enable the receiver to correct the errors."

Then the theorem:

"Theorem 10: If the correction channel has a capacity equal to H_y(x) it is possible to so encode the correction data as to send it over this channel and correct all but an arbitrarily small fraction ε of the errors. This is not possible if the channel capacity is less than H_y(x)."

The correction channel's capacity must be at least the equivocation. Enough capacity and almost every error can be undone; less and it cannot. It is a bound in both directions.

Warren Weaver, introducing the paper to a general readership in 1949, put the equivocation in plain words. It

"measures the average uncertainty in the message when the signal is known. If there were no noise, then there would be no uncertainty concerning the message if the signal is known. If the information source has any residual uncertainty after the signal is known, then this must be undesirable uncertainty due to noise."

The theorem says how much of that undesirable uncertainty a second channel can take away.

Weaver also set out three levels at which the problem of communication can be posed:

"LEVEL A. How accurately can the symbols of communication be transmitted? (The technical problem.) LEVEL B. How precisely do the transmitted symbols convey the desired meaning? (The semantic problem.) LEVEL C. How effectively does the received meaning affect conduct in the desired way? (The effectiveness problem.)"

Shannon's theory, he said, "although ostensibly applicable only to Level A problems, actually is helpful and suggestive for the level B and C problems." Theorem 10 is a Level A result: it is about symbols arriving accurately. What Ashby does with it, in the sections below, is to read it at Level C. His regulator does not correct a message; it corrects conduct, the departure of a variable from where it is wanted. He said so in 1961: Shannon's theorems on the correction of noise "are fundamentally applicable to all processes of control, the correction of noise now becoming the correction of any deviation from what is desired, from the goal."

The two belong together because they describe one operation from its two ends. Weaver says that a theory built for the accurate transfer of symbols will prove useful for the question of how what is received affects conduct, and does not say how. Ashby does. His regulator is a Level C device, and when he maps Theorem 10 onto it every term changes level. In his words, Shannon's noise "corresponds to our 'disturbance'," his correction channel "to our 'regulator R'," and his message of entropy H "becomes, in our case, a message of entropy zero, for it is constancy that is to be 'transmitted'." The book Ashby names for his notation is the 1949 volume, in which Weaver's essay is the first thirty pages. Whether the theorem still holds once its terms have been renamed for a regulator is the question, and the section below on The word "homologous" attempts to answer it.

From counting to regulation¶

Below are key extracts from chapters of An Introduction to Cybernetics (1956), the textbook Ashby wrote for readers with no more than school mathematics. The book has three parts:

• Part I, Mechanism, treats the single machine;
• Part II, Variety, chapters 7 to 9, treats sets of possibilities and their transmission;
• Part III, Regulation and Control, chapters 10 to 14, is where the law is stated, in chapter 11.

Ashby gives the reason for the order at the head of Part III:

"The two previous Parts have treated of Mechanism (and the processes within the system) and Variety (and the processes of communication between system and system). These two subjects had to be studied first, as they are fundamental. Now we shall use them."

The four chapters below are the ones the law draws on directly. Chapter 7 defines variety; chapter 8 says what a machine does to it; chapter 9 turns the count into Shannon's entropy; chapter 10 says what regulation is. The approach has been to present it in his words as much as possible, with what it contributes to the law.

Chapter 7: a set, and its variety¶

The variety of a set is "either (i) the number of distinct elements, or (ii) the logarithm to the base 2 of the number." Before the definition Ashby shows why there has to be a set at all. Communication, he says, "necessarily implies the existence of a set of possibilities, i.e. more than one, as the following example will show."

"A prisoner is to be visited by his wife, who is not to be allowed to send him any message however simple. It is understood that they may have agreed, before his capture, on some simple code. At her visit, she asks to be allowed to send him a cup of coffee; assuming the beverage is not forbidden, how is the warder to ensure that no coded message is transmitted by it? He knows that she is anxious to let her husband know whether or not a confederate has yet been caught.

The warder will cogitate with reasonings that will go somewhat as follows: 'She might have arranged to let him know by whether the coffee goes in sweetened or not—I can stop that simply by adding lots of sugar and then telling him I have done so. She might have arranged to let him know by whether or not she sends a spoon—I can stop that by taking away any spoon and then telling him that Regulations forbid a spoon anyway. She might do it by sending tea rather than coffee—no, that's stopped because, as they know, the canteen will only supply coffee at this time of day.' So his cogitations go on; what is noteworthy is that at each possibility he intuitively attempts to stop the communication by enforcing a reduction of the possibilities to one—always sweetened, never a spoon, coffee only, and so on. As soon as the possibilities shrink to one, so soon is communication blocked, and the beverage robbed of its power of transmitting information. The transmission (and storage) of information is thus essentially related to the existence of a set of possibilities. The example may make this statement plausible; in fact it is also supported by all the work in the modern theory of communication, which has shown abundantly how essential, and how fruitful, is the concept of the set of possibilities."

His conclusion is the sentence the rest depends on:

"Communication thus necessarily demands a set of messages. Not only is this so, but the information carried by a particular message depends on the set it comes from. The information conveyed is not an intrinsic property of the individual message."

The next section adds the observer: how many elements a set has depends on who is telling them apart. That sentence is quoted further down.

Chapter 8: what a machine does to variety¶

Chapter 7 has already said that many copies of one machine, all run under a single transformation, cannot spread further apart:

"as time progresses the variety in the set cannot increase and will usually diminish."

Change the input, making the same change to every copy, and the count still cannot rise:

"change of parameter-value to the whole set cannot increase the set's variety."

He names this the law of Experience, and puts it in a sentence:

"information put in by change at a parameter tends to destroy and replace information about the system's initial state."

Chapter 8 adds a second machine, T, driving the first, U, and asks how far the copies of U can then be driven apart. The answer:

"the U's cannot gain in variety at one step by more than the variety present in the T's. This is the fundamental law of the transmission of variety from system to system."

The gain is capped as well:

"once U has increased in variety by the amount in T, all further increase must cease."

Read together: an input changed alike for every copy cannot raise the count, and an input that differs from copy to copy can raise it by no more than the variety in the input. Shannon has the analogue in his own terms, as Ashby's journal entry of 1952 noticed: his Theorem 7 says the output of a finite-state transducer has entropy "less than or equal to that of the input." I will come back to the word "transmission," because Ashby does.

Chapter 9: entropy, a count weighted by probability¶

Chapter 9 begins by saying why the counting of chapter 8 is not enough:

"If the transmission is to go on for an indefinitely long time, the variety must be sustained, and therefore not like the case studied in S.8/11, in which T's transmission of variety stopped after the first step. Now any determinate system of finite size cannot have a trajectory that is infinitely long."

So he admits a transformation that is not single-valued, under one restriction:

"It is the unchangingness of the probability that provides the law or orderliness on which definite statements can be based."

That is the Markov chain, and entropy is defined on it. He then asks where the probabilities come from, and the answer is a procedure:

"If a set has variety, and we take a sample of one item from the set, by some defined sampling process, then the various possible results of the drawing will be associated with various, corresponding probabilities."

A set of traffic lights, on for 25, 5, 25 and 5 seconds, is found by a motorist who turns up at random times in its four states about 42, 8, 42 and 8 per cent of the time,

"if this particular method of sampling be used."

Shannon's entropy is then

"a measure for the quantity of variety shown by a Markov chain at each step," and "information cannot be transmitted in larger quantity than the quantity of variety allows."

When the probabilities are equal, H

"is then equal to log n, precisely the measure of variety defined in S.7/7."

Variety is the count; entropy is the count weighted by a sampling process; information is what a source transmits, and it is bounded by the count.

Ashby said as much in 1965, agreeing with a suggestion of R. B. Banerji:

"information theory is basically just counting, and simply a branch of combinatorics."

Shannon's skill, he went on, lay

"not in inventing a new philosophy or a new mystery but in showing how the counting could be extended into cases that would quite defeat the counting methods of the bank-teller."

Chapter 10: regulation is the blocking of transmission¶

Chapter 10: the regulator's essential function "is that it shall block the transmission of variety from disturbance to essential variable." Chapter 8 counted how many of the differences at one machine can still be told apart at the next. A regulator is a machine placed between the disturbances and the essential variables so that, whatever the disturbance does, the essential variables show as few distinct states as possible, and in the ideal case one.

Setting the law beside the theorem¶

The law by counting¶

Chapter 11, the law itself, from a game played on a table of outcomes, with a row for each disturbance and a column for each of the regulator's moves. Ashby takes the hardest case first, "only those tables in which no column contains a repeated outcome," where the table itself absorbs none of the disturbance and the regulator must meet all of it. There a regulator with n distinct moves can cut the outcome's variety to an nth of the disturbance's, and no lower: "only variety in R can force down the variety due to D; variety can destroy variety." He then gives the general case. Where each outcome is repeated k times in a column, the same argument holds with the repetition counted as slack: the outcome's variety is at least the disturbance's, less log k, less the regulator's.

The same theorem in Shannon's currency¶

Still chapter 11:

"what is essentially the same theorem will be proved in the case when the variety is spread out in time and the fluctuation incessant—the case specially considered by Shannon."

The disturbance, the regulator and the outcome are treated as three sources. The hardest case then becomes a statement about their entropies: with the regulator's move given, the outcome is at least as uncertain as the disturbance. Then "whatever the causal or other relations between D, R and E, algebraic necessity requires that their entropies must be related" so that the outcome's entropy has a floor: at best, the disturbance's entropy less the regulator's. The slack of the general case enters here as well, as a term K subtracted from the floor. The law in counts becomes the law in bits by the sampling step of chapter 9.

The regulator is a channel¶

"The law of Requisite Variety says that R's capacity as a regulator cannot exceed R's capacity as a channel of communication." This is the sentence that turns a table game into a statement about information.

And the channel is Shannon's¶

The same section names Theorem 10, "which says that if noise appears in a message, the amount of noise that can be removed by a correction channel is limited to the amount of information that can be carried by that channel. Thus, his 'noise' corresponds to our 'disturbance', his 'correction channel' to our 'regulator R', and his 'message of entropy H' becomes, in our case, a message of entropy zero, for it is constancy that is to be 'transmitted'. Thus the use of a regulator to achieve homeostasis and the use of a correction channel to suppress noise are homologous."

He drew it in his journal in 1955 and printed it in 1958. His figure 3 is Shannon's figure 8 with the letters changed.

The word "homologous"¶

A coding theorem beside a count¶

Theorem 10 is a coding theorem. It is about a rate over long sequences, an error fraction that can be made as small as you like, and a code that has to exist to achieve it. Requisite variety in its counting form is about one act, exact, with no code. Ashby does not call the two identical. His word is for a correspondence of structure. His journal shows he knew where the difficulty lay. On 3 November 1953, having proved the law by counting, he noted that "The extension to Shannon's entropy does not go very easily if the input states (of x₁) have various probabilities." What he had that is exact is the entropy form, the inequality of chapter 11, which with its slack term holds by algebra for any three sources. And the channel sentence has a theorem behind it that is not his. Roger Conant, his student, proved in 1969 that the regulation a regulator achieves is bounded by its channel capacity plus a constant, which is the sentence in mathematics. In summary: Theorem 10 is the picture Ashby drew, the entropy form is his exact form, and Conant's theorem is the proof of the channel sentence. The entropy form and Conant's theorem do not depend on the analogy. Only the picture does, and the picture differs from Shannon's at two points, taken next.

The quantity¶

The picture differs from Shannon's first in the quantity. Theorem 10 bounds the correction channel by the equivocation, the uncertainty about the message once the signal has been received. If the message has entropy zero, that uncertainty is zero too, and the theorem asks nothing of the channel. What Ashby bounds by is the entropy of the disturbance. In 1958: "the amount of noise that can be prevented from appearing in the outcomes is limited to the entropy that can be transmitted through the correction channel." In his journal at Stanford, on the page where he first drew the figure: "Then Shannon's theorem says that if Noise has entropy N, the output can have zero entropy only if R's capacity is at least N." Shannon's theorem does not say that; the substitution of the disturbance's entropy for the equivocation is Ashby's. What it yields is his own inequality of chapter 11, the entropy of the outcome at least that of the disturbance less that of the regulator, with the slack for what T absorbs, now in Shannon's vocabulary. Theorem 10 supplies the picture and not the bound. That conclusion is drawn here from the two statements; neither man states it.

Where the correction channel taps the line¶

It differs second in where the correction channel taps the line, and the two figures show it. In Shannon's the observer sees the message sent and the message received, and what goes down the correction channel is made from the difference. In Ashby's the channel leaves the line where the noise enters, before the main channel, so the regulator sees the disturbance itself.

The boxes do not match one to one either. Shannon has four: the observer that sees, the channel that carries, the correcting device that acts, and the main channel the noise enters. Ashby has two. R sees and carries; T is the main channel and the correcting device in one, for it is in the table that R's move and D's meet to give the outcome.

How Shannon's noisy channel can become a table at all, Ashby says in the same chapter. The case "when T is 'noisy'—when T has an extra input that is affected by some disturbance that interferes with it" seems at first outside the formulation. His answer is to move the boundary: "the boundaries should be re-drawn so as to get T's input of noise (S.9/19) included as a component in D … and T has no third input, so the formulation agrees with that of S.11/4." The noise becomes part of the disturbance, and what is left on the line is a table from inputs to outcomes, known by experiment and nothing else: "Experiments can only provide such tables." That is how he reads any system, and it is what Fig. 3 does to Fig. 8.

Ashby's two regulators¶

That is the regulator of chapter 11, which takes its input from D. Klaus Krippendorff took Ashby's course at Illinois in 1962–63. In his 2009 paper on Ashby's information theory he draws the two kinds side by side as his first figure: "when regulators pick up the disturbances before they affect the essential variables, anticipatory regulation, and when regulators pick up the effects of the disturbances on the essential variables, error-controlled regulation, which involved a feedback loop." Roger Conant, in the 1969 paper cited above, calls them cause-controlled and error-controlled. He proves what Ashby had asserted in 1958, that the error-controlled regulator is "fundamentally incapable of being 100 percent efficient." Except when the disturbance is itself predictable, "error-controlled regulators cannot succeed in maintaining the outcome sequence constant since a constant sequence carries no information."

Conant is careful about what the wiring does and does not touch. His inequalities, he says, "have not depended in any way on where R got its information, i.e., on what affected R." The bound is the same in both figures: regulation cannot exceed what R knows about D, by whatever route it knows it. What the wiring decides is how much R can know. Fed from D, it can know all of D and hold E constant. Fed from E, it knows D only through the error, and the better it regulates the less error there is to learn from, so a constant E would leave it knowing nothing. The bound is wiring-blind; whether a regulator can get the information to meet it is not, and that is why Krippendorff draws both. The regulator that resembles Shannon's observer is the error-controlled one, the one that cannot have the information a perfect regulation needs. The figure Ashby drew to match Shannon's is of the other kind.

A second redrawing¶

Krippendorff's second figure is a second redrawing of Shannon's system, after Ashby's Fig. 3 and unlike it: the correction channel R leaves at the sender and ending at a box T where the received signal and the correction meet, which is the placing of the correcting device read off above. He adds, without saying by whom, that "Today, Shannon's 10th Theorem is considered a special case of Ashby's Law of Requisite Variety."

Two places where the argument is thinner than it looks¶

The first is that the entropy form is provable from definitions. A statement true by algebra says nothing about the world by itself. What makes requisite variety say something is the partition: which variables are the disturbance, which the regulator, which the essential variables, and what their permitted range is. Ashby put the last of these first.

"The essential variables E are given, and also given is the set of states η in which they must be maintained if the organism is to survive … These two must be given before all else. Before any regulation can be undertaken or even discussed, we must know what is important and what is wanted."

For an organism they are fixed by survival, and his example is the species: "the cat must keep itself dry, the fish must keep itself wet." For a firm they are fixed by law: insolvency ends it. For a language model they are fixed by nobody until a sponsor assigns them. The partition of disturbance and regulator into states is then the observer's, but not arbitrary: two states need stay distinct only if they call for different responses. The test of a partition comes afterwards, in whether the regulator built on it holds the line. Ashby makes the drawing of the boundaries a method, and spends six sections on it (§11/16–11/21): several disturbances at once, a noisy T, an uncontrolled initial state, a target with more than one condition, several regulations interacting, each brought inside the table by letting D, T or E be a vector, with the warning: "There is, of course, no suggestion here that the noise, as a disturbance, can be allowed for magically by merely thinking differently about it." The table is general because the boundaries move; what the inequality then says depends on where they were put. One boundary it cannot absorb. In Design for a Brain Ashby gives "a butterfly and a bird in the air, the bird chasing the butterfly, and the butterfly evading the bird," and says of them: "The bird has as environment the air and the butterfly, while the butterfly has the air and the bird. The whole may reasonably be assumed to be state-determined." He uses the pair to show feedback, and says nothing there about who wins. In the Introduction he comes back to it, at the end of chapter 12, as the case where disturbance and response "occur alternately": "a prey attempts to regulate against an attack by a predator, when the whole struggle progresses through alternating stages of threat and parry … The outcome will depend on some relation between the predator's whole attack and the prey's whole response." In the table of chapter 11 the disturbance plays first and the regulator replies knowing its move; here neither plays first, and each is the other's disturbance. He names where the answer would have to come from: "In its real form it is the Battle of Life; in its mathematical form it is the Theory of Games and Strategies," von Neumann's, "not yet fully developed," and "already too extensive for more than mention here." He does draw the case, as the last figure of the book: a machine with several players, "each trying to achieve a goal in G, working simultaneously, and interacting competitively within M. (The possibility of competition between regulators has not been considered explicitly in these chapters till now.)" The drawing is as far as it goes; there is no inequality for it. The inequality can be written for the bird against the butterfly and for the butterfly against the bird, and both hold, but neither says which regulation suffices on the day, because each animal's variety is spent against a disturbance that adapts to it. The 1958 paper opens by admitting the case to the law's domain, "The disturbances may be actively hostile, as are those coming from an enemy, or merely irregular, as are those coming from the weather," and refers both to "the formalism that is already well known in the theory of games." Conant excluded the case in so many words, "situations, such as games, in which S can change its behavior so as to oppose the regulation," and Ashby's list of unsolved problems has it in one line: "Apply Shannon's tenth theorem (law of requisite transmission) to game theory." Control theory has since reached the worst-case opponent and the regulator informed about its disturbance under other names, minimax design and side information, without citing Ashby or Conant, and the opponent that learns under none that I can find. The regulators in this post all face a disturbance that does not fight back. A paper of mine now in revision argues that the law's whole empirical content lives at that level and that the inequality is a limit of the second-law kind, an inequality and not a conservation. Ashby's energy analogy holds by role, one accepted limit beside another, not by form. Why he reached for the first law and not the second, which has the form of his own, he does not say, but the book shows two reasons. The second law's history has no story of engineers giving up a hope, and the moral he drew every time was that story: "every erg of energy that came out had first to be got in. For a time many engineers felt profoundly disappointed." And the second law carries an entropy, which he had told his readers to keep apart from Shannon's: "Sometimes the second law of thermodynamics is appealed to, but this is often irrelevant to the systems discussed here" (§7/24), and any attempt "to play loosely, and on a merely verbal level, with the two entropies of Shannon and of statistical mechanics" is "like moving in a jungle full of pitfalls" (§9/13). The first law gave him the moral without the entropy. The reasons are inferred; the quotations are his.

The second is that Shannon's theorems carry assumptions. The entropy form, the channel sentence and Theorem 10 are in Shannon's terms, and Ashby says in chapter 9 what those terms take for granted:

"Shannon's measure, and the various important theorems that use it, make certain assumptions. These are commonly fulfilled in telephone engineering but are by no means so commonly fulfilled in biological work, and in the topics discussed in this book."

He lists three. The probabilities must form a complete set. The source must be one whose next step depends only on its present state. And the system must have "been allowed to go on for a long time so that the states have reached their equilibrial densities." The law by counting needs none of them, because it counts and uses no probabilities. The entropy form, the channel sentence and Theorem 10 need all three, and the second is the Markov chain of chapter 9. Ashby's instruction follows: "Shannon's results must therefore be applied to biological material only after a detailed check on their applicability has been made." Shannon said much the same about his own theory to the Macy group in 1951: it "was formed precisely to work with the problem of communication." In his journal he had been blunter. In January 1953, within a fortnight of the entry that wanted "something I can count," he wrote: "It is now clear that most of Shannon's technical methods are of little use to me." He said as much in public in 1961, telling an audience of engineers that quantities of information should be measured "either combinatorially or by the method of McGill and Garner (as being more suitable than the specialized methods of Shannon)." What he kept from Shannon was the theorems, as a limit. None of this was doubt about the theory. Considering in 1968 whether Shannon's methods were adequate for psychology, psychiatry and sociology, Ashby recorded "my opinion that it is sufficient."

Variety, entropy, information¶

Variety is a property of a set. It is fixed by the observer's distinctions, and it equals the number of distinct elements, or the logarithm of that number. Ashby's rule for it:

"a set's variety is not an intrinsic property of the set: the observer and his powers of discrimination may have to be specified if the variety is to be well defined."

Entropy, in Shannon's sense, is a property of a set of probabilities over the set. It is fixed by a defined sampling process, or by the equilibrium of a running source. It equals the count weighted by probability, and the logarithm of the count when the probabilities are all equal. Ashby's rule for it: the word is used "solely as it is used by Shannon."

Information is a property of a source or a channel. It is what the source transmits, up to the channel's capacity, so it is at most the variety and at most the capacity. Ashby's rule for it: it "cannot be transmitted in larger quantity than the quantity of variety allows."

One trap runs through all of this, and Ashby marks it in chapter 8: any attempt

"to treat variety or information as a thing that can exist in another thing is likely to lead to difficult 'problems' that should never have arisen."

No quantity passes. There are counts at the input and counts at the output, and the second cannot exceed the first. His own chapter is titled "transmission," so the trap is set beside the warning.

A machine of today¶

Is a language model one of Ashby's machines? Ask it of the network alone and the answer is yes. Given its weights and the tokens so far, a forward pass returns one distribution over the next token, the same every time: a machine with input, which Ashby says "is identical with the 'transducer' of Shannon, which is defined as a system whose next state is determined by its present state and the present values of its parameters." Ask it of the model as used and the answer changes, because the token is not read off the distribution but drawn from it, and the drawing takes a random number. Set the temperature to zero and the most probable token is always taken; the whole is determinate. Raise it and the next token is a draw, Ashby's coin of chapter 9: "the transformation and the initial state are not sufficient to define a unique trajectory … they define only a set of trajectories. The definition given here is supplemented by instructions from the coin."

He had a name for the machine that results. Chapter 12 defines it: "a new class of absolute system: it is one whose states change with time not by a single-valued transformation but by a matrix of transition probabilities," the Markovian machine, with one condition, that "the values of the probabilities must be unchanging." The determinate machine is "the extreme form of a Markovian machine in which all the probabilities have become either 0 or 1," and between the two: "If the probabilities are all very near to 0 or 1, we get a machine that is almost determinate in its behaviour but that occasionally does the unusual thing." (The 1956 printing sets the digits in these two sentences as the letters O and I; they are given here as digits.) Temperature is the knob that moves a model along that line. What the knob changes is not the machine but what is being counted. A determinate machine has no variety of its own: "variety cannot exist in a transducer (at any given moment), for a particular transducer at a particular moment is in one, and only one, state." A model sampled at temperature is a set of determinate machines, one for each draw of the random number, and the variety of its outputs is the set's. He had made that his working rule in May 1953, in his journal: "If I assume, as I do, that all my entities, or systems & sub-systems, are determinate, then anything that can have variety must be represented by a set, not an individual." Which set, and whether to speak of the set or of a member, is the observer's choice, and that is all "probabilistic" means here. The condition is what a deployed model keeps and a retrained one breaks: fine-tune it, or change what it is given before the conversation, and the matrix is another matrix. By his rule that is not the same machine doing something new; it is a new machine.

Which of the two descriptions is right, determinate or probabilistic, is for him a question of where the observer draws the line, and he says so in the same chapter: "Whether a given real machine appears Markovian or determinate will sometimes depend on how much of the machine is observable." Put the random number inside the box and the model is Markovian. Put it outside, as one more input, and it is a determinate machine with a noisy input, which is the redrawing of §11/18 again. My Kybernetes paper takes the second route, listing the draw as an input to each step and temperature as the setting that fixes how much of the learned variety reaches the output.

The Markovian machine also regulates, in its own way, and this is where the work now in progress starts. Ashby's name for the method is "hunt and stick": the machine wanders from state to state until it reaches one at which it stops, and "Movement to a goal by the process of hunt and stick is thus homologous … to movement by a determinate trajectory for both are the movement of a machine to a state of equilibrium." His examples are a flypaper, a golfer looking for a lost ball, and a rat that cannot remember the maze but stops at food. A model sampled at temperature, with a checker that ends the run when an output passes, is this machine. The paper in development asks what such a checker does to the set of outputs the model can be made to produce, and finds Ashby's answer in the sections used above: the checker fixes a partition, and what the model can be rewarded for is relative to that partition and not a property of the model alone.

The two observers¶

The argument begins with an observer and ends with one. Chapter 7 needs an observer to fix the distinctions that are counted. Theorem 10 needs an observer "who can see both what is sent and what is recovered," and it is that observer's data, sent down the correction channel, that the theorem measures. Ashby's regulator is that observer with a job: it sees the disturbance, it sees the essential variables, and what it can do about the difference is limited by how many distinct responses it can make. That is why the law is stated in the currency of communication. A regulator is not a thing that has enough force. It is a thing that can tell enough cases apart, and has a distinct response for each.

Sources¶

Shannon. Claude E. Shannon, "A Mathematical Theory of Communication", Bell System Technical Journal 27, July 1948, 379–423, doi 10.1002/j.1538-7305.1948.tb01338.x (Parts I–II; Part II, §12 "Equivocation and Channel Capacity", Theorem 10 and Fig. 8), and October 1948, 623–656, doi 10.1002/j.1538-7305.1948.tb00917.x. A freely readable re-typesetting of the whole paper is at Harvard. Theorem 10 is at p. 68 of the 1949 book edition, Shannon and Weaver, The Mathematical Theory of Communication, University of Illinois Press. Warren Weaver, "Recent Contributions to the Mathematical Theory of Communication", in the same volume: the three levels §1.2 (p. 4), the equivocation §2.5 (p. 20), Level A "helpful and suggestive" §3.1 (p. 24).

Ashby. An Introduction to Cybernetics, Chapman & Hall, 1956, archive.org: §7/5 (pp. 123–124), §7/6 (p. 125), §7/7 (p. 126), §7/24–7/25 (pp. 136–139), §10/1 (p. 195), §8/11–8/12 (pp. 152–154), §9/2 (pp. 161–162), §9/11–9/13 (pp. 173–177), §10/6, §11/5–11/7 (pp. 204–207), §11/8–11/9 (pp. 207–209), §11/11 (pp. 210–211), §11/16–11/21 (pp. 216–218), §12/1. "Requisite Variety and Its Implications for the Control of Complex Systems", Cybernetica 1(2), 1958, 83–99; reprinted in Roger Conant, ed., Mechanisms of Intelligence: Ross Ashby's Writings on Cybernetics, Intersystems, 1981, freely readable at the W. Ross Ashby Digital Archive, and in George J. Klir, ed., Facets of Systems Science, Springer, 1991, doi 10.1007/978-1-4899-0718-9_28 — Fig. 3 and the closing sentence; "the amount of noise that can be prevented" and the error-controlled regulator at pp. 195–197 of the Conant reprint.

Variety, in his journal. W. Ross Ashby's journal, pp. 3959–3960 (4 July 1952), 4312–4313 (30 December 1952), 4336–4337 and 4346–4347 (January 1953), 4385–4387 (12 February 1953), 4474 (2 May 1953), 4657 and 4658–4659 (3 November 1953), 4846–4847 (15 April 1954) and 5242, with his index cards for "Variety" and "Requisite Variety, Law of", at the W. Ross Ashby Digital Archive, © The Estate of W. Ross Ashby. The extracts are my readings of his handwriting. The January 1953 pages carry no date and lie between entries dated 1 and 14 January; the entry on p. 5242 faces one dated 19 December 1955.

The bird and the butterfly. W. Ross Ashby, Design for a Brain, 2nd ed., Chapman & Hall, 1960, §3/10–3/11; An Introduction to Cybernetics, §12/22 (pp. 240–241), for the Battle of Life and the Theory of Games; Conant 1969 (above), §II, for the exclusion of games; "Some Unsolved Problems of Cybernetics", in Roger Conant, ed., Mechanisms of Intelligence, 1981, p. 431. Control theory: Tamer Başar and Pierre Bernhard, H∞-Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach, Birkhäuser, 2nd ed. 1995; Nuno C. Martins, Munther A. Dahleh and John C. Doyle, "Fundamental Limitations of Disturbance Attenuation in the Presence of Side Information", IEEE Transactions on Automatic Control 52(1), 2007, doi 10.1109/TAC.2006.887898; neither cites Ashby or Conant (deposited reference lists, checked 7 October 2026). The search for a treatment of the adapting opponent under requisite variety was run on OpenAlex, Semantic Scholar and OpenCitations the same day and found none.

Ashby's later statements. "Measuring the Internal Informational Exchange in a System", Cybernetica 8, 1965, 5–22, for "just counting". "The Contribution of Information Theory to Pathological Mechanisms in Psychiatry", British Journal of Psychiatry 114, 1968, 1485–1498, for "sufficient". "Cybernetics Today and Its Future Contribution to the Engineering Sciences", Foundation for Instrumentation Education and Research, New York, 1961, for McGill and Garner and for "the correction of any deviation" (Conant reprint pp. 328–329). All three are reprinted in Conant, ed., Mechanisms of Intelligence, at pp. 142–143, p. 386 and p. 333. Shannon's Theorem 7 is in Part I, §8, of the 1948 paper.

Conant. Roger C. Conant, "The Information Transfer Required in Regulatory Processes", IEEE Transactions on Systems Science and Cybernetics SSC-5(4), 1969, 334–338, doi 10.1109/TSSC.1969.300226 — equation (9); error-controlled and cause-controlled regulation, §§V–VI.

A machine of today. An Introduction to Cybernetics, §8/5 (p. 143), §9/2 (p. 162), §8/10 (pp. 151–152), §12/8–12/9 (pp. 225–226), §12/10–12/12 (pp. 227–232) for hunt and stick; Ashby's journal, p. 4474, 2 May 1953. The Kybernetes paper is "The LLM as Variety Transducer", Kybernetes, 2026, and the paper in development is on verifier-based training of language models, at rsorourke.com/publications when it appears.

The partition. The three-level account of the partition, and the classification of requisite variety as a second-law-kind bound rather than a conservation law, are argued in The Status of Requisite Variety, Revisited, in revision for Kybernetes; working version at rsorourke.com/publications/2607-dewhurst.

Scope. The argument stays inside chapters 7 to 11 of the Introduction and Theorem 10. How Ashby came by the word, the set theory under it and his dealings with Shannon are in the companion post, "Ashby's Variety & Shannon's Entropy - A History". Ashby's later calculus of interaction information for many-variable systems, and the account of what became of it in Klaus Krippendorff, "Ross Ashby's information theory: a bit of history, some solutions to problems, and what we face today", International Journal of General Systems 38(2), 2009, doi 10.1080/03081070802621846, are a different subject; the same paper's Figures 1 and 2 and the paragraph between them are used above for the two kinds of regulator.