Ashby's Variety & Shannon's Entropy - A History

Variety, in Ashby's sense, is the number of distinct elements in a set, or the logarithm of that number. The definition is in chapter 7 of An Introduction to Cybernetics (1956), and the law of requisite variety is built on it. In this post we explore where the word and the definition came from. We start with Shannon. Then follow with the theory of sets, from Bourbaki by way of Jacques Riguet. And finally, what he wrote in his journal. The three are taken in the order in which they happened, from 1948 to 1956. This history supplements the companion post on the law itself.

Shannon and the Cyberneticians¶

A footnote in Part III of Shannon's famous paper, in the text as it was reprinted in 1949, reads:

"Communication theory is heavily indebted to Wiener for much of its basic philosophy and theory. His classic NDRC report, The Interpolation, Extrapolation, and Smoothing of Stationary Time Series (Wiley, 1949), contains the first clear-cut formulation of communication theory as a statistical problem, the study of operations on time series. This work, although chiefly concerned with the linear prediction and filtering problem, is an important collateral reference in connection with the present paper. We may also refer here to Wiener's Cybernetics (Wiley, 1948), dealing with the general problems of communication and control."

He reviewed Wiener's book in 1949 and congratulated him on "an excellent introduction to a new and challenging branch of science."

Ashby took the name of his subject from the same book. The Introduction opens: "Cybernetics was defined by Wiener as 'the science of control and communication, in the animal and the machine'—in a word, as the art of steermanship, and it is to this aspect that the book will be addressed." Where it reaches communication it names the two together, "the theory developed by Shannon and Wiener," and in chapter 8 says that "after the pioneer work of Shannon and Wiener" certain general laws are known to hold over every instrument that codes a message. The one place he separates them: Shannon's entropy is the sum of p log p "multiplied by –1 whereas the definition given by Wiener in his Cybernetics for 'amount of information' is the same sum of Pi log Pi unchanged (i.e. multiplied by +1)." He draws two rulers to show the two readings "are clearly equivalent": Wiener measures the gain in information when a distribution narrows, Shannon the uncertainty that remains.

Shannon was also in correspondence with Warren McCulloch, who chaired the Macy conferences on Cybernetics. On 2 June 1949 he thanked McCulloch for some reprints and described his own laboratory:

"There is considerable activity here in information theory, computing machines and other 'Cybernetic' questions, and a fresh viewpoint is very stimulating."

McCulloch spent that summer in Europe, and in October he reported back:

"I have been lecturing at an incredible rate in France, England, and Scotland to groups of psychiatrists, electro-encephalographers, neurophysiologists, engineers, logicians, mathematicians, and even anatomists, to say nothing of neurosurgeons. It was always on the same general theme of the circuit action of the nervous system, sometimes on its servo aspects, sometimes on its digital aspects."

He had also been to Manchester: "I wish you had been with me the day I spent in Manchester with Turing and Jones. Williams was unfortunately away. I had always thought of his tubes for memory as something which existed on paper. It was a delight to see them in action."

Shannon gave papers at two of the Macy conferences: "The Redundancy of English" in March 1950 and "Presentation of a Maze-Solving Machine" in March 1951. At the 1951 meeting, in the discussion of a paper by Donald MacKay, he said:

"it seems to me that we can all define 'information' as we choose; and, depending on what field we are working in, we will choose different definitions. My own model of information theory, based mainly in entropy, was formed precisely to work with the problem of communication. Several people have suggested using the concept in other fields where in many cases a completely different formulation would be more appropriate."

Ashby gave his papers on the homeostat and the mechanical chess player at the next meeting, in March 1952. Shannon was not there.

McCulloch and Ashby had met the year before, in January 1951, when both were in Paris for the Colloque "Les Machines à Calculer et la Pensée Humaine," where a photograph was taken of Ashby, McCulloch, Grey Walter and Wiener together, printed in Pierre de Latil's book of 1953 as "the four pioneers of Cybernetics." Kline reports that McCulloch met Ashby at the Ratio Club "on a visit to England in 1951," and that after the 1952 meeting he wrote to Donald MacKay: "Ross Ashby did a superb job of presenting complicated affairs simply and clearly and has made for himself many friends and admirers, even among those who heckled him most at the meeting." The hecklers' question was whether the homeostat learned. Kline's account is that "Pitts and McCulloch sided with Ashby, because if Shannon's Theseus learned, so did the homeostat." Wiener, who "privately questioned Ashby's mathematical ability," wrote in the 1954 edition of The Human Use of Human Beings that "Ashby's brilliant idea of the unpurposeful random mechanism which seeks for its own purpose through a process of learning is not only one of the great philosophical contributions of the present day, but will lead to highly useful technical developments in the task of automatization."

Shannon made the same journey a year after McCulloch, and in November 1950, back from two months in Europe, wrote:

"I was amazed at the amount of activity there, especially in England, on Information Theory, Cybernetics, computing machines, and the like. The programming techniques on the Edsac at Cambridge are probably in advance of anything in this country."

McCulloch had gone to lecture, and his subject was the nervous system as a circuit. The word Cybernetics is not in his letter: he says "servo" and "digital". Shannon reports what he found others doing, and names three fields. Each of them ends at a British computer, McCulloch at the memory tubes in Manchester and Shannon at the EDSAC in Cambridge.

Between Shannon's two letters the quotation marks have gone and the word has gained a capital. I read the first as a label borrowed for work in his own laboratory and the second as the name of a field set beside his own.

The traffic ran the other way too: Ashby had Shannon's vocabulary before he had any set theory. In the chess paper of 1952 he wrote that

"Shannon and Descartes can agree that 'a noiseless transducer or determinate machine can emit only such information as is supplied to it.'"

The transducer is Shannon's term for a system whose next state is determined by its present state and its input, and in the Introduction Ashby says his own machine with input "is identical with the 'transducer' of Shannon." Four years after the chess paper the two were in one volume: Shannon co-edited Automata Studies with John McCarthy, and Ashby's paper on the intelligence-amplifier is in it. Shannon had by then praised Ashby's homeostat in print, in 1953, "as a basis for learning machines and brain models."

In March 1956 his editorial "The Bandwagon" warned that information theory owed part of its publicity to "connections with such fashionable fields as computing machines, cybernetics, and automation." The next year he wrote the Encyclopaedia Britannica article on Cybernetics, at Wiener's suggestion, and said in it that Cybernetics "overlaps" information theory. The review, the Macy remark, the homeostat sentence, the editorial and the article are found in the historian Ronald Kline's engaging book: The Cybernetics Moment: Or Why We Call Our Age the Information Age (2015).

How Ashby came to "variety"¶

Ashby kept a journal from 1928 to 1972, and his family has put every page online with his own index. It shows the word being found, and then the law, and in what order.

On 4 July 1952, six months before the word, he writes N for "the number of distinct states" in a set and proves two theorems: under one transformation the number cannot grow, and under a second transformation it cannot grow either. These are two results that the Introduction states in chapter 7, the decay of variety and the law of Experience. His comment: "The proofs of both demand nothing more than the ability to count."

The word becomes a term on 30 December 1952. The entry is headed "On the flow of 'variety' from system to system," and it gives his reason in a bracket:

"I want to get away from the Shannon method of entropies + averaging over infinitely long messages; I want something I can count."

He then proves that a machine driven by another can show no more lines of behaviour than its driver, and adds: "This theorem seems to be the analogue of Shannon's Theorem 7, about the noiseless transducer." The theorem is the one the Introduction calls the fundamental law of the transmission of variety, and the comparison with Shannon is his own.

The step to sets is dated 2 May 1953, two weeks before the letter that reports Bourbaki's books in hand:

"If I assume, as I do, that all my entities, or systems & sub-systems, are determinate, then anything that can have variety must be represented by a set, not an individual."

His summary of the entry is "Anything that can have variety is really a set." He did not take sets from Bourbaki and then count their elements. He was counting, saw that only a set can be counted, and found in Bourbaki the mathematics for it. That's my reading of the dates.

How Ashby came to sets¶

The Introduction defines variety as a property of a set. Ashby did not begin there. Writing in 1958 about another author who, he said, "did what I did in 1952—used the mathematical language of analysis and continuous functions," he went on:

"This language now seems unnecessarily clumsy and artificial; for it has been found (Ashby, 1956) that the concepts of set theory, especially as expounded by Bourbaki (1951), are incomparably clearer and simpler, while losing nothing in rigour. By the change to set theory, nothing in fact is lost, for nothing prevents the elements in a set from being numbers, or the functions from being continuous, and the gain in generality is tremendous."

Nicolas Bourbaki was the pen name, chosen in 1935, of a group of French mathematicians who had decided to write a treatise together, with no credit to any individual. A contemporary called Bourbaki "the polycephalic mathematician", the many-headed one. Armand Borel, a member for twenty years, described it as "a truly unselfish, anonymous, demanding work by people striving to give the best possible exposition of basic mathematics, moved by their belief in its unity and ultimate simplicity," and recalled a plan, at one stage, for "twenty-seven books, encompassing most of mathematics." The group gave its reason in a manifesto of 1948. Mathematics had grown so far, it said, that one had to ask

"whether the domain of mathematics is not becoming a tower of Babel, in which autonomous disciplines are being more and more widely separated from one another, not only in their aims, but also in their methods and even in their language."

Bourbaki's answer, in the historian Leo Corry's summary, was to present "the whole picture of mathematical knowledge in a systematic and unified fashion, within a standard system of notation, addressing similar questions, and using similar conceptual tools and methods in the different branches." The theory of sets was to be the common ground. The treatise is called the Éléments de mathématique, in the singular, and Borel says the missing "s" was "one way for Bourbaki to signal its belief in the unity of mathematics." Ashby put the result in one sentence in 1960:

"this school has shown how the theory of sets, in a simple basic form, can be gradually extended and developed, without the least loss of precision or the least change in the fundamental concepts, into the realms of topology, algebra, geometry, theory of functions, differential equations, and all the various branches of mathematics."

He came to it in the spring of 1953, through the French mathematician Jacques Riguet, whose calculus of relations he had just read. His letters to Riguet are in his family's archive. On 19 May 1953 he wrote:

"I now have Bourbaki's Theory of Sets, and also his Algebra. All this is very new to me, but I am sure that this branch of mathematics, much more general than the numerical and linear, is going to be of the highest importance in the abstract theory of machines and of mechanisms in the brain."

He was reading it in French, and in May 1953 the only part of the Theory of Sets in print was the summary of results, a booklet of statements without proofs first published in 1939. The first two chapters followed in 1954 and the English translation in 1968. His references are to the French editions, and the French words stayed with him: "une application" for a mapping in a report of 1962, "échelle" in his journal in 1956.

I think the appeal is not hard to see, though he does not state it. He wanted for the sciences of the brain what Bourbaki wanted for mathematics. Chapter 1 of the Introduction says of Cybernetics:

"it offers a single vocabulary and a single set of concepts suitable for representing the most diverse types of system. Until recently, any attempt to relate the many facts known about, say, servo-mechanisms to what was known about the cerebellum was made unnecessarily difficult by the fact that the properties of servo-mechanisms were described in words redolent of the automatic pilot, or the radio set, or the hydraulic brake, while those of the cerebellum were described in words redolent of the dissecting room and the bedside—aspects that are irrelevant to the similarities between a servo-mechanism and a cerebellar reflex."

The letter of 19 May 1953 also joins his machine to Shannon's:

"I was very excited by seeing in Bourbaki's Algebra, Chapter I, a definition of a 'law of external composition' that is exactly what I consider to be the essence of a 'machine' and exactly what Shannon, in his theory of information, defines as a 'noiseless transducer' (he means, roughly, a determinate machine)."

Within a month of opening Bourbaki's books he had put his own machine and Shannon's transducer under one definition. Eighteen months later he told Riguet that the book he was then writing, the Introduction, "will contain no set theory as such, though it will be based on those concepts." In 1959 he wrote to Riguet that "the work in Cybernetics particularly is crying out for a text that, as far as I can see, only you and I together can write." The book was never written; one paper was. In 1961 the two published together, in the first volume of the Journal of Theoretical Biology, "The Avoidance of Over-writing in Self-Organizing Systems".

He was still working at it in the summer of 1956. On 31 July, a week after his talk at the Dartmouth conference, he wrote in his journal aboard the S.S. Maasdam: "What is 'structure'? Bourbaki has undoubtedly given the answer, but what does it mean in my terms?"

In the Introduction variety is the number of elements in a set. Its constraint is a restriction to a subset, and in 1967 he said whose idea that was:

"Bourbaki has shown, especially in the section 'Echelles d'ensembles et structures', that restriction to a subset is always the essential operation that generates properties, relations, patterns, structures"

Bourbaki had a dispute of its own about foundations, over category theory. It was devised between 1942 and 1945 by Samuel Eilenberg and Saunders Mac Lane, and Eilenberg became a member of the group. A chapter on categories was commissioned for the treatise, and in Corry's words "the promised chapter on categories never appeared as part of the treatise." In 1957 another member, Alexander Grothendieck, pressed the group to put categories into the foundations. Borel, a member at the time, records the verdict: it was "rather clear that if we followed that route, we would be bogged down with foundations for many years, with a very uncertain outcome." The treatise stayed on its set-theoretic foundations. Pierre Cartier, a member from 1955, said in a 1997 interview that Grothendieck left the group in anger over it; I have seen that interview only in an unverified copy, and give the reference in the Sources.

Categories and Cybernetics have since been joined from the other side. In a paper of 2021 four category theorists, Matteo Capucci, Bruno Gavranović, Jules Hedges and Eigil Fjeldgren Rischel, proposed "a categorical framework for processes which interact bidirectionally with both an environment and a 'controller'," and wrote: "We believe that 'cybernetic' is an appropriate name for the processes that can be described in this framework."

I would put the relation the other way round. It is Cybernetics that is categorical, and Ashby's letter to Riguet of October 1954 is my evidence. He defines his machine in one line:

"I define the machine (E) with input (I) as a mapping of I x E in E, letting the states E be the output without further mapping into output symbols."

To couple it to a second machine, "which may be an observer," he adds "a special mapping c" from the states of one to the inputs of the other. A machine that shows its state outward and takes an input back to its next state is what the categorical literature calls a lens, the simplest of the bidirectional structures that the 2021 paper names cybernetic. Joining machines by such mappings is composition. Ashby had asked Riguet the year before, "Is it possible that an algebra is the proper representation of a machine, or of a mechanism in the brain?" Category theory is an algebra of things joined by mappings. The identification of his machine with a lens, and the claim built on it, are mine.

How the law came, in his journal¶

The law comes in three entries, the first on 12 February 1953, where he proves it in Shannon's terms and sums up: "Regulation (destruction of information) requires extra information." On 3 November 1953 he proves it again by counting, and the summary reads: "Only variation can force variation down." On 15 April 1954 he strips it down:

"the theorem of p. 4658 is best torn right away from all ideas about machines & shown for what it is — a property of certain rectangular tables."

The summary of that entry is the earliest place I have found the name: "Law of Requisite Variety stated with respect to a rectangular table." Of these three, the proof in entropies came before the proof by counting, which is the reverse of the order in the Introduction. The counting argument is older than both. It is one half of a theorem he wrote out in January 1953, giving the limits between which a set's variety can move in one step. He did not call that the law, then or later, and the identification is mine.

The link to Theorem 10 is the last piece, made at Stanford late in 1955 by redrawing Shannon's diagram, and the summary of the entry reads: "Law of Requisite Variety as law for suppression of noise." What the counting and the theorem have to do with each other, and how far the correspondence holds, is the subject of the companion post, "Ashby's Variety & Shannon's Entropy".

Sources¶

Shannon and the Cyberneticians. The Wiener footnote is in Part III of "A Mathematical Theory of Communication", attached to the sentence on Wiener and Fourier theory; it is quoted from the 1949 book edition. Shannon to Warren S. McCulloch, 2 June 1949 and 30 November 1950, and McCulloch to Shannon, 20 October 1949, Warren S. McCulloch Papers, Mss.B.M139, American Philosophical Society, Philadelphia; read from the Society's digitised page images. The Macy papers are listed in Claus Pias, ed., Cybernetics: The Macy Conferences 1946–1953. The Complete Transactions, Diaphanes, 2016. Ashby, "Can a Mechanical Chess-Player Outplay its Designer?", British Journal for the Philosophy of Science 3(9), 1952, 44–57; An Introduction to Cybernetics, §8/5 (p. 143), for the transducer. C. E. Shannon and J. McCarthy, eds., Automata Studies, Princeton, 1956. Ronald R. Kline, The Cybernetics Moment: Or Why We Call Our Age the Information Age, Johns Hopkins University Press, 2015: Shannon's 1949 review of Cybernetics p. 95; his remark at the 1951 Macy meeting p. 60, from the conference transactions; his absence in 1952 p. 61; the homeostat p. 156, from Shannon's "Computers and Automata", 1953; "The Bandwagon", IRE Transactions on Information Theory 2(1), 1956, p. 3, as quoted at p. 103; the Britannica article p. 83. Kline prints the 2 June 1949 letter at p. 57 without the word "very"; the text above follows the letter. McCulloch and Ashby: the Ratio Club p. 52; McCulloch to MacKay, 23 April 1952, and Wiener on the homeostat, p. 55; Pitts and McCulloch on the homeostat p. 53. The Paris photograph: Pierre de Latil, La Pensée Artificielle, Gallimard, 1953, English edition Thinking by Machine, 1956; reproduced in Philip Husbands and Owen Holland, "Warren McCulloch and the British Cyberneticians", Interdisciplinary Science Reviews 37(3), 2012, 237–253, Fig. 1, and at ashby.info. Ashby on Wiener: An Introduction to Cybernetics, §1/1 (p. 1), §7/5 (p. 123), §8/5 (p. 143), §9/14 (pp. 177–179).

Variety, in his journal. W. Ross Ashby's journal, pp. 3959–3960 (4 July 1952), 4312–4313 (30 December 1952), 4336–4337 and 4346–4347 (January 1953), 4385–4387 (12 February 1953), 4474 (2 May 1953), 4657 and 4658–4659 (3 November 1953), 4846–4847 (15 April 1954) and 5242, with his index cards for "Variety" and "Requisite Variety, Law of", at the W. Ross Ashby Digital Archive, © The Estate of W. Ross Ashby. The extracts are my readings of his handwriting. The January 1953 pages carry no date and lie between entries dated 1 and 14 January; the entry on p. 5242 faces one dated 19 December 1955.

How Ashby came to sets. Ashby to Jacques Riguet, 19 May 1953 and 29 October 1954, page images at the W. Ross Ashby Digital Archive, © The Estate of W. Ross Ashby. Ashby's journal, p. 5305, 31 July 1956, at the same archive. Bourbaki's manifesto is "L'architecture des mathématiques", 1948; English translation, "The Architecture of Mathematics", American Mathematical Monthly 57(4), 1950, 221–232, p. 221, quoted here from Corry, p. 319. Corry p. 320 for his summary and for the publication dates of the Theory of Sets; the English translation is Bourbaki, Theory of Sets, Hermann, 1968. "Une application" is in Ashby's "The Set Theory of Mechanism and Homeostasis", 1962. An Introduction to Cybernetics, §1/6 (p. 4), for the single vocabulary. Matteo Capucci, Bruno Gavranović, Jules Hedges and Eigil Fjeldgren Rischel, "Towards Foundations of Categorical Cybernetics", Proceedings of Applied Category Theory 2021, EPTCS 372, 2022, 235–248, arXiv:2105.06332 — the abstract. The 1958 passage is from "Requisite Variety and Its Implications for the Control of Complex Systems", as above, pp. 192–193 of the Conant reprint. The 1960 sentence is Design for a Brain, 2nd ed., Chapman & Hall, 1960, §19/1. The 1967 sentence is "The Place of the Brain in the Natural World", Currents in Modern Biology 1(2), 1967, 95–104, p. 98. Armand Borel, "Twenty-Five Years with Nicolas Bourbaki, 1949–1973", Notices of the American Mathematical Society 45(3), 1998, 373–380, PDF — the pen name p. 374, the description p. 376, the twenty-seven books p. 377, Grothendieck's proposal and the verdict p. 378. Leo Corry, "Nicolas Bourbaki and the Concept of Mathematical Structure", Synthese 92(3), 1992, 315–348 — category theory and the unpublished chapter p. 332. Grothendieck's departure: Marjorie Senechal, "The Continuing Silence of Bourbaki: An Interview with Pierre Cartier, June 18, 1997", The Mathematical Intelligencer 20(1), 1998, 22–28, doi 10.1007/BF03024395, read only in an unverified web copy. W. Ross Ashby and J. Riguet, "The Avoidance of Over-writing in Self-Organizing Systems", Journal of Theoretical Biology 1(4), 1961, 431–439, doi 10.1016/S0022-5193(25)00259-000259-0). Ashby to Riguet, 5 June 1959, at the archive page above. "The polycephalic mathematician" is André Delachet's phrase, quoted by R. P. Boas in 1949, as reported in Michael J. Barany, "Impersonation and personification in mid-twentieth century mathematics", History of Science 58(4), 2020, 417–436, doi 10.1177/0073275320924571.

The Introduction. W. Ross Ashby, An Introduction to Cybernetics, Chapman & Hall, 1956, archive.org: §1/6 (p. 4) for the single vocabulary, §7/7 (p. 126) for the definition of variety, §7/24–7/25 (pp. 136–139), §8/5 (p. 143) for the transducer, §8/11 (pp. 152–153), §8/16 (p. 158) for Bourbaki and Riguet, §11/11 (pp. 210–211) for Theorem 10.

Scope. This post is history. What the law of requisite variety says, and how it stands to Shannon's Theorem 10, is the subject of the companion post, "Ashby's Variety & Shannon's Entropy".