Supertzar, or the Hand of Doom

The sleep of reason produces utility monsters.

These days, microsite manifestos on AI and superintelligence seem to grow like mushrooms. The notion of “superintelligence” is just as rhizomatic, with offshoots reaching from fears of an implacable utility monster transforming the entire universe into paperclips all the way to joyful anticipation of post-scarcity utopia. Since not everyone who ever wrote about superintelligence is a complete crank, it is worth taking a look at whatever substantive core these arguments may have to see what sorts of philosophical commitments they entail.

The first mention of the concept in a technical publication seems to be in a 1966 paper “Speculations Concerning the First Ultraintelligent Machine” by the mathematician Irving J. Good. Good is famous for the Good-Turing probability estimator that originated in the wartime efforts at Bletchley Park, as well as for having served as a technical consultant to Stanley Kubrick during the making of 2001: A Space Odyssey. The bulk of Good’s paper is fairly technical, devoted to mathematical exposition of ideas underlying probabilistic information retrieval, formation of associations based on experience building on ideas of Hebb, and the like. However, the opening sentence says:

The survival of man depends on the early construction of an ultraintelligent machine.

The beginning paragraph of Section 2 also cranks it up to eleven right away:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind … . Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously.

The concept of “intelligence explosion” (which was articulated by Good in an earlier publication from 1962) is thus presented as a logical deduction from certain premises. In effect, Good presents a thought experiment, where we are to imagine a possible world which is just like ours in every respect except one: In this world, there exists “a machine that can far surpass all the intellectual activities of any man however clever.” In this world, then, an intelligence explosion is sure to take place simply as a consequence of the definition of ultraintelligence.

This thought experiment and its conclusion are the substantive core of all subsequent invocations of the idea. Its psychological appeal is undeniably powerful, but therein lies its potential weakness. To pinpoint this weakness, let us first inquire into the nature of this possible world and, in particular, whether Good’s definition of ultraintelligence does not do too much violence to all the usual ceteris paribus clauses one would have to invoke in order for the inference to the stated theory to be unproblematic. In other words, if we postulate this one small difference between Good’s world and ours, we better hope that it does not explode into a whole mess of differences that would make the background of the theory as strange and counterintuitive as the theory itself.

Small worlds vs. large worlds

The key distinction here is between small worlds and large worlds. In the context of statistical decision theory, this distinction was made by Leonard Jimmie Savage, one of the key thinkers associated with the subjective (or Bayesian) approach to statistics and also someone who had known Good rather well. In his 1951 book The Foundations of Statistics, Savage builds up the apparatus of Bayesian decision theory by first presenting an axiomatic framework based on two constructs: that of an act as a mapping from states of the world to consequences and that of a preference ordering among the various consequences. Starting from a list of axioms Savage defends as a reasonable foundation of decision theory from the personalist viewpoint (which he contrasts with the objective, or frequentist, viewpoint), he arrives at subjective probability and at the notion of expected utility maximization subject to the axioms of von Neumann and Morgenstern.

To all of this, Savage attaches a key caveat that it is only rational to apply all these constructs to “small worlds,” i.e., those in which we can always look before we leap. In other words, we know all the states of the world and can anticipate the consequences of all our acts. This is needed, for example, so that the notion of expected utility is well-defined. As Savage says,

though the “Look before you leap" principle is preposterous if carried to extremes, I would nonetheless argue that it is the proper subject of our further discussion because to cross one’s bridges when one comes to them means to attack relatively simple problems of decision by artificially confining attention to so small a world that the ‘‘Look before you leap" principle can be applied there. I am unable to formulate criteria for selecting these small worlds and indeed believe that their selection may be a matter of judgment and experience about which it impossible to enunciate complete and sharply defined general principles.

In other words, Savage argues that the concept of expected utility maximization only makes sense in a world where the decision-maker can take all conceivable information into account before choosing an act. All possible mistakes that could have been made will already have been corrected. There are “no alarms and no surprises” in small worlds. But, according to Ken Binmore, this qualification excludes the worlds of scientific inquiry, of microeconomics, and of high finance. As Binmore puts it in his 2009 book Rational Decisions,

in a large world, the possibility of an unpleasant surprise that reveals some consideration overlooked in Pandora’s¹ original model can’t be discounted. As Savage puts it, in a large world, Pandora can only “cross certain bridges when they are reached.” Knee-jerk consistency is then no virtue. Someone who insists on acting consistently come what may is just someone who obstinately refuses to admit the possibility of error. In brief, Savage agrees with Ralph Waldo Emerson that foolish consistency is the hobgoblin of small minds. Only when our small minds are encased in a small world does he regard consistency as an unqualified virtue.

The world Good invites us to imagine is just such a large world because his definition tacitly rolls up scientific inference, macroeconomics, finance, and psychology into one tangled mess. For one, the definition of an ultraintelligent machine as one that surpasses humans in any intellectual activity is too open-ended. Is it possible to enumerate all conceivable human intellectual activities in advance? From history, we know that the introduction of new technologies, machines, social-economic arrangements, and political systems unlocks new intellectual activities and interests by amplifying human abilities. Asking us to imagine a machine that surpasses humans in any such activity tacitly assumes that we can forecast any such developments. So, the world that differs from ours “only” in one respect (ultraintelligent machines exist in that world but not in ours) is not a small world in Savage’s sense, it is a large world with huge indeterminacy in just about any realm we can think of: sociology, economics, technology, science, psychology. Before we can even begin to deduce intelligence explosion, we are confronted with an explosion of differences that makes Good’s world utterly incomparable to ours. It has too many bridges we can only cross when we get to them.

Establishing the phenomenon

Fundamentally, Good’s thought experiment is not a scientific one, it is psychological. Thought experiments of this type are not uncommon in analytic philosophy, and this is the domain where analytic philosophy can often become indistinguishable from science fiction. The 1980 book Real People: Personal Identity without Thought Experiments by the philosopher Kathleen Wilkes begins with a careful analysis of what it means to set up a successful thought experiment, i.e., the one in which “the jump from data to theory is relatively small.” As examples of such successful thought experiments in physics, she cites Simon Stevin’s proof of the law of equilibrium for a frictionless chain lying on an inclined plane or Einstein’s argument that classical electromagnetism is incompatible with the theory of relativity. What makes these thought experiments successful, in Wilkes’ description, is that they succeed in establishing the phenomenon in one’s imagination. On the other hand,

precisely because the relevant background is adequately fixed, so that the result of the imagined state of affairs is immediately clear. Were it not so, the inference could not be ‘unproblematic’. By contrast, when we have thought experiments in philosophy, there are as we shall see problems in making the inference—precisely because of the ambiguous uncertainty concerning the relevant background conditions, leaving it unclear whether we have indeed ‘established a phenomenon’. This means that our intuitions run awry, and the inferences are not only problematic, but the ‘jump’ from the phenomenon to the conclusion is made the larger because of the further need to imagine just what these backing conditions, under the imagined circumstances, would be. The ‘possible world’ is inadequately described.

In other words, successful thought experiments can be about small worlds only, and it is precisely the impossibility of adequately describing the background of the large world that prevents the offered thought experiment from successfully establishing the phenomenon. As an example of this, Wilkes takes aim at Derek Parfit’s famous thought experiments:

Consider next one of the familiar thought experiments to do with personal identity: that we might all split like amoebae. It is obviously and essentially relevant to the purposes of this thought experiment to know such things as: how often? Is it predictable? Or sometimes predictable and sometimes not, like dying? Can it be induced, or prevented? Just as obviously, the background society, against which we set the phenomenon, is now mysterious. Does it have such institutions as marriage? How would that work? Or universities? It would be difficult, to say the least, if universities doubled in size every few days, or weeks, or years. Are pregnant women debarred from splitting? The entire background here is incomprehensible. When we ask what we would say if this happened, who, now, are ‘we’?

Good’s, and everyone else’s, visions of superintelligence are equally incomprehensible for more or less the same reasons. They fail to successfully establish the phenomenon because the definitions they take as self-evident are anything but self-evident. One way in which this is evident is the unwarranted intermixing of mathematical or logical arguments with psychological notions, such as desire or drive.

Psychologism and its discontents

As I already noted above, Good’s definition of ultraintelligence is unworkable because it discusses human intellectual activities in a timeless setting of Savage’s smal world. It assumes that human psychology is fixed and that the range of human intellectual activities can be circumscribed in advance. From this, Good argues, ultraintelligent machines will be able to bootstrap their abilities in an intelligence explosion. This sort of “recursive self-improvement” is now one of the core concepts taken for granted in all the discussions on AI risk. However, a more explicitly psychological argument was put forward in 2008 by Stephen Omohundro in his paper “The Basic AI Drives.” Omohunrdo, a mathematician and a physicist by training, spent a few years as faculty in the department of computer science at my university and, among other things, was one of the authors (together with Stephen Wolfram) of a 1987 technical report envisioning a “tablet personal computer of the year 2000.”

Omohundro starts from the same premises as Good, but invokes the idea of dominated strategy in game theory to argue for the inevitability of recursive self-improvement by advanced AI systems. In a more technical paper from 2007, he argues that a superintelligent machine would converge to von Neumann-Morgenstern rationality by eliminating suboptimalities. In mathematics, this is a familiar argument, where one can arrive at a putative optimum by an iterative process that finds a local improvement from the current configuration. When no further improvement is possible, we arrive at a fixed point and then, if we wish to show global optimality, we have to argue that this fixed point is unique. This, for example, is the structure of the Dutch book theorems in subjective probability: Starting from the requirement of coherence of probability assessment (in the sense of de Finetti), we argue that any candidate assessment that fails to be coherent can be revised to a coherent one by means of conditionalization. To this mathematical argument, Omohundro attaches a psychological construct of drives any such superintelligent machine will inevitably possess: Efficiency, Self-Preservation, Acquisition, and Creativity. However, as argued eloquently by Václav Beneš (who, before going on to a distinguished scientific career as an applied mathematician at Bell Labs, wrote a dissertation on Quine at Princeton), mathematical theorems about the possibilties and limitations of complex systems (either formal systems of mathematical logic or adaptive systems like neural nets) are devoid of empirical content about human psychology precisely because they are mathematical statements. Aiming his critique at John Myhill’s psychological interpretation of Gödel’s and Church’s theorems, Beneš writes:

The psychological interpretation suggested by Mr. Myhill … is purely incidental in the sense that the same argument could be used to support a mechanical, cybernetic, biological, or even theological interpretation.

Omohundro’s concept of drives is vulnerable to the same critique: It either does not admit a psychological interpretation (in which case it can be formalized without appealing to vaguely Freudian terminology, but then it applies to any technological system) or it does (but then it is purely incidental to the mathematics, and there is no need to appeal to the trappings of mathematical rigor to discuss it).

Leaving aside the mathematical content of Omohundro’s argument², the psychological appeal of these ideas is undeniable. But it is precisely the psychologism that undermines them because, once again, it forces us to give up a tidy small world where utility maximization makes sense in favor of a nebulous large world, where the relevant background is just as incomprehensible as the one in Parfit’s discussion about creatures that divide every autumn and fuse every spring.

The fables of subcreation

I am not arguing here against the need to envision possible catastrophic risks of advanced technologies and to use these analyses to influence policy. There is a need for such enlightened doomsaying. However, cloaking these discussions in mathematical arguments obscures their fundamental human-directedness. They are fables of what J.R.R. Tolkien termed “subcreation” in his essay “On Fairy-stories:”

Children are capable, of course, of literary belief, when the story-maker's art is good enough to produce it. That state of mind has been called “willing suspension of disbelief.” But this does not seem to me a good description of what happens. What really happens is that the story-maker proves a successful “sub-creator.” He makes a Secondary World which your mind can enter. Inside it, what he relates is “true”: it accords with the laws of that world. You therefore believe it, while you are, as it were, inside.

Such fables are simultaneously more powerful and more honest than thought experiments because, on the one hand, they are explicitly designed to provide a reflection of our psychological drives and constitution and, on the other, they are incredibly effective at establishing the phenomenon in Wilkesian sense. Rather than plowing through the X-risk literature, one can find much better discussion of the profound dangers of unfettered technocratic control in Tolkien’s Lord of the Rings or in C.S. Lewis’ That Hideous Strength. I would go so far as to say that the construction of fables and their hermeneutics is what makes continental, rather than analytic, philosophy the right framework for discussing the human-directed risks of technology, such as when Jean-Pierre Dupuy focuses on the antihumanist aspects of cybernetics.

1. In accordance with the classics, Pandora is Binmore’s moniker for an archetypal decision-maker.

2. Elliott Thornley argues that Omohundro’s argument has mathematical flaws. (Yes, I am aware that I am linking to Alignment Forum here.) In addition to this, for any large-scale decentralized decision-making system, as any assemblage of “ultraintelligent machines” will necessarily be, there are also computational (in)tractability issues involved in solving the problem of information structure design (that is, finding the right observation channels and avoiding deadlocks and other vicious circles) that has to be solved before the utility maximization problem can even be formulated.