The Paradox of Highly Optimized Tolerance

There really is no antimemetics division.

As 2025 is drawing to a close, I think I can finally articulate my thoughts on social and technological risks pertaining to AI. I am not talking about superintelligence explosion (because I don’t find these arguments coherent) and I don’t buy “this is just another moral panic” line either (in the spirit of process philosophy, I take Heraclitus’ “no man ever steps in the same river twice” fairly seriously). From the process philosophy viewpoint, the right framework for understanding the impact of AI is as a network of multilayered open systems residing in a complex environment. This viewpoint was already prefigured by William James in A Pluralistic Universe:

everything is in an environment, a surrounding world of other things, and ... if you leave it to work there it will inevitably meet with friction and opposition from its neighbors.

Alfred North Whitehead’s monumental Process and Reality presents what is probably the most fully articulated process ontology that elaborates upon this open-systems view. In the new year, I will devote a few posts to exploring Whitehead’s thought in the context of technology. Here, though, I want to present a more focused take which, nevertheless, contains a kernel of process-oriented thinking.

I am a techno-pragmatist. I view technology as a means of problem-solving rather than of truth-seeking. This design viewpoint will naturally “meet with friction and opposition” at every layer, as it should be. Some of this friction and opposition has roots in material reality, some is grounded in societal and political domains. Nevertheless, instrumental rationality à la Max Weber is at the core of the technological worldview, and here we encounter a very important feature: In any sufficiently complex open system, implementing instrumental rationality at a particular layer of abstraction will inevitably expose (or even create) vulnerabilities in other interconnecting layers. Abstraction hides a great deal of complexity from view, and this is both its main virtue and its primary peril.

For me, the relevance of this to AI was evident even if I couldn’t articulate it precisely. However, things really clicked in my mind as I was carefully studying the work of Jean Carlson and John Doyle on highly optimized tolerance. Their first paper on this, published in 1999 in Physical Review E, was meant to provide an alternative to the physics-based “science of complexity” associated with places like The Santa Fe Institute. According to the Santa Fe worldview, features like self-organization, criticality, and universality (characterized by things like power law behaviors) were as applicable to the brain, the global economy, and other sociotechnical infrastructures as they were to frustrated spin glasses and other varieties of statistical physics models. Carlson and Doyle, however, argued very convincingly that the statistical physics viewpoint overlooks the multilayered, hierarchical nature of both engineered and evolved systems. Thus, they proposed the notion of Highly Optimized Tolerance (or HOT, for short) as an alternative explanation for the appearance of power laws in designed, rather than self-organizing, systems. Their main motivating example was the Internet:

The Internet is one example of a system which may superficially appear to be a candidate for the self-organizing theory of complexity, as power laws are ubiquitous in Internet statistics. It certainly appears as though new users, applications, workstations, PC’s, servers, routers, and whole subnetworks can be added and the entire system naturally self-organizes into a new, robust configuration. Furthermore, once on line, users act as individual agents, sending and receiving messages according to their needs. There is no centralized control, and individual computers both adapt their transmission rates to the current level of congestion, and recover from network failures, all without user intervention or even awareness. It is thus tempting to imagine that Internet traffic patterns can be viewed as an emergent phenomena from a collection of independent agents who adaptively self-organize into a complex state, balanced on the edge between order and chaos, with ubiquitous power laws as the classic hallmarks of criticality.

The core of the Internet, the Internet protocol IP, presents a carefully crafted illusion of a simple but possibly unreliable datagram delivery service to the layer above typically the transmission control protocol, or TCP by hiding an enormous amount of heterogeneity behind a simple, very well engineered abstraction. The TCP in turn creates a carefully crafted illusion to the applications and users of a reliable and homogeneous network. The internal details are highly structured and nongeneric, creating apparent simplicity, exactly the opposite from SOC and EOC. Furthermore, many power law statistics of the Internet are independent of density congestion level, which can vary enormously, suggesting that criticality may not be relevant.

The creation of the Internet is, indeed, one of the success stories of instrumental rationality. However, as Carlson and Doyle point out next, it comes at a price:

Interestingly and importantly, the increase in robustness, productivity, and throughput created by the enormous internal complexity of the Internet and other complex systems is accompanied by new hypersensitivities to perturbations the system was not designed to handle. Thus while the network is robust to even large variations in traffic, or loss of routers and lines, it has become extremely sensitive to bugs in network software, underscoring the importance of software reliability and justifying the attention given to it.

…

This ‘‘robust-yet-fragile’’ feature is characteristic of complex systems throughout engineering and biology.

Abstraction and virtualization are indispensable tools for enabling effective interaction. At the same time, they require systems to be open, to have the capacity to affect and to be affected by other systems. This is where we see emergence of unforeseen behaviors or new failure modes. Software that was used to design and implement some kind of a beneficial “user illusion,” such as various user interfaces, is an open attack surface for designing and implementing malicious user illusions (or even manipulating users and systems without any overt indication that this is happening). The possibility of this kind of “hijacking” is a universal trait of complex multilayered architectures, both engineered (think about computer viruses, cryptocurrency scams, DDoS attacks, etc.) and biological (think about actual viruses, parasites, cancers, autoimmune disorders, etc.). This is also a bug/feature in complex social systems like modern markets and democracies — highly beneficial abstractions like voting, finance, law, social networks are vulnerable to hijacking by virtue of their (relative) openness in the sense that they interact with other architectural layers in the overall system, and the constraints imposed on one layer will inherently deconstrain others.

In this sense, AI is, indeed, a normal technology. It aims to instrument certain kinds of “user illusions,” e.g., the illusion of communicating with intentional, anthropomorphic entities, and these systems are RLHF’d ad infinitum to force them to operate in a HOT state. However, this is exactly where we come face to face with the paradox of highly optimized tolerance¹. Language is, itself, a multilayered architecture, where robustness at one layer can mask a great deal of of complexity at lower layers. Have you noticed the extra “of” in the preceding sentence? In case you haven’t, this is exactly what I am talking about: We are trading off the accuracy in the semantic layer against speed of processing at the syntactic layer, so small typos, repetitions and the like may not register at all if we pay attention to sentences and not to individual tokens like word parts or letters. We can go lower down the hierarchy of layers and talk about neuronal activity that takes place in our brains as we interact with the world. Much of it is un- or sub-conscious. That includes the tremendous web of associations and dynamical links on which conscious activity supervenes, and that’s where language offers both a powerful interface for abstraction and an attack surface vulnerable to hijacking. To my mind, this removes much of the mystery of Golden Gate Claude — and also shows what Sigmund Freud got largely right. The constraints of language deconstrain both the creation of new linguistic structures (and thus what can happen at layers above language) and the activity taking place in layers below language. This is the case both when humans interact with other humans and when they interact with AI systems. The only difference is that we somehow think that mechanistic interpretability is essentially distinct from the psychoanalytic hour just because we can mathematize the former but not the latter.

One of the last books I read this year was There Is No Antimemetics Division by qntm. The central conceit of the novel, that there is a secret organization devoted to protecting the world from antimemes (informational entities that can erase the traces of their having been experienced by human minds), could only have been born in our current cybernetic moment. We are surrounded by antimemes, such as the market, the TCP/IP protocol, social network engagement maximization, and now also AI assistants and companions. They increasingly demand our attention without drawing much attention to themselves, and it is important that we all remember: there really is no antimemetics division.

Happy New Year!

1. Yes, this (and the title of the post) is a reference to Popper’s paradox of tolerance.