The Importance of History in Technology and Security Policy Analysis

The Importance of History in Technology and Security Policy Analysis

Adam Elkus

While many people in military, law enforcement, security, and intelligence professions have always thought of their superiors as overly rote and mechanical, they may very well live to see their boss actually become a machine. In various domains of security and conflict, analysts increasingly predict that machines will take on progressively larger components of human reasoning and decision-making. [1] It is tempting to argue that such technologies have little real precedent and that they will fundamentally revolutionize security and conflict. [2] Certainly disruptive shifts in the basic character of security and conflict have occurred. In particular, immense sociotechnical changes that occurred from the late 1700s to 1945 were perhaps some of the most profound and wide-ranging in human history. [3] That said, what predictions about both technology and security have in common is that they are often embarrassingly wrong. [4]

Nonetheless, futurism tends to be one of the more dominant approaches to technology and security policy analysis. While looking into a crystal ball certainly is useful, analysts should also consult their history books. History can be useful for policymakers dealing with emerging technologies and security problems. Yes, understanding the policy implications of even highly advanced computer technology can be aided by the study of history. History is not destiny, and policymakers who do not carefully appraise its potential utility also certainly can misuse history. [5] But very few complex problems are completely new. And examining the ways in which even highly science fiction-like technologies and their attendant security problems arise from precedents set decades back can help analysts better grasp the their problems and potentials today.

This article begins by describing the mechanization of decision-making and some of the concerns that it raises. Afterwards, the article provides an extended case study in past debates over limited rationality in both humans and machines. The article explains how concerns over rationality and decision-making helped fuel a wave of cognitive science and artificial intelligence research in strategy game play, and how conceptions of rationality and intelligence in chess and go differ. Lessons for policy analysis of computers and decision problems are briefly summarized in the concluding paragraphs.

Mechanizing Decision-Making: Trust and Risk

It is often argued that today’s problems with technology and security are fundamentally revolutionary in nature. [6] Closer analysis often reveals problems with sweeping calls to rewrite everything we know about security due to the emergence of particular technologies and allegedly novel threats to security. [7] Additionally, technological futurists often assume that the technologies and concerns they write about are completely new. Historians of technology, however, have traced the origins of many of today’s most advanced technologies (and the concerns that arise from them) to processes that began decades or even centuries ago. [8] Nonetheless, as Arthur C. Clarke once said, technology appears to many as a form of magic rather than a tool engineered by humans. [9] If technology is thought of as magical in general, computers are commonly perceived to be a form of black magic. It does not take long, for example, for understandable concerns about cybersecurity and cyberconflict to morph into hilariously apocalyptic warnings of cyber-doom.  [10] The converse also holds true. Decision-makers often have highly unrealistic expectations about what computers can do in security situations. [11]

Much of these analytical problems stem from historical ignorance. As naval historian B.J Armstrong has argued, analysts neglect to consult the long-run history of unmanned systems when penning their latest thinkpiece. [12] In general, many futurists are almost comically ignorant of the ways in which computing, artificial intelligence, and robotics have historically been highly intertwined with security affairs. For example, how many big data boosters know that the Metropolis-Hastings algorithm, one of the most prominent statistical techniques in a data scientist’s toolbox, owes its origin to work on the atomic bomb? [13]  Do many analysts writing about the supposed security novelties of artificial intelligence grasp how much squabbles over Pentagon money have shaped how cognitive science and artificial intelligence have evolved as fields of study over many decades? [14] Even relatively recent events, such the role of automation in the USS Vincennes-Iran Air Flight 655 disaster, have been forgotten. [15] That is a shame, as incidents like the Flight 655 disaster have much to teach us about the risks of automating core decision functions. [16]

The accurate observation that futurist concerns and policy analyses about emerging technologies and security problems are often historically illiterate, however, dos not negate the seriousness of the problems they seek to solve. In numerous domains, policymakers face questions of how much human decision-making should be augmented or replaced by humans. One need only look at debates over the future economic impact of emerging computing technology to a get a sense of how much it might change. [17] Much of future human economic activity could very well consist of programs doing business with other.[18] This “second economy” would involve humans as the principals, rather than the agents, of economic activity. [19] Fleets of coordinated robotic vehicles may replace human traffic as multi-robot systems become more advanced and robust. [20]

Of course, technology brings progress but also problems; excitement over change tends to yield to grim acknowledgement of the complications and risks that inevitably follow. [21] Security policymakers are often more aware of this due to the fact that the same technology which helps a cop catch a criminal often makes it that much easier for the criminal to escape justice. Nonetheless, government and private sector entities with security responsibilities have often been early adopters of the most advanced emerging technologies and play a key role in bringing them about. [22] Law enforcement, military, cyber, intelligence, and private sector security communities in particular will likely substantially augment or replace human decision-making functions with computers in the near future.[23] Many of these communities already use (and have always used) computing to help them cope with adversarial situations and security management. But as advances in computing continue, more and more components of human decision-making will likely be mechanized.

While much of this trend towards computerizing decision-making may sound like science fiction, chances are you may already have experienced its effects. And if you are a Los Angeles area commuter looking to avoid paying a public transit fare, you might have even experienced it firsthand. Take, for example, the partnership between the University of Southern California’s Teamcore Group and Los Angeles law enforcement agencies. [24] Many law enforcement and homeland security problems involve the scheduling of operations and the allocation of scarce resources. Police patrolling is a key example of this problem.[25] Transit security is difficult because stopping even simple crimes such as fare evasion requires finding the most efficient way to use scarce police resources to patrol sprawling metropolitan transit systems. [26] Frequent Small Wars Journal contributor John P. Sullivan and others were part of a project to help use an artificial intelligence system to augment existing law enforcement transportation security. [27] The system scheduled randomized police patrols using game theory. [28] Critically, the system was also robust against the everyday interruptions of planned activities that are commonplace in law enforcement. [29]

The Los Angeles transit example is a useful case study of mechanizing decision-making for the following reasons:

1. The system was designed to perform a component of what used to be a core police function: operational planning and decision-making. As Sullivan and Sid Heal have written planning and decision-making are some of the hardest challenges in law enforcement and security. Sullivan has focused on the complexities of information, intelligence, and operations in emergency scenarios. [30] Heal has written broadly about the complexities of planning large-scale emergency operations. [31] Hence, the system has mechanized a key part of law enforcement planning and decision.
2. The system is good at something humans are not always good at: randomization. In adversarial situations, randomizing your behavior is important to avoid being exploited by the opponent. However, humans are not random in how they make decisions. [32] A computer faces no such restrictions, and randomized algorithms are a big topic of study in computer science. [33] The system Sullivan and others developed and tested schedules randomized patrols. [34]

In contrast to sci-fi visions, uses cases such as public transit security are mundane and practical realizations of emerging intelligent systems technologies. Most, if they were not directly told that the Los Angeles county law enforcement establishment was using an AI to optimize transit security, would have never known about it in the first place. And as demands for pervasive surveillance rise after the Paris attacks, systems like these will become more commonplace. The only way that policymakers can reconcile public demands for security with the limited pool of analysts, officers, and operators available is to either increase the amount of resources allocated to domestic security or use technology to make what policymakers already have work better. Given the reality of budget shortfalls in many industrialized Western states, the latter is much more likely than the former.

What To Think About Machines That Think

What’s the downside of mechanizing decision-making? Plenty, as it turns out. As with any complex technical system involved in security matters the increasingly mechanical nature of security decision-making poses risks as well as opportunities. Scholars of technology and automation are well aware of the unintended consequences of automation on effective decision-making and choice. [35] A bigger problem lies in questions of trust and understanding.  As historian Donald MacKenzie argues, the mechanization of reasoning and decision-making inherently raises questions of trust. How do we know that the computers that we depend on are dependable? [36] An added layer of complexity is that these computers are performing human economic functions by simulating – with varying degrees of fidelity – the reasoning processes of humans they are meant to replace.

All of this raises the unavoidable question of how we can think about machine reasoning despite it being different from our own in almost every way. The answer – and the problem--- can partly be found in something called the intentional stance. Most of the time we think about complex computing systems as if they were other people. We use language such as “the computer won’t work,” implying that our malfunctioning MacBook is a recalcitrant laborer that has gone on strike. In short, we attribute life-like intentions to lifeless tools. One reason we do this is that we often prefer humanizing what is simply layers of mechanical hardware, software, and networks rather than deal forthrightly with the occasionally alienating nature of information technology. [37]

However, it also may just be human nature to attribute mind, purpose, and intent to everything we interact with. We have the ability to ascribe mentalistic attributes such as intentions to others and this capability emerges at an early age. [38] But what does it mean to do this? Some philosophers and cognitive scientists believe that when we try to predict the behavior of other people, we rely on a folk psychological construct called beliefs, desires, and intentions. Beliefs denote what someone else knows about the world, desires what they want to do, and intentions denote desires that they are committed to realizing through action. [39]

Some might even argue that we internally simulate others’ minds when we use folk psychology to think about what they are thinking and what they might do. [40] In short, we treat other people as rational agents, where rationality is defined vaguely as the ability to make actions at that are in its own best interests. [41] Because computer programs differ widely in the way they make decisions compared to humans, this conception of rationality is often used as a way of modeling and predicting the behavior of complex artificial intelligence and robotics systems. We treat artificial agents as rational for lack of a better way to understand them. [42]

Rationality need not entail omniscience. Rationality can be limited for a variety of reasons while still being rational. [43] Finally, while intelligence and rationality are distinct concepts, they are also nonetheless closely related. [44] Hence we often automatically make the assumption that the technical system we are dealing with is a rational agent and ascribe human characteristics such as beliefs, desires, and intentions to it. [45] Only its designer likely will fully understand how it works, and even then the designer will likely seek to predict is behavior based on such crude and seemingly unscientific psychological attributions. [46]

One of the biggest problems in understanding the policy implications of mechanizing decision-making is what kind of rational agent we expect our mechanical proxies to be. There is little consensus as to what rationality means. Rationality could refer to the appropriateness of a thought or choice, but it also might refer to the appropriateness of the process that produced a decision. Even more confusingly, rationality might refer to whether the choice yielded an outcome consistent with someone’s preferences or improved their evolutionary fitness. [47]

To most audiences, “rationality” colloquially refers to the appropriateness of an action. But the problem is that rationality’s meaning is much more complex. To see how problematic this might be for the regulation of artificial intelligence programs and policy analysis concerning them, consider that all intelligent systems – biological or artificial – are constrained in how they search for good solutions. It is physically impossible for them to quite possibly consider every option or outcome, and they are biased in various ways toward certain regions of the search space they explore when they ponder what to do next. [48] Hence, a definition of rationality that ignores the process of selecting actions and how this process might reflect the environment the system was designed to be successful in is likely not going to be useful for policymakers. [49]

In general, assumptions about rationality and action selection in synthetic intelligence represent a philosophical, scientific, technical, and above all else legal minefield that policy analysts will nonetheless be forced to traverse as intelligent systems become more prominent in security concerns. [50] However, history can also help us understand the terrain and landscape that must be traversed, even if it is unlikely to reveal all of the explosive traps lurking underneath.  Evidence for this proposition can be seen in Cold War efforts to think about limitations on human and machine decision-making, which will be explored in the following section.

When we examine the historical record, we can see a historical confluence between problems of mechanizing decision-making and debates over intelligence and rationality.  As historians such as Margaret Boden have observed, the trend towards the mechanization of decision and the view of decision as mechanical has in some ways been in progress for centuries. [51] In order to create a descriptive language that allows us to understand how mechanical creations do things we do, we must have first regarded ourselves as mechanical in some shape or form. Alan Turing, for example, famously analogized a program to a human doing out math problems on a scratchpad of paper. [52]

While much of these questions lie at the core of Western science and philosophy, the harsh and unforgiving nature of Cold War demands on political and military decision-making accelerated the rise of attempts to both create mechanical minds and understand rationality and decision behavior via mechanical metaphors about human mind and behavior. [53] We can learn much from both the origins and long-term effects of these demands. History does not necessarily yield any easy answers. However, it may provide useful questions. The benefit of looking at the history of science’s quest for mechanical minds (and to explain human minds as mechanical) that we can see how rationality and intelligence has been formalized completely differently depending on what kind of problem, task, and environment is being used to think about it. [54]

Cold War Rationality and Its Discontents

In the Cold War, a new understanding of strategy emerged. It focused less on the appropriateness of the choices made by the President or general and more on the process by which the decision-maker made choices. [55] This image of strategy arose from a problem rooted in the military, intelligence, and security problems of the day: how limited information processing power of both humans and machines could be reconciled with the need for both to perform well under stress. Cold War computers were dramatically limited in computing resources compared to today’s ultra-powerful and fast systems, hence their users had to carefully economize on how they used computing to solve problems. [56] Similarly, applied research in air defense problems and other similar endeavors inspired a new view of human information processing limits in decision-making. [57]

Popular conceptions of Cold War approaches to rationality often envision a group of Dr. Strangelove-like nuclear scientists blathering on about mineshaft gaps while the world awaits destruction. [58] This view of cold and broadly robotic perceptions of rationality and behavior is not without some foundation. [59] However it is also a caricature. Contrary to popular stereotypes, there was no one monolithic Cold War understanding of rationality. As Lawrence Freedman observed in a recent survey of Cold War intellectual history, there was no one conception of rational behavior that dominated during that era. Instead scientists, policymakers, and others often disagreed about what rationality was and what aspects of rationality were important to study. [60]

Why did researchers disagree so much about how to study rationality? The answer partly lies in a larger trend – the movement towards the mechanization of mind in scientific research. Researchers in cognitive science, artificial intelligence, and other similar sciences came to regard machines as capable of replicating human behavior and human behavior as a kind of mechanical process. [61] As many historians of the time observe, much of this research was built on a particular set of metaphors for describing both individual and group behavior. These imaginative metaphors often conceptualized mind and behavior in terms of information, computation, feedback, control, communication, and other broadly machine-like conceptions of mankind. [62]

Because rationality could serve as an explanation of how both biological and machine intelligence made decisions in support of larger goals and purposes, rationality was a bridge between often highly complex and diffuse research concerns. [63]A key difference between different beliefs about rationality, however, may simply have been different choices about how to study rationality. Famous economist and strategist Thomas Schelling, for example, constructed his beliefs about strategy and rationality from a combination of history, psychology, and abstract mathematical thought experiments.[64] In contrast, other researchers utilized laboratory experiments and contrived decision problems to generate knowledge about strategic decision-making. [65] Because of this, researchers unsurprisingly arrived at different images of rational behavior.

The Cold War research program, however, was far broader than just rationality. Cold War politics fueled research efforts devoted broadly to understanding human cognition and intelligence. [66] Fears about economic competition with Japan as well as a desire to solve pressing late Cold War military problems also motivated substantial investment in a failed project to solve – once and for all – the problem of building true artificial intelligence. [67] While the effort was unsuccessful, it reflected a larger, longstanding military interest in mechanizing decision-making and building cooperative human-machine systems for security and defense. [68]

In general, the problem that really interested policymakers – crisis decision-making at varying levels of political and military organization in an emergency scenario featuring the Soviet Union and other Eastern bloc states – could not be studied directly. The past provided some clues, but as Bernard Brodie observed in 1949, for every nugget of wisdom there were ten fallacious clichés about the allegedly “timeless” relevance of some military shibboleth. [69] Contemporary confrontations and crisis situations provided natural experiments, but as we know today, much of what researchers believed about these episodes at the time was faulty at best and completely wrong at worst. [70] Hence researchers often turned to games as environments for studying rationality and other related mental and behavioral properties.

Strategy Games, Rationality, and Intelligence

The emergence of both mechanical images of rationality and the mechanization of strategy and decision-making in conflict can be traced to a particular kind of shared primordial soup: the war game. [71] However, as evidenced by the fact that the Prusso-German kriegspiel game is a variant of the older strategy game chess, the conception of modeling strategy via tabletop game did not start with the military. Chess and its variants and precursors have been played for centuries, and have arguably played a key role in creating a basic understanding of strategy as the art of an general issuing commands to proxies on a top-down map. [72]

Research in artificial intelligence, cognitive science, and the playing of strategy games is an interesting case of how perceptions of intelligence and rationality may diverge even within the same type of environment used to study it experimentally. Games have often been used to develop AI agents by providing a specific task environment to develop their behavior within. More broadly, games have been useful to cognitive scientists for studying human expertise and intelligence within a limited domain that is nonetheless complex and potentially representative of higher-order intelligence. [73]

Strategy games, believed to be the pinnacle of intelligent human reason and behavior, were used as a laboratory for studying basic cognitive mechanisms in perception, memory, and problem-solving. Strategy games were also used to study search, learning, and other fundamental artificial intelligence mechanisms in computational agents. [74] Methods in both disciplines utilized ranged from simulations of human cognitive processes in play to the programming of complete game-playing programs. Research in chess is the most famous example of this broad research program, but many others certainly exist. [75]

Researchers did not just study chess and other games to unlock the mysteries of human intelligence. They also did so as part of the larger research effort to unlock the mysteries of human and machine rationality. Researchers such as Herbert Simon held a strong conviction that both humans and computers were alike in that only procedural rationality– the study of the process by which agents made decisions – could explain their problem solving approaches. Researchers involved in these efforts claimed that results could generalize to the study of bureaucratic and even national decision-making. [76] Given that chess-playing systems and game theory models in social science shared prominent similarities (such as the minimax algorithm for zero-sum game playing), this was not an unreasonable assumption at the time. [77]

However, chess was not the only strategy game domain in which researchers pondered the nature of intelligence and rationality. The chief problem of chess is the enormous amount of possible moves that must be considered. Hence procedurally rational systems that play chess represent theories of how to simulate or at a minimum functionally replicate the manner in which human experts efficiently store knowledge about the game and search for good moves. Because experts acquire heuristic knowledge (rules of thumb and shortcuts), they only need to consider the most useful options for decisions, not the entire set of options. But this has much to do with the specific characteristics of chess as a strategy game. Other games are different. [78]

The study of computer go stands as an interesting contrast to this conception of procedural rationality. Chess is seen as a stereotypically Western approach to viewing strategy while go is allegedly more representative of Eastern ways in strategic behavior. More importantly, go is still fundamentally different than chess. [79] Algorithms that have proven successful in go reflect a new converging understanding of decision-making in artificial intelligence and the behavioral sciences rooted in the balancing of exploitation and exploration in search. To understand what these two terms mean, consider the commonly used metaphor of someone trying to climb a mountain. [80]

Imagine that you are trying to find a path that leads to the highest point on a mountain range. You have a choice of either sampling new paths that might potentially lead to the goal (without knowing where they will lead) or following paths that you know take you up the mountain range. Computer go algorithms (usually referred to as Monte Carlo tree search algorithms) have to solve this problem in some shape or form. [81] Just as the problem of dealing with the impossible amount of possible choices to consider in chess was seen as representative of organizational decision-making problems, the exploration/exploitation tradeoff has commonly been used to describe organizational decision problems. [82]

Finally, one should observe that general game playing research in artificial intelligence – whether the games are traditional games, computer games, or arcade games – have been testing grounds for AI that can play any arbitrary game given to it. Researchers believe that these environments can help shed light on less narrow and modular intelligence; after all, Deep Blue could beat Garry Kasparov at chess but would be completely unable to play poker or backgammon. [83] Recent advances in machine learning, for example, have exploited the general game playing environment called the Atari Learning Environment. [84]

While Cold War problems of rational decision-making under constraint played a prominent role in motivating game-playing research, these security motivations also shared room with different beliefs about the justifications for scientists programming computers to move knights and pawns around. Chess was seen as a simple yet tremendously complex environment for studying higher-order properties of intelligence. [85] To some extent, this raises a larger question of whether games are a valid substitute for the real world at all. Success even in a highly difficult game does not guarantee that we have learned much about how a computer program could model or replace human decision-making in the real world. [86]

Games after all, are contrived environments played by those who have already been socialized into their rulesets and conventions. However, intelligence, cognition, and rationality in the real world likely emerges from how humans and other animals deal with unstructured and dynamically changing environments. [87] Inspiration for some of today’s most interesting research in cognitive science and artificial intelligence comes from the immense learning and development performed by human infants, not chess grandmasters. [88] Perhaps this is in keeping with Turing’s belief that the best way to produce machine intelligence was to teach it like a parent instructs a child. [89] However, it also might have confounded researchers in the behavioral and computational sciences exerting so much time and energy to capture the complex knowledge and heuristics of chess experts if they knew that decades later, many of their successors would be trying to capture the hidden wisdom of babies. [90]

Nonetheless, inasmuch as strategy games captured core aspects of the larger class of what one cognitive scientist dubbed “adversarial problem-solving,” research in how machines and humans played them contributed much to our understanding of how people find solutions under severe constraint. [91] We still have much to learn about how people and machines play from even simple adversarial games such as the child’s game of rock-paper-scissors. [92] And we have barely even begun to explore far more complex strategy games than chess or go that are played and enjoyed by millions of gamers worldwide. [93]

Much of this work increasingly dovetails with efforts to create better computer generated forces for military and security training, simulation, and modeling. [94]

Human After All: Lessons for Policy Analysis

While it is often asserted that current technological problems in security and conflict are unprecedented and that we can learn little from history, this article uses a historical analysis of game-playing research in artificial intelligence and cognitive science to suggest otherwise.  There is much we can learn from even the recent past concerning technological problems in security policy. A look at Cold War concerns over rational decision-making and differing conceptions of procedural rationality in game domains suggests that the biggest problems for mechanizing security decision-making lie in how we can understand whether or not a program is acting efficiently to realize the goals and preferences we have encoded in it. While this presumption is debatable, it also recognizes that both humans and machines are limited in their information processing power and other variegated capacities for finding solutions to complex problems.

Perhaps the most important lesson we can learn is that even the most advanced machines are simply extensions of ourselves. [95] Long before we dreamt of intelligent machines, we described both ourselves and other animals in mechanical and machine-like terms. [96] We have always also relied on machines and technology to augment and extend our humanity, and this capability has been key to the success of human civilization. [97] Even today, the notion of “autonomous” machines ignores the very real role of their human creators and users in practical use-cases. [98] Hence, the most difficult problem of computing in security policy is the degree to which computers are both products of deliberate construction and self-perceptions and behave remarkably different than humans do in the same task environments. [99]

However, history suggests that understanding the design assumptions behind how technical artifacts are built can help us better predict, control, and understand them. Computer systems may act rationally, but the definition of rationality used to construct may vary widely. Because there is a lack of fundamental consensus as to how to abstract and model fundamental aspects of intelligence and rationality in computers, systems will likely vary widely in how they go about making decisions. [100]

While policymakers may believe that a system of relevance in a security problem will behave in a certain way, pioneering computer and cognitive scientists presciently noted decades ago that no one really knows what a complex computational artifact will do until the program is actually run. [101] Even if computers are incapable of generating real novelty, Turing correctly notes that they are nonetheless always capable of surprise. [102] Or, less romantically, they do what you tell them to, not necessarily what you want them to. [103] Computers are “stupid” and in practice many programs are often hostage to the basic assumptions of their programmers. [104]

Policymakers should ask the scientists and technicians responsible for whatever model or system that the security organization relies on what assumptions they have made about the decision environment and how the program copes with it before they take any answer or solution the machine supplies them on faith. They should also frequently challenge their own assumptions regarding what the computers will do. Computers can certainly be tools that help deliver useful answers to problems. They can also be yet another means for policymakers to deceive themselves about whether they are making progress on a problem or making it worse. [105]

In closing, none of this should be taken as an argument that the only problems with computers are policymakers not using them appropriately --- although the misuse of technology is sadly a constant in American security history. [106] At the end of the day, even the most intelligent system is, after all, no match for genuine human stupidity. But there is a concrete reason nonetheless why fighter pilot John Boyd put people and ideas ahead of machines.  [107] Boyd understood something that scientists today increasingly recognize: creativity, innovation, and novelty cannot be programmed. [108] They have to arise organically from interaction with a complex world, and from experimentation and repeated failure. Humans are still far better at this than computers.

Nonetheless, if we can bolster the grey matter in between our ears with an external brain in the form of computing, we should carefully consider the opportunities, risks, and uncertainties involved in doing so. While we cannot predict the future of fast-moving technologies, we can at least look to the past for guidance, understanding, and continuity.

End Notes