Wargaming as a Social Science Experiment
By Andrew L. Crabb
In August of 2022, my supervisor at the Joint Special Operations University challenged me with an unusual task. She wanted me to find ways to embed into our courseware operational factors identified in the Department of Defense’s (DoD) Women, Peace, and Security (WPS) Strategic Plan and Implementation framework. Although my first thought was that we might be stepping into a political minefield, as she explained the WPS factors—looking at the operational environment (OE) via the lens of gender culture, dynamics, and perspectives—it became readily apparent that this was a facet of planning that was new and held merit.
Since special operations forces often claim the “human domain” as its operating milieu (Cleveland, 2016), gaining better and more nuanced insights into the OE was a worthy goal. Examining an OE’s gender culture, dynamics, and perspectives for operational advantages and considerations was an aspect that, in 30 years of operational planning and teaching operational planning, I had never considered. As I assessed the challenge, I began a long process that would lead us to embed a social science experiment in a wargame during Joint Special Operations University (JSOU)’s Special Operations Planning Course (SOPC).
This article relays the challenges and initial benefits of utilizing a wargame as the basis for a social science experiment. It is presented as a primer for those considering doing something similar, and with the understanding that the lessons I learned were as a fledgling social scientist under the tutelage of others with far greater experience. Since the experiment continues to be run and the data continues to be interpreted, the experiment’s final conclusions and findings will be published separately at a later date.
Militaries have long utilized wargaming to gain insights into operational plans and human decision-making. Wargames are commonly classified as analytical, educational, and experiential; the typology and desired outcomes help drive the wargame’s design (Appleget, 2020, 5). US military members often first encounter wargames when large headquarters use them to assess emerging operational concepts or to gain insights into their own and their adversary’s plans and capabilities. Another way those in uniform experience wargames is during operational planning.
In the US military’s Joint Planning Process, the Course of Action Analysis & Wargaming step utilizes an analytical wargame to assess the strengths and weaknesses of a proposed course of action. While both situations are common ways military members participate in wargames, a less common occurrence is using wargames as social science experiments. The social sciences (anthropology, sociology, history, economics, political science, psychology, etc.) are generally said to study human behavior, culture, and norms at the individual and group levels. This includes armed conflict and the systematic employment of violence. Social science experiments utilize controlled and repeatable conditions to understand the effects of different variables on given outcomes. If properly constructed, utilizing wargames as a vehicle for social science experiments can provide useful data and insights into the complex problems of armed conflict.
Because over the academic year, SOPC would have multiple iterations, we could meet both the “repeatable” experimental criteria and produce a relatively large “N” (the data’s sample size) (Yang, 2015, 13). Additionally, as course proctors, we could re-design the course’s wargame specifically to meet other goals of the experiment. Finally, it was important that the addition of the experiment did not compromise the course’s learning outcomes.
After taking some time to ponder the task, I developed a pathway that drew on several recent experiences, including attendance at the US Army War College’s Game Design Course and as a student in an international relations doctoral program. The pathway I developed took me through three significant challenges. With help from colleagues and outside institutions, these challenges all proved surmountable.
Challenge #1: The Research
Our first step was to learn more about WPS. During my doctoral program, we were taught that “the path of all knowledge leads through the question (Alvesson & Sandberg, 2013).” We formulated a WPS-related research question. “What gender-related information exists in Joint doctrine and best practices to plan and execute Joint operations?”
We then conducted a literature review to see if there were gaps in the literature. Knowledge gaps can lead researchers to areas in which original work, or furthering the work of others, might be conducted. Through research, we learned that WPS was an initiative sparked by a United Nations resolution that sought to include women in the processes of conflict cessation; it also established protections for women, children, and oppressed communities during times of war (United Nations (UN), 2000). The UN’s WPS goals gained attention and momentum, ultimately finding expression in 2017 as a bi-partisan Act of Congress. The DoD followed the law with the WPS Strategic Framework and Implementation Plan (DoD, 2020).
Among other things, the DoD WPS Strategic Plan mandated that the DoD “train personnel on the needs, perspectives, and security requirements of men and women; protecting civilians from violence, exploitation, and trafficking in persons; and international humanitarian law and international human rights law. Apply gender analyses to improve DoD program design and targeting.” It was the last portion that drew my attention. In the operations world, program design could be interpreted to mean building operational concepts, targeting recalled operational limitations and considerations (e.g., constraints and restraints, operational risk).
Surprisingly, the literature review of doctrinal Joint Publications demonstrated that, before the enactment of the 2017 WPS legislation, the main Joint Publications (2-0 Intelligence, 3-0 Joint Operations, 5-0 Planning, 3-05 Special Operations) had no references (i.e., zero attributions) to women, gender, or children. After 2017, all the updated publications had a significant amount of information concerning women and gender. More surprisingly, we found the new information extremely insightful and useful (e.g., operational-level gender considerations for targeting, gender patterns of life, gender cultural insights, and gender-political power dynamics).
In addition to the literature review, we also met with leading experts in the WPS field. They provided outside readings to enhance our overall understanding of WPS. They also reviewed the experiment and survey designs. We then held a workshop with subject matter experts from the WPS Community of Interest. Their input and assistance were of immeasurable value. In sum, there was broad consensus that many WPS factors had yet to be operationalized and that our initiative had great promise and merit.
Employing the gap spotting technique (Alvesson & Sandberg, 2013), we identified that there was a doctrinal gap in assessing considerations for children in the operational environment (i.e., still no references, even in the updated publications); we decided not to pursue this gap until doctrine addressed it better. However, as educators, we identified a second gap that further work needed to be done to formally incorporate WPS factors into operational-level planning. This was the gap we would pursue.
So, after completing the literature review, meeting with experts in the field, and going through the gap-spotting process, the research challenge was complete. WPS was newly incorporated into doctrine but, as an operational-level planning consideration, was largely absent from our courseware and unit-level best practices. To determine how best to teach and incorporate this new doctrine into operational planning, we would investigate the question via a wargaming social science experiment.
Challenge #2: Experimental Design
To do this, we first constructed hypotheses concerning the outcomes of operationalizing WPS factors (e.g., gender analyses provided via intelligence and information updates). Our hypotheses were:
- Hypothesis #1: Operational Planners will assess that comprehension of gender dynamics, perspectives, and culture in the OE can aid the successful conduct of special operations.
- Hypothesis #2: Operational Planners will assess that comprehension of gender dynamics, perspectives, and culture in the OE will be more beneficial to the conception and execution of indirect special operations.
- Hypothesis #3: Operational Planners will assess that education and training in gender dynamics, perspectives, and culture in the OE will make them more effective planners.
To test the hypotheses, we then began constructing an outline of what a social science experiment might look like. The classic experimental design identifies a dependent variable (sometimes known as the outcome variable). Then it utilizes an independent variable (sometimes known as a predictor variable) to identify relationships between the outcome and the independent variable (Abiodun-Oyebanji , 2017, 48). We identified the assessed value of gender analyses (i.e., the value of assessments derived from gender culture, dynamics, and perspectives) as the dependent variable, and education on gender analyses & provision of such analyses during planning as the independent variable.
As mentioned previously, the specifics of our design had the experiment occur within the confines of SOPC. SOPC is a two-week course that takes place six times per year. SOPC prepares students to design, plan, and present a concept for a Joint special operations forces (JSOF)-focused operational-level campaign. During the latter portion of the course, students (2-4 groups of 6-8 individuals) participate in a wargame to evaluate their operational plan’s strengths and weaknesses. During the wargame, each participant is assigned to play a role as one of the operational actors (Friendly, Enemy, Neutral Forces, Operational Environment). Before the wargame, all participants received an “intelligence packet” with supplemental information to aid their participation.
The experiment established a control group of students who did not receive gender analysis education and whose intelligence update packet contained no gender analyses. The independent variable groups received the gender analysis education; the intelligence packets had the same intelligence updates as the control group, with the addition of the gender analyses.
An important aspect of the SOPC wargame is that it is a matrix wargame. Matrix games utilize structured argumentation for different actors to advocate for the success of their proposed actions (Mouat, 2018). The Wargame Facilitator then crowd-sources (i.e., surveys the group) to determine the probability of success, with each player providing their assessed probability of success. The Facilitator listens to who has the most convincing arguments for probability, finalizes the probability percentage, and the actor rolls the dice. The dice roll’s outcome determines success or failure, and the Facilitator provides context and answers questions concerning the outcome. One key aspect of the experiment is if those provided gender intelligence use it in the wargame to strengthen their argumentation and thus increase their success probability.
The crucial portion of the experiment comes following the wargame. Students complete a survey that asks them to assess (among other things) the value of gender intelligence in operational planning. Survey design is a high art, and the best advice we received concerned survey adjustment.
The student post-wargame survey design begins with biographical data, transitions to general wargame and operational questions, and ends with specifics on gender intelligence and considerations for operational planning. Concerning the biographical data, I received coaching on what to ask without compromising anonymity—if respondents sense their background could reveal their identity, they often won’t take the survey! I also was mentored to build a menagerie of Likert-scale questions mixed with other question types (e.g., matching, fill-in-the-blank, etc.) and open-ended narrative questions for students to fill out.
The Likert scale and other numerical valuation questions can provide data that can be quantified. Narrative questions can be assessed using thematic-coding techniques (with computer-based tools like NVivo) to quantify data better. Another great recommendation provided by a subject matter expert was to “triangulate” the results by having observers and facilitators also fill out records. Observers can provide objective data (e.g., the number of times gender intelligence was utilized, which players using or not using gender intelligence won or lost each turn) and subjective data (e.g., assessments on the students’ comprehension and use of gender intelligence). Taken together, the result was a mixed-method experiment with both qualitative and quantitative data. As we continue the experiment, we will seek to improve the experimental design while maintaining the consistency of the assessed data.
Challenge #3: The Institutional Review Board
As we approached finalization, one outlier task stubbornly refused our repeated attempts for resolution. To conduct human subjects research (HSR), Federal guidelines and regulations mandate that the research go through an Institutional Review Board (IRB) for approval. Through her previous work, my supervisor had significant experience going through the IRB process. Because our experiment was essentially an assessment focused on how best to educate students, she was confident that an IRB would find our work educationally exempt from further HSR requirements (which can be onerous).
The difficulty was that my institution does not have an IRB. Finding an outside IRB willing and able to review our experiment proved challenging in the extreme. Several times an adjacent DoD educational institution would assure us they’d provide a review, only to have something arise that would force us to look elsewhere. Finally, as our third of six SOPC iterations (for FY23) approached, we found a partner to review the assessment.
As expected, before approval, the IRB did mandate some revisions and clarifications. It also required that I complete further HSR training (provided online by Collaborative Institutional Training Initiative (CITI) Program). After a several weeks-long process, the IRB completed its review and did find the experiment exempt from additional HSR requirements [Author’s note: specifically, the IRB found that because the results of the WPS Wargaming Assessment were not generalizable to the larger Non-DoD population, the experiment was exempt from further review].
Initial Impressions and Takeaways
To date, we have conducted one formal iteration of the WPS Wargaming Assessment and have three more before completion in September 2023. As previously mentioned, we intend to seek publication of the social science experiment’s findings once the academic year is complete. While it is too early to draw final conclusions, several trends and factors have become apparent.
As frequently happens in survey work, we had students who declined to take the survey, along with surveys that were only partially filled out or otherwise missing answers. The answers to narrative questions were generally thin and had the highest rates of non-responses. We also have learned that each survey needs to be coded or have a unique filename for easy reference and cataloging. Finally, we are transitioning the survey fully to our Blackboard Ultra website, which can better assure anonymity and has tools for data assessment. All these challenges are correctable and attainable but require additional work. We are moving out on them all.
Concerning the experiment’s hypotheses, our initial takeaway is that when operational planners are provided gender education and intelligence, they find it valuable for planning and executing simulated special operations. However, currently, our N is small, and there are many caveats to that general impression. Among our students, that finding is also not universal.
The biggest surprise was that the social science experiment effort, research-to-design-to-implementation, uncovered a gap in our own courseware and unit-level practices (as observed through our work with special operations units). While we make great efforts to maintain the currency of our course, we did not pick up on the post-2017 additions of gender analysis to the Joint doctrinal family of publications.
Expounding on that last point, like most operational planning courses, we have taught different ways to assess human culture and how to utilize those assessments in operational planning; protection of vulnerable populations usually expresses itself through operational limitations (constraints/restraints) and operational risk. More specifically, collateral damage mitigation and protection of civilians/non-combatants is usually considered most prominently during the targeting process at the tactical level of war. However, we have now incorporated gender analyses as one method to evaluate the human element of the operational environment. We have also added the protection of civilians and vulnerable populations as a prominent operational-level consideration, even in the incipient stages of operational-level planning. This also aligns with the emerging DoD mandates related to the Civilian Harm Mitigation and Response Action Plan (CHRM-AP).
Building wargames as social science experiments is not easy. In our case, we had to carefully construct the experiment to not detract from the overall learning outcomes of our course. We had to ensure we didn’t artificially inflate the value of gender intelligence and information. While the WPS Wargaming Assessment has been a significant undertaking, the initiative has already improved our course and professorate. Going forward, we hope to continue building JSOU’s social science experimentation experience and expertise. We also hope our efforts might lead other institutions to take similar initiatives. The result would enhance wargaming’s credibility as a tool, not just to assess plans and operations but also to gain insights into the larger, wickedly complex problems of irregular warfare and special operations.
[Note: Professor Andrew L. Crabb will speak on Wargaming as a Social Science Experiment during a panel presentation at the Connections 2023 Wargaming Conference. The conference will be held in Ft. McNair, Washington D.C. from 21-23 June 2023.]
Bibliography
Abiodun-Oyebanji, Olayemi. (2017). Research Variables: Types, Uses, and Definition of Terms.
Alvesson, M., & Sandberg, J. (2013). The context of constructing and formulating research questions. SAGE Publications Ltd, https://doi.org/10.4135/9781446270035
Appleget, Jeff. 2020. The Craft of Wargaming. Naval Institute Press.
Cleveland, Charles, James Linder, and Ronald Dempsey. "Special Operations Doctrine: Is It Needed?" Prism 6, no. 3 (2016): 5-19. Accessed May 31, 2023.
“Experimental Wargames to Address the Complexity--Scarcity Gap.” In Spring Simulation Conference (SpringSim 2020). Fairfax, VA, USA: Society for Modeling and Simulation International (SCS), 2020. https://doi.org/10.22360/SpringSim.2020.ANSS.010.
Mouat, Tom. "Practical Advice on Wargaming." 2018. http://www.mapsymbs.com/PracticalAdviceOnMatrixGames.pdf.
Perla, Peter P., and Michael C. Markowitz. “Conversations With Wargamers:” Fort Belvoir, VA: Defense Technical Information Center, January 1, 2009. https://doi.org/10.21236/ADA494298.
Perla, Peter P, Michael Markowitz, and Christopher Weuve. “Game-Based Experimentation for Research in Command and Control and Shared Situational Awareness,” n.d.
Reddie, Andrew W, and Bethany L Goldblum. “Evidence of the Unthinkable: Experimental Wargaming at the Nuclear Threshold.” Journal of Peace Research, October 14, 2022, 002234332210947. https://doi.org/10.1177/00223433221094734.
Rubel, Robert C. “THE EPISTEMOLOGY OF WAR GAMING,” 2023.
U.N. Security Council, 4213th Meeting. "Resolution 1325 (2000) [Women, Peace & Security]" (S/RES/1325). 31 October 2000. https://documents-dds-ny.un.org/doc/UNDOC/GEN/N00/720/18/PDF/N0072018.pdf?OpenElement
Valeriano, Brandon G., and Benjamin Jensen. “Wargaming for Social Science.” SSRN Electronic Journal, 2021. https://doi.org/10.2139/ssrn.3888395.
Weuve, Christopher A., Peter P. Perla, Michael C. Markowitz, Robert Rubel, Stephen Downes-Martin, Michael Martin, and Paul V. Vebber. “Wargame Pathologies:” Fort Belvoir, VA: Defense Technical Information Center, September 1, 2004. https://doi.org/10.21236/ADA596774.
"Women, Peace, and Security Strategic Framework and Implementation Plan." Department of Defense Public Affairs. Department of Defense, June 1, 2020. https://media.defense.gov/2020/Jun/11/2002314428/-1/-1/1/WOMEN_PEACE_SECURITY_STRATEGIC_FRAMEWORK_IMPLEMENTATION_PLAN.PDF.
Yang, Keming. Making Sense of Statistical Methods in Social Research. 1 Oliver’s Yard, 55 City Road, London EC1Y 1SP United Kingdom: SAGE Publications Ltd, 2010. https://doi.org/10.4135/9781473914636.