Falling Short in Measures of Effectiveness

Falling Short in Measures of Effectiveness

Ashley Franz Holzmann and Whitney O'Connell

Psychological Operations (PSYOP) is one of the United States Army Special Operations Command’s (USASOC) critical capabilities.^[i] It is also an invaluable asset to conventional operations, and is incredibly effective when aligned with kinetic operations during planning phases.^[ii][iii] Psychological Operators (PSYOPers) disseminate selected information in order to persuade, change, and influence the cognitions, emotions, and behaviors of a target audience (TA) in line with United States objectives.^[iv]

PSYOP has been an integral part of military operations since the world’s earliest documented battles, with the first cases being cited as occurring over 3,000 years ago.^[v][vi] The idea of understanding, and ultimately exploiting, the motivations and vulnerabilities of an enemy in order to merge victorious in battle has been utilized by some of history’s most recognizable leaders, such as Genghis Khan, Alexander the Great and Tamerlane; some of history’s most infamous dictators, such as Kim Il Sung, Mao Zedong and Adolf Hitler; and has been documented in some of military history’s greatest written works, such as The Arthashastra and Sun Tzu’s The Art of War.^{[vii][viii][ix][x][xi][xii]}

Despite the widespread use of PSYOP amongst United States military commanders dating as far back as The Battle of Bunker Hill, it was not until the 20^th century that this method of warfare was officially instituted as a United States Army military operation specialty.^[xiii] The efforts of the Psychological Warfare (also known as PSYWAR) units in WWI greatly contributed to the successes of Allied Forces, and subsequently influenced the basis of modern advertising.^[xiv][xv] On the 16^th of October, 2006, the PSYOP Branch was formed. Since the beginning of warfare, PSYOP has been instrumental in persuading enemy, friendly and neutral populations to take actions favorable to allied objectives in order to preserve diplomacy and avoid or optimize kinetic conflicts.^[xvi]

Evaluation of the PSYOP, or Military Information Support Operations (MISO), series is Phase VII of the seven-phase MISO process.^{[xvii][xviii]} Teams on the ground often overlook or discard the evaluation phase because its importance is either not fully understood, or it is considered insignificant; coupled with the pressure from kinetic minded commanders who want immediate results, this step is often placed solely on the shoulders of the PSYOPers on the ground—who have little incentive to consider such complex scenarios.

The result of this inattention to evaluations is that PSYOPers are forced into one of two data manipulation methods in an attempt to defend the funds that were allotted for the programs being executed: (1) series impact indicator data and supporting information is manipulated into a skewed version of effectiveness; (2) data or measures that are not relevant to the disseminated products are claimed as a measure of series effectiveness.^[xix] The PSYOP community would dramatically increase its effectiveness and credibility by specifically correcting three common institutional pitfalls: having non-specific target audiences to measure effectiveness, failure to properly document and track impact indicators, and substandard reporting practices; failing to address these pitfalls will only continue the trend of drawing inaccurate conclusions from well-conceived, well-planned and costly operations.

The Pitfalls

The First Pitfall: Having a Non-Specific Target Audience Relating to Measures of Effectiveness

The first pitfall in PSYOP, particularly in more strategically-oriented programs, is having a non-specific Target Audience (TA), which results in error with regards to understanding and evaluating the Measures of Effectiveness (MOEs).^[xx]

According to MISO doctrine, the TA should be very specific, and at least four levels of specificity are recommended when targeting any group.^[xxi][xxii] However, in practice, this requirement often is either forgotten or fails to be properly implemented. The reasons for this trend are varied: there is either not the depth of analysis, the mission calls for more flexibility, or the demands of the Operational Commanders are such that specific TAs are ignored in favor of broader Operational or Strategic level messaging—mandated by approval authorities and individuals that are unfamiliar with PSYOP.

Psychological Operations are often designated as series, which are the overarching plans required to determine applicable product development. The high degree of specificity required by PSYOP doctrine tends to restrict the effectiveness of those operations, most particularly where the time constraints for series approval are concerned. The amount of series that would have to be created for a doctrinally correct MISO program at the strategic level is infeasible. Similarly, it is not an easy or a quick process to approve an entire MISO series. A broad TA allows PSYOPers an amount of flexibility to message the population of an approved series.

When MOE is collected and analyzed in respect to the entire TA, the data that is collected will be not provide any discernible trends. An example of this concept would be to initiate a public media campaign to inform and influence the American population’s perception of the DOD opening up combat branches to women. PSYOP is not legally allowed to target American populations—this is an example to see how a contracted PR firm would run such a campaign on behalf of the DOD. If a wide TA was assigned to this mission, it would encompass the entire American population. Without adequately assigning sub-TAs to account for differences in political views, social values, and religious followings, the results of the data on the media campaign would effectively be indiscernible: Democrats’ change in views would, likely, cancel out Republicans’, religious Christians may, generally, measure in an opposite direction from atheists and agnostics, and so forth. It would be impossible to accurately measure how effective the DOD was at changing the levels of acceptance regarding women serving in combat arms using the approach explained.

The proposed solution is to create broad MISO series, with narrow groups being targeted for messages, and subsequently, for MOE. To continue using the Democrat and Republican example, imagine if the population of America was asked to give their opinions on a social policy. Without separating the respondents by their political affiliation, even decades of trend collection and analysis would show a flat line and the data would be of little use to any politician attempting to understand and separate Republican voters in Florida to Democratic voters in Connecticut.

Civilian campaigns neither attempt to target every individual in a country, nor do they evaluate their success off of the entire population. Political campaigns focus on key demographics in regional locations. The same is true for any brand in the marketing environment. Everyone in the country may view the products and information being disseminated by a campaign, but the products are still tailored to appeal to niche groups. Those niche groups are, in turn, the same populations analyzed to measure the success of the campaign.

Even in these marketing examples, a specific product is used to target a specific audience, and that audience is measured for effect and for future targeting. PSYOP should be performed in the same way.

The Second Pitfall: Failure to Properly Document and Track Impact Indicators

The second pitfall is the improper documenting and tracking of impact indicators.^[xxiii]

Impact indicators differ from MOE and MOP, in that impact indicators are specific events and are often benchmarks that the practitioners wish to achieve. As is common in PSYOP doctrine, there is a lack of definitive information concerning impact indicators. The MISO manuals often give too little information, or contradict each other. Impact indicators are not solely used by the military—they are better described by civilian institutions:

Impact indicators describe progress made towards higher-level goals. They are akin to statements of purpose, describing those objectives that are shared with other development partners and national government agencies, such as reducing poverty, increasing access to justice, or improving the accountability of national institutions.^[xxiv]

The greatest contributor to this pitfall is the misunderstanding of the difference between Measures of Performance (MOP) and MOE. This is a long standing issue within the branch that has been addressed before, but not with systemic success.^[xxv] Measures of Performance are the numbers of actions performed. Examples of MOP are the number of phone calls received for a tip line, or the amount of leaflets dropped on a PSYOP.

PSYOPers frequently use the number of disseminations of the PSYOP message—the MOP—as the MOE, which is invalid reasoning because the mere receipt of a message is not an actual effect: it is only the catalyst of an effect, and thus why it is only a performance measure. It is imperative to measure both MOP and MOE, though, because MOE is the way the actions taken are measured; the MOP is the quantification of those actions. This is shown in more detail in Table 1, below.

Table 1: Difference Between MOP / MOE / Impact Indicators

Another frequent contributor to the MOP/MOE confusion is the tendency of PSYOPers to generate largely adequate MOE without also generating a plan to collect MOE data. More specifically, criteria that would make excellent MOE is established, but there is no process or ability for the PSYOP unit to collect such data.

The key to correctly measuring effectiveness is applying depth to the analysis early on and consistently throughout the process. As an example, McDonald’s decides to target “males aged 8-12 who view Saturday morning cartoons and live in Florida” for a campaign. They have a specific TA, and they also gather data before starting, so that they may compare the end results with the data collected before the campaign. McDonald's is ensuring that they will appeal to this audience by specifically creating a Happy Meal with a toy designed to appeal to boys aged 8-12. McDonald's decides that they want to disseminate their advertising in the entire South, but they will still only measure effectiveness in Florida.

For this example, McDonald's has the ability to track credit card data and can compare that data to census information. They use that information to find the number of people living in Florida who have kids, and are purchasing Happy Meals. McDonald's ensures that they establish a baseline by reviewing past data. McDonald’s will probably pay for TV ad space on Saturday mornings on cartoon channels.

Planning is essential throughout the process of creating a series, and collecting MOE should be considered from the very beginning of the planning process.^[xxvi] This is a point stated in many locations within PSYOP doctrine, but not done consistently in practice.^[xxvii] MOE should be finalized before any products are even created, as understanding the MOE and developing baseline data enables the MOE process to have a starting point. Solidifying MOE includes knowing how the data will be collected, who will be collecting it, how often it will be collected, whether it will be qualitative or quantitative data, who will be analyzing it, and what format the MOE data will be in so that it may be included in the results for the series evaluation.

Part of the reason that psychology has come such a long way in the scientific community is because it has oriented itself toward the scientific methods of other practices. Writing peer-reviewed journal articles based off of research and experiments has helped propel the profession of psychology to heights in the scientific community that were inconceivable just a few decades ago. PSYOP has yet to make equally great strides with regards to MOE because the branch as a whole has yet to implement scientific methodology within its programs, as well as a systematic and professional method of documenting findings. Both of these desiderata are absolutely paramount to increasing the overall effectiveness of the branch.

The Third Pitfall: Substandard Reporting Practices

The third pitfall is the substandard reporting practices implemented within the community. When operations are successful those successes are often lost to all but the members of the MISO teams executing those successful operations.

MOE are often flawed in MISO series because they are usually more of a reflection of MOP, or an unjustified logical leap, than they are a scientific process. While the current reporting methodology may satiate the requirements for congressional oversight, it does nothing for understanding and tracking PSYOP efforts over time, or even demonstrating the relevancy of PSYOP as a whole. Accuracy is paramount, and correcting the first two pitfalls is a step in the right direction when it comes to reporting MISO series. Creating a report that documents the evaluation of a MISO series, though, is inarguably the most important step.

Effectiveness is more than a bullet point that increased or decreased, it has to be a conversation and a logically formed argument that demonstrates that there is a very high possibility that the changed behavior was a result of a PSYOP team’s efforts.

Reports should be used as a MISO hub of information, where successful MOE are broken down and sent out to the community and taught in the PSYOP Qualification Course to incoming PSYOPers in training. Instead, the current loose MOE are stored in local network share drives or on external hard drives, in a folder within a folder within a folder, never to be opened again. The community as a whole has no forum for PSYOP successes to be spread or discussed or learned from.

Knowledge management is a systemic issue within the branch. There is no unit designated as the MOE unit within PSYOP. There is no effective way to learn from the mistakes of failed series, or learn from the successes of perfectly executed series beyond word of mouth and mentorship from senior leaders. To further compound the problem, because current PSYOP operations do not follow consistent scientific methods, there is no consistent way to compare operations.

The purpose of evaluation in the civilian and scientific communities is to use the information to make better decisions that will be more likely to effectively persuade, change, and influence the selected TAs. Without proper reporting and proper storage of reports, this cannot be accomplished. If McDonald's and Hollywood have this figured out, why can’t America’s Special Operations community understand the validity behind such reasoning? PSYOP should be at the forefront of researching and understanding behavior change and effectiveness, and the branch should make more earnest efforts to demand and encourage the evaluation of data and then the instruction of that information to the current and incoming members of the branch.

Fixing the Measurement Problem

PSYOP’s understanding of MOE can be revitalized by recognizing and utilizing the Hierarchy of Effects model (HOE). The HOE model is a scientifically recognized methodology of in the fields of marketing and psychological influence. The Dictionary for Public Relations Measurement and Research defines this model using its definition for attitude, which states that it is, “a predisposition to act or behave toward some object; a motivating factor in public relations; composed of three dimensions: affective (emotional evaluation), cognitive (knowledge evaluation), and connotative (behavioral evaluation).”^{[xxviii][xxix]} The HOE model is often referenced as the CAB model—an acronym denoting the cognitive phases that an individual must progress through in order to achieve true behavioral change: the Cognitive, Affective and Behavioral Phases.^[xxx]

The Cognitive Phase

This phase hinges on the assertion that an individual must first have awareness of an idea, and understand that idea, in order to eventually behave in a way congruent with that idea.^{[xxxi][xxxii]} The PSYOP community habitually confuses the Cognitive Phase as a MOE, when in actuality it is a MOP. For example, a MISO team plans to disseminate 500 handbills a day in two cities every Wednesday for a month. The handbill will attempt to persuade the TA to call a 911 hotline to report criminal activity in the area. Often, the PSYOP team will report 4,000 handbills disseminated as a MOE. Merely disseminating does not allow the team to collect information on whether the TA knows the purpose for which 911 is used, and what happens when they use it, or if the TA dialed the number because of the handbill. In order to begin to assert that a MISO product has an effect, the MISO organization must first ensure that the TA has completed the Cognitive Phase by showing that the TA both received the message, and that the TA understood the message.

The Affective Phase

The second phase of the model is the Affective Phase.^[xxxiii] This emotion-focused phase analyzes how the TA progresses in terms of favorability or conviction. Once a person understands the message being conveyed to them, they will start to sway one way or another in how they feel toward that idea. If the message being distributed to the TA is fashioned correctly, it should play on the cognitive dissonance that the TA experiences with the given idea.

Cognitive dissonance exists when any person has two competing values or schemas.^[xxxiv] The friction between the Christian value set and the militarist value set creates cognitive dissonance, and this friction is often solved by either justification, whereby an individual invents or believes in a current rationale for making their behavior palatable and comprehensible to themselves; or via compartmentalization, whereby an individual attributes a competing value set to situation-specific experiences.^[xxxv]

A person satiates their conscience when they have competing values by attributing specificity to those values—that is, they identify certain instances where it is okay to act in one way, and other situations where the opposite action is more appropriate. Examples of this pathology can be seen through such thoughts as ‘It’s okay for me to kill this person because it’s for my nation,’ or ‘It’s okay for me to not call 911 even though I know it’s morally right to report criminals because I’m keeping my head down.’

Cognitive dissonance is the raison d’etre for the PSYOP branch, and the majority of the branch’s efforts should subsequently focus on exploiting psychological chasms created by this dissonance in order to persuade TAs to adopt values in line with US, Partner Nation and strategic interests. When the TA feels more strongly about the desired value, they have developed a preference for the idea. At this point, they have completed the Affective Phase that is required for behavior change.

Developing an intention is a first step towards objective change and can be measured through verbal commitments, such as the number of people responding affirmatively to the question ‘Would you report illegal activity on the 911 line?’ PSYOPers can measure if their campaign is effective by how dramatically the number of people saying yes rises over time in relation to the baseline data established during the creation of the series. This shows an increase in the intention to perform the behavior. It also addresses the cognitive and affective phase—the TA knows of the desired behavior and responds to it.  PSYOPers must ensure that they capture true objective data for MOE, rather than self-reported and likely skewed data collected through surveys that collect intention.

The Behavioral Phase

The final phase of achieving a desired behavior change is the Behavioral Phase.^[xxxvi] This is the phase in which the TA actually implements an objective change. This behavioral change is objective in the sense that it should be observable, measurable, and specific.^[xxxvii]

Objective behavioral data can be collected through a variety of methods—networking with the Partner Nation that is working with the PSYOP team or even using dedicated intelligence (S2) support. The S2 has an existing capability for collecting intelligence from various sources that can support MOE. In order to successfully and thoroughly collect and document MOE, integration of and coordination with the S2 shops at every applicable echelon is essential.

Connecting the Cognitive Phase, Affective Phase, and the Behavioral Phase can aid the MISO teams in making a strong argument for evidence of effectiveness. The ability to show that a TA received, understood, inculcates and then acts upon those beliefs in line with the desired behavior allows PSYOPers to connect the dots for measuring effectiveness.

Fixing the Reporting Problem

The branch of PSYOP is a professional organization that has earned its role in the military community. What it is lacking, however, is standardized and peer-reviewed research—the reports that each PYSOP organization should publish on the series that they have implemented. Lessons learned are often forgotten, and one team’s success does not lead to a future team’s success.

Psychological Warfare has to be constantly reassessed and improved upon to meet the increasingly complicated demands of the contemporary operating environments in which PSYOP functions. To improve the professionalism and effectiveness of the branch, it must implement a rigorous cycle of assessment, planning, and execution of operations. This system would include analysis of previous campaigns, capturing and documenting of best practices that have been proven through scientifically-credible metrics, and knowledge management techniques that would make these best practices accessible to the entire PSYOP community and ensure they are incorporated into the curriculum of the various services’ qualification courses.

Failure to implement such a system will result in the branch using psychology only in name and not in actual practice. In order to reach this level of analysis, it is imperative that thorough assessment reports be generated regarding the specific instances leading up to the selection of a TA, the exploitation of that TA’s psyche, and the resulting shifts in cognitions, affect, and behaviors. By standardizing these reports, the branch will be able to begin to pick out the outliers of success and learn from those successes so that those efforts may be duplicated.

Having a report that highlights successful PSYOP is equally important to having a report that highlights unsuccessful PSYOP. In fact, the unsuccessful studies can be more telling. By reporting the pitfalls, millions of dollars may be saved from fruitless endeavors and instead used to make the PSYOP branch increasingly effective.

In a perfect world, there would be a unit assigned to maintain these reports. There is a print and product creation Battalion (BN) within the PSYOP Groups. An option available would be for PSYOP to create a MOE BN that regularly archives and structures the lessons learned into cohesive reports, briefs teams before they deploy, and debriefs teams on their return. This BN would, ideally, be staffed with Behavioral Scientists with an understanding of cognitive analysis.

Another method would be to implement a MOE Detachment at the Group level. One for each PSYOP Group. These Detachments could rotate throughout the missions in each Area of Operation (AO) for the separate BNs. Such a section has existed in PSYOP and was implemented in the Afghanistan mission with the Military Information Support Task Force—Afghanistan. The section was named the Validation, Testing and Evaluations Detachment (VTED) and it was staffed by contracted civilian Behavioral Scientists with security clearances. They used surveys and modeled the MOE argument off of the HOE model very successfully. They also integrated with the S2 on several occasions, and the most successful iterations of the VTED were when the Detachment was fully integrated with the S2 and strictly followed the HOE model. By establishing VTEDs at the Group level, the oversight of MOE could be maintained by the Group Commanders and MOE reporting could be more easily standardized. The VTEDs could also be responsible for storing the reports and allowing for the reports to be easily accessed by PSYOP teams with similar mission sets.

Conclusion

Successfully understanding and communicating the MOE of a series is how PSYOP proves its legitimacy as a branch and continues to grow within the Special Operations community.

Attention to the details of effectiveness is paramount in the understanding and implementation of PSYOP. The PSYOP community has always been a community filled with potential. Psychological Warfare will always be present on the battlefield, and has been since the earliest historical accounts. In order to remain at the cutting edge, it is important for PSYOP to continue to improve as a branch by understanding the tradecraft that PSYOP is based upon.

The PSYOP community would dramatically increase its effectiveness and credibility by specifically correcting three common institutional pitfalls: having non-specific target audiences to measure effectiveness, failure to properly document and track impact indicators, and substandard reporting practices. Failing to address these pitfalls will only continue the trend of drawing inaccurate conclusions from well-conceive, well-planned and costly operations. It is important that PSYOPers understand that psychological research is necessary in order to improve upon and understand ways to change and affect human behavior.

End Notes