Work

Empirical Elephants—Why Multiple Methods are Essential to Quality Research in Operations and Supply Chain Management

Year: 2008
Type: article
Abstract: Much like the blind men and the elephant, researchers in the field of operations and supply chain management (OSCM) often study natural phenomena (business processes are our "elephants") using widely varied perspectives. Though we seek objectivity, each of us is prone to form an attachment to a given perspective, influenced by our training and by the perspectives of our mentors and peers. As editors, numerous conversations with researchers have given us cause to be concerned that there is a tendency in the academic community to discount the findings of others who are using their different "senses" (research methods) to examine various parts of the elephant. In this editorial, we take the opportunity to briefly review a variety of research methodologies. It is our strong belief that multiple approaches are required in order to develop a holistic understanding of operations and supply chain management phenomena. Only by being open to reading, digesting and synthesizing multiple perspectives can we get a true picture of the elephant. For a couple of reasons, we limit our comments to address only empirical research methods. First, while recognizing that modeling and purely analytical techniques have produced large advances in OSCM, we also take the position that the study of OSCM is a social science. Research in OSCM is more than problem solving; it is the study of business processes with the objective of developing theories that explain them. Business processes contain both natural and social phenomena. As such, while systems and decisions affecting business processes can be modeled, examinations of empirical data are crucial for the development and validation of models. It is especially important that we uncover the often complex social and behavioral elements involved in OSCM. Biological sciences can explain the universal characteristics of an elephant, but in practice different elephants have personalities and individual characteristics that must be observed in order to be understood. A second reason for limiting this discussion to empirical methods is the fact that the Journal of Operations Management is entirely dedicated to the publication of theory-driven, empirical research. The JOM is the only OM journal stating such a focused mission, and we are committed to it. At the same time, we would not want the journal to become too focused in terms of the empirical methodologies employed. We want to encourage research employing a wide range of perspectives and methods. In the following paragraphs we very briefly examine the benefits and limitations of several broad methodological approaches, including primary data collection (survey, case, experiments) and secondary data collection. This is clearly not an exhaustive guide or comparison of empirical techniques. There are numerous resources for this, including the recent article by Roth (2007). Our goal is to remind readers that every approach has good and bad points. There is no universally superior methodology. In the discussion we also present several recent or forthcoming JOM articles that exemplify each approach. Finally, we discuss potential synergies between the various approaches which can provide a holistic view of the elephant. Following the seminal work of Flynn et al. (1990), there has been a notable growth in empirical research generally, and in survey-based research in particular. While Flynn et al. (1990) provided an overview of numerous empirical techniques, their work focused on survey-based research. Survey research is a common methodological approach in the fields of management and marketing, with well-established protocols that are summarized by Flynn et al. (1990) and many others. The question is not whether survey-based research is appropriate, but when is it appropriate? A survey provides a low cost, non-invasive means for measuring aspects of an operational or supply chain issue. Measures included in a survey can be designed to target specific factors or attributes which may not be directly observable. For example, many behavioral variables affecting an operational process are "latent," and can be assessed only via perceptual measures. Moreover, even directly observable attributes are often not measured by organizations, either because they are not of interest or because they cannot be measured in a cost effective manner. Frequently, measures that are published by organizations do not report data at the needed unit of analysis. For example, measures of costs or sales at the plant or lower levels are not typically available from published sources. Via a survey, data can be collected directly from the individual or unit directly responsible for managing a certain operation of interest. Much of the controversy surrounding surveys has to do with the use of perceptual measures. The use of perceptual measures is not necessarily wrong. In fact, they may be necessitated by the nature of the variable of interest. This is clearly the case when variables are a function of behaviors or organizational norms. Even when variables are materially observable, a perceptual measure may be more efficient due to problems with comparability. Take, for instance, the case of measuring operating costs. In many cases an OSCM researcher is not interested in the absolute cost performance of an operating process; he/she is interested in the unit's costs relative to those of comparable operating units. The problem, of course, is that rarely are any two operating units completely comparable. Moreover, the adjustments required to achieve comparability are not feasible given the researcher's limited resources and access to information. Consequently, it becomes efficient to rely on a respondent's rating of relative costs, given his/her expert knowledge and ability to mentally adjust cost performance estimates for relevant differences between operating units. Under these circumstances, a respondent's "perceptual" estimate of relative cost may be more accurate than an estimate provided by reports created by the organizations' respective accounting systems. While a more objective measure of cost would be ideal, it might be infeasible to attain. Survey-based research also has important limitations. A well known difficulty with perceptual measures is the potential for measurement errors stemming from subjectivity and bias. These issues have been thoroughly discussed in the literature, and numerous controls and remedies have been forwarded as means for dealing with the limitations of perceptual measures. In the case that the survey is the only source of data used in the study, common method bias is of course also a serious concern. Again, the literature offers preventive measures and controls. It is clearly important that researchers take the utmost care to lessen and control for sources of bias, thereby improving confidence in the validity of the research findings. This need is important for any research methodology, but bias is perhaps a most salient threat in survey research using perceptual measures. Other concerns related to survey research address difficulties with respondents' interpretations of measures, potential lack of knowledge, and representations of the unit of analysis. In addition, we have heard a common lament that business managers, particularly in manufacturing, are suffering from survey fatigue, and increasingly unlikely to respond to surveys. While each of these limitations is important, none necessarily represents a fatal flaw in the methodology. Numerous publications in the operations and broader business literatures offer suggestions on how to mitigate inherent problems. Bad research using any methodology usually starts with poor, or even lazy, research design. This is analogous to a blind man examining an elephant's foot. If he isn't careful, he is likely to get stepped on. Rabinovich et al. (2007) use transaction cost theory and data from a survey of 196 internet commerce firms (ICFs) to show that low levels of asset specificity and uncertainty drive ICFs to establish relationships with third party logistics firms. Notable in this research is the evidence explaining why firms choose to establish multiple party relationships across a supply chain. The findings refine the higher level observation from publicly available data that these relationships exist. Schoenherr and Mabert (2008) use 252 responses from purchasing professionals to show that a more homogenous bundle of goods/services in the request for quotation process leads to better outcomes. This study illustrates how a potential shortcoming, a low response rate of 5.4%, can be mitigated and must be evaluated within the specific study context. The authors use a carefully targeted contact sample and employ several measures to assess non-respondent bias. The end result is a study that offers critical insight to a modern tool for purchasing professionals (online reverse auctions), captures important behavioral factors, and mitigates the limitations through the use of appropriate study design and application of statistical tools. Survey research has clearly been the most common methodology represented in JOM articles in the past. We expect that this methodology will continue to be well represented in the journal, and we will continue to insist that the highest standards and the best practices for ensuring reliability and validity are maintained. Case or field-based studies provide a qualitative approach to studying a phenomena in-depth, particularly poorly understood or emerging phenomena. Primarily used as a theory-building approach, cases studies have been effectively employed in a large variety of situations, and excellent guides for conducting such research exist in both the broader business (Eisenhardt, 1989; Yin, 1989) and in the operations management literature (Meredith, 1998). Benefits of case studies include the ability to examine a topic in great depth. Researchers can focus on a specific topic and/or company(s), allowing a thorough examination of numerous factors and nuances. Case studies provide a richness of description and first-hand observation of phenomena in a natural setting. Often ***case studies yield unintended insights, which can lead to new avenues of inquiry. The best case studies provide a foundation for further examination. Limitations of case studies include cost and time, inability to generalize and prescribe, and potential for bias in the perceptions of the researchers. This research approach, when well done, can be expensive in terms of both financial cost, as well as the time spent by both researchers and participating managers. While case studies can provide great depth, it is difficult to generalize results from a few companies to a broader population. Finally, since there is a high degree of involvement of researchers with the subjects, there is a good chance of bias in terms of interpretation. As with other research techniques, each of these and other limitations can be substantially mitigated with the use of proper techniques. Wu and Choi (2005) examine eight cases, gathering data from pairs of suppliers to identify five archetypes of supplier–supplier relationships. This study aptly demonstrates how careful and thorough application of appropriate case study techniques can yield substantial insights and provide a strong platform for future research. For this reason, the article by Wu and Choi (2005) was recognized as the best JOM paper of 2005. Closs et al. (in press) study business units in six Fortune 500 companies to better understand drivers of product portfolio complexity and important competencies for managing them. Complexity management competencies are explicated using a socio-technical systems theoretical perspective. Case based research can be seen as risky, as it is time consuming and may yield fewer publication opportunities per research hour of effort. For this reason, advisors sometimes caution their doctoral students against conducting case research early in their careers. However, this type of research is excellent for establishing a foundation for follow-on research streams. Many theory-building opportunities remain in the field of OSCM, and we encourage the publication of more case study research in the JOM. The primary advantage of experiments is that they provide a high degree of control over threats to internal validity by limiting influences that might otherwise explain observed relationships between independent and dependent variables. Through controls and random assignments to treatments, specific effects can be isolated, and strong inferences of causality can be developed. The obvious limitation to experiments pertains to external validity, as the laboratory environment creates an artificial setting that may not adequately represent true decision making conditions. Carter and Stevens (2007) conducted experiments involving 97 MBA and business Ph.D. students in order to study the interplay of electronic reverse auction parameters. Their findings indicate the relative effects of price information, number of competing bidders, and supplier need for contract have on final bid price and perceptions of buyer opportunism. Bendoly and Swink (2007) designed experiments studying the effects of various levels of information availability upon the actions and perceptions of 362 MBAs in an environment where multiple projects are pursued simultaneously. The findings suggest that greater visibility of situational information impacts project outcomes mainly by affecting a project manager's perceptions regarding both the behavior of project managers and the priority given to his/her task. The study extends existing information processing theory by developing and examining notions of informed rationality and reciprocity, as well as examining the effect of information on post-task sense making. As interest grows in studying the effects of behavioral and cognitive processes on decision making in OSCM, we expect to see more experiments in our field. Seawright and Sampson (2007) provide an interesting empirical examination of wait perception bias. They use a video method of data collection to study customer perceptions, finding that the presence of an explanation for long waits is helpful. Interestingly, it makes little difference whether the explanation focuses on customers or servers as the cause of waiting. This type of work provides important theoretical grounding and refinement of the intersection between psychological and operations management principles. The growing use of communications and information systems to collect data throughout the supply chain has created numerous opportunities for research that employs secondary data. The availability of data collected by corporations, government agencies, news agencies, industry groups, and other parties is growing very rapidly. A pair of recent articles (Roth, 2007; Fisher, 2007) have called for more research using secondary data, and examined some of the benefits and limitations involved with this type of data. We wholeheartedly endorse and support the use of such data and believe it is of great importance for the future of OSCM research. Benefits of secondary data sources include wide availability and lower data collection expense. For example, collaboration of companies using IT systems to exchange data offers greater opportunities to collect data across multiple supply chain entities. Secondary data are generally more "objective" than self-reported survey data, yet the quality of the data is dependent on the standards and protocols that govern their collection, and these factors are usually outside the control of the researcher. Since the researcher is dependent on another party for data collection, she/he must assume that the data were collected correctly. Other limitations of secondary data research also exist. First, available data might not include measures that tap the construct of interest exactly. Researchers are frequently left with proxy measures that may only partially represent or reflect the theoretically relevant variables. A second limitation is that data from computers or reports are not always accurate, or devoid of bias. Many data (e.g., financial reports) are quite noisy, making statistical significance of hypothesized relationships hard to establish. In addition, errors and misreports in data can be systemic, as Enron and accounting scandals have shown us. After all, the observations of people are often the source of most business data, even if it is ultimately delivered via a computer. Third, while there are vast amounts of data locked away in corporate databases—it takes careful work and high levels of trust to gain access to such data. Many of us have experienced the disappointment of being denied promised data despite having signed numerous non-disclosure agreements and having promised confidentiality to the point of exhaustion. A popular study by Hendricks and Singhal (2003) estimates the shareholder wealth affects of supply chain "glitches" that resulted in production or shipment delays using a sample of 519 public announcements made during 1989–2000. The insight that "Supply chain glitch announcements are associated with an abnormal decrease in shareholder value of 10.28%" (Hendricks & Singhal, p. 501) is tremendously motivating to managers and academics alike. Accordingly, this paper was awarded the annual JOM best paper prize. Shah and Shin (2007) use data collected from Bureau of Economic Analysis at the U.S. Department of Commerce to examine relationships between inventory, IT investment and financial performance. The study demonstrates that inventory levels have changed non-uniformly across three sectors (manufacturing, wholesale and retail). In addition, the results provide further support for the absence of a direct link between IT investment and financial performance. These descriptive results prompt some important questions for future research. Heim and Field (2007) offer an examination of over 1000 online retailers via data collected from an online site containing customer ratings of e-service quality. Their study points to some of the key drivers of customer satisfaction. Cousins et al. (2006) use results of a survey to highlight the importance of informal socialization processes in creating relational capital and improving supplier relationships/outcomes. While IT helps facilitate quick and seamless exchange of routine transactional information, their work shows that informal relationships between buyers and sellers facilitate the ability of useful and important information to travel rapidly and accurately through a supply network. Fransoo and Wiers (2006) use a case study approach to examine some of the psychological factors underlying the application of MRP and ERP planning processes. The detailed case study of a single company profiles daily downloads from the MRP planning system of planned versus actual released orders. The findings indicate that increased task complexity, without time pressure, leads to an increase in variety of actions. As noted by Fransoo and Wiers (2006), such research into the behavioral factors and a "better understanding of human planners is needed to develop better training programs for schedulers" (p. 821) in order to go beyond limited training consisting of a "press this button now" approach. Mantel et al. (2006) administer a controlled-experiment in which 305 managers were asked to complete a set of responses to various purchasing planning scenarios. Their study produces evidence indicating that the approach to making a simple make-buy decision is not necessarily rational, but can be biased based on several factors. In particular, strategic vulnerability and core competency both have substantial influence on the end decision. This work offers parallels with the pathbreaking work of economists Daniel Kahneman and Vernon Smith, 2002 Nobel Prizewinners, in showing that human beings do not make decisions with complete economic rationality. While somewhat less prestigious than a Nobel prize, the work of Mantel et al. (2006) was recognized with the JOM best paper prize at the 2007 Academy of Management conference. Boyer and Hult (2005) combine customer survey data from two online grocery companies to predict actual customer purchases (data derived from corporate databases)—a technique that combines survey and secondary data. Bendoly and Cotteleer (2008) utilize a pair of case studies to examine circumvention of IT-systems rules following implementation of ERP systems. The case studies are carefully selected to pair examples of sustained versus non-sustained compliance with IT-supported protocols using the same ERP package. The data from the case studies were supplemented by a controlled-experiment involving written scenarios and corresponding questions presented to 335 managers. The study findings indicate that users of IT systems may "retain strong intentions to circumvent systems in the presence of perceived task-technology misfit" (Bendoly and Cotteleer, 2008, p. 23). The combination of case study and controlled-experiment provide important validation for the findings. A body of research made up of a variety of research methodologies is not unlike a diversified portfolio of financial holdings. The collection is more likely to yield highly productive outputs with lowered risks (in this case, lowered risk of biased findings). Our review of a handful of recent articles published in JOM highlights the variety of approaches employed by our authors. We applaud and encourage such diversity of empirical approaches in the Journal of Operations Management. While we cannot promise to personally possess expertise in every potential methodology, we can promise to do our level best to find appropriate reviewers and to work to expand our portfolio of tools. In closing, we want to reiterate our belief that all methodologies have something to offer. Moreover, the merits of one methodology do not necessarily invalidate another. It is human nature to hold the greatest appreciation for things with which we are most familiar and comfortable. We believe that OSCM research, and the OSCM community as a whole, will benefit if we can set aside our biases in an effort to understand and appreciate the benefits and limitations of all empirical methodologies. After all, blind men use all their senses to compensate for the lack of vision. Why should we as researchers disparage any avenue of inquiry that will help describe the elephant?
Language: English
Open Access status: bronze
OpenAlex ID: W2153954935
FWCI: 42.69
Cites: 23
Cited by: 317
Related to: 10