Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology
Year: 2025
Type: article
Abstract: Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.
Source: BMC Biology
Authors: Elliot Gould, Hannah S. Fraser, Timothy H. Parker, Shinichi Nakagawa, Simon C. Griffith, Peter A. Vesk, Fiona Fidler, Daniel G. Hamilton, Robin N. Abbey-Lee, Jessica K. Abbott, Luis A. Aguirre, Carles Alcaraz, Irith Aloni, Drew Altschul, Kunal Arekar, Jeff W. Atkins, Joe Atkinson, Christopher M. Baker, Meghan Barrett, Kristian Bell, Suleiman Kehinde Bello, Iván Beltrán, Bernd J. Berauer, Michael Grant Bertram, Peter D. Billman, Charlie K. Blake, Shannon Blake, Louis Bliard, Andrea Bonisoli-Alquati, Timothée Bonnet, Camille Nina Marion Bordes, Aneesh P. H. Bose, Thomas Botterill-James, Melissa Anna Boyd, Sarah A. Boyle, Tom Bradfer-Lawrence, Jennifer Bradham, Jack A. Brand, Martin I. Brengdahl, Martin Bulla, Luc Bussière, Ettore Camerlenghi, Sara E. Campbell, Leonardo L. F. Campos, Anthony Caravaggi, Pedro Cardoso, Charles J. W. Carroll, Therese A. Catanach, Xuan Chen, Heung Ying Janet Chik, Emily Sarah Choy, Alec Philip Christie, Angela Chuang, Amanda J. Chunco, Bethany L. Clark, Andrea Contina, Garth A. Covernton, Murray P. Cox, Kimberly A. Cressman, Marco Crotti, Connor Davidson Crouch, Pietro B. D’Amelio, Alexandra Allison de Sousa, Timm Fabian Döbert, Ralph Dobler, Adam J. Dobson, Tim S. Doherty, Szymon Marian Drobniak, Alexandra Grace Duffy, Alison B. Duncan, Robert P. Dunn, Jamie Dunning, Trishna Dutta, Luke Eberhart-Hertel, Jared Alan Elmore, Mahmoud Medhat Elsherif, Holly M. English, David C. Ensminger, Ulrich Rainer Ernst, Stephen M. Ferguson, Esteban Fernandez-Juricic, Thalita Ferreira-Arruda, John Fieberg, Elizabeth A. Finch, Evan A. Fiorenza, David N. Fisher, Amélie Fontaine, Wolfgang Forstmeier, Yoan Fourcade, Graham S. Frank, Cathryn A. Freund, Eduardo Fuentes-Lillo, Sara L. Gandy, Dustin G. Gannon, Ana I. García-Cervigón, Alexis C. Garretson, Xuezhen Ge, William L. Geary, Charly Géron, Marc Gilles, Antje Girndt, Daniel Gliksman, Harrison B. Goldspiel, Dylan G. E. Gomes, Megan Kate Good, Sarah C. Goslee, J. Stephen Gosnell, Eliza M. Grames, Paolo Gratton, Nicholas M. Grebe, Skye M. Greenler, Maaike Griffioen, Daniel M. Griffith, Frances J. Griffith, Jake J. Grossman, Ali Güncan, Stef Haesen, James G. Hagan, Heather A. Hager, Jonathan Philo Harris, Natasha Dean Harrison, Sarah Syedia Hasnain, Justin Chase Havird, Andrew J. Heaton, María Laura Herrera-Chaustre, Tanner J. Howard, Bin-Yan Hsu, Fabiola Iannarilli, Esperanza C. Iranzo, Erik N. K. Iverson, Saheed Olaide Jimoh, Douglas H. Johnson, Martin Johnsson, Jesse Jorna, Tommaso Jucker, Martin Jung, Ineta Kačergytė, Oliver Kaltz, Alison Ke, Clint D. Kelly, Katharine Keogan, Friedrich Wolfgang Keppeler, Alexander K. Killion, Dongmin Kim, David P. Kochan, Peter Korsten, Shan Kothari, Jonas Kuppler, Jillian M. Kusch, Malgorzata Lagisz, Kristen Marianne Lalla, Daniel J. Larkin, Courtney L. Larson, Katherine S. Lauck, M. Elise Lauterbur, Alan Law, Don-Jean Léandri-Breton, Jonas J. Lembrechts, Kiara L’Herpiniere, Eva J. P. Lievens, Daniela Oliveira de Lima, Shane Lindsay, Martin Luquet, Ross MacLeod, Kirsty H. Macphie, Kit Magellan, Magdalena M. Mair, Lisa E. Malm, Stefano Mammola, Caitlin P. Mandeville, Michael Manhart, Laura Milena Manrique-Garzon, Elina Mäntylä, Philippe Marchand, Benjamin Michael Marshall, Charles A. Martin, Dominic Andreas Martin, Jake Mitchell Martin, April Robin Martinig, Erin S. McCallum, Mark McCauley, Sabrina M. McNew, Scott J. Meiners, Thomas Merkling, Marcus Michelangeli, Maria Moiron, Bruno Moreira, Jennifer Mortensen, Benjamin Mos, Taofeek Olatunbosun Muraina, Penelope Wrenn Murphy, Luca Nelli, Petri Niemelä, Josh Nightingale, Gustav Nilsonne, Sergio Nolazco, Sabine S. Nooten, Jessie Lanterman Novotny, Agnes Birgitta Olin, Chris L. Organ, Kate L. Ostevik, Facundo Xavier Palacio, Matthieu Paquet, Darren James Parker, David J. Pascall, Valerie J. Pasquarella, John Harold Paterson, Ana Payo-Payo, Karen Marie Pedersen, Grégoire Perez, Kayla I. Perry, Patrice Pottier, Michael J. Proulx, Raphaël Proulx, Jessica L Pruett, Veronarindra Ramananjato, Finaritra Tolotra Randimbiarison, Onja H. Razafindratsima, Diana J. Rennison, Federico Riva, Sepand Riyahi, Michael James Roast, Felipe Pereira Rocha, Dominique G. Roche, Cristian Román-Palacios, Michael S. Rosenberg, Jessica Ross, Freya E. Rowland, Deusdedith Rugemalila, Avery L. Russell, Suvi Ruuskanen, Patrick Saccone, Asaf Sadeh, Stephen M. Salazar, Kris Sales, Pablo Salmón, Alfredo Sánchez-Tójar, Leticia Pereira Santos, Francesca Santostefano, Hayden T. Schilling, Marcus Schmidt, Tim Schmoll, Adam C. Schneider, Allie E. Schrock, Julia Schroeder, Nicolas Schtickzelle, Nick L. Schultz, Drew A. Scott, Michael Peter Scroggie, Julie Teresa Shapiro, Nitika Sharma, Caroline L. Shearer, Diego Simón, Michael I. Sitvarin, Fabrício Luiz Skupien, Heather Lea Slinn, Grania Polly Smith, Jeremy A. Smith, Rahel Sollmann, Kaitlin Stack Whitney, Shannon Michael Still, Erica F. Stuber, Guy F. Sutton, Ben Swallow, Conor Claverie Taff, Elina Takola, Andrew J. Tanentzap, Rocío Tarjuelo, Richard J. Telford, Christopher J. Thawley, Hugo Thierry, Jacqueline Thomson, Svenja Tidau, Emily M. Tompkins, Claire Marie Tortorelli, Andrew Trlica, Biz R. Turnell, Lara Urban, Stijn Van de Vondel, Jessica Eva Megan van der Wal, Jens Van Eeckhoven, Francis van Oordt, K. Michelle Vanderwel, Mark C. Vanderwel, Karen J. Vanderwolf, Juliana Vélez, Diana Carolina Vergara-Florez, Brian C. Verrelli, Marcus Vinícius Vieira, Nora Villamil, Valerio Vitali, Julien Vollering, Jeffrey Walker, Xanthe J. Walker, Jonathan A. Walter, Pawel Waryszak, Ryan J. Weaver, Ronja E. M. Wedegärtner, Daniel L. Weller, Shannon Whelan, Rachel Louise White, David William Wolfson, Andrew Wood, Scott W. Yanco, Jian D. L. Yen, Casey Youngflesh, Giacomo Zilio, Cédric Zimmer, Gregory Mark Zimmerman, Rachel A. Zitomer
Institutions: The University of Melbourne, Whitman College, UNSW Sydney, Macquarie University, Monash University, County Administrative Board, Lund University, University of Massachusetts Amherst, Institute of Agrifood Research and Technology, Ben-Gurion University of the Negev, University of Edinburgh, Indian Institute of Science Bangalore, US Forest Service, Southern Research Station, Aarhus University, Indiana University – Purdue University Indianapolis, Deakin University, King Abdulaziz University, University of Hohenheim, Swedish University of Agricultural Sciences, University of Connecticut, Southern Illinois University Edwardsville, University of Guelph, University of Zurich, California State Polytechnic University, Centre National de la Recherche Scientifique, Centre d'Etudes Biologiques de Chizé, La Rochelle Université, Bar-Ilan University, University of Tasmania, Rhodes College, Royal Society for the Protection of Birds, Wofford College, Linköping University, Czech University of Life Sciences Prague, University of Gothenburg, University of Tennessee at Knoxville, Universidade Federal de Santa Catarina, University of South Wales, University of Lisbon, Colorado State University, Drexel University, Salisbury University, University of Groningen, McMaster University, University of Cambridge, University of Florida, Elon University, BirdLife international, The University of Texas Rio Grande Valley, University of Toronto, University of Auckland, University of Glasgow, Northern Arizona University, Max Planck Institute for Biological Intelligence, Bath Spa University, University of Alberta, Technische Universität Dresden, The University of Sydney, Jagiellonian University, Brigham Young University, Université de Montpellier, Institut des Sciences de l'Evolution de Montpellier, Institut de Recherche pour le Développement, University of South Carolina, Imperial College London, European Forest Institute, Clemson University, University of Birmingham, University College Dublin, San Jose State University, St. Norbert College, Purdue University West Lafayette, University of Göttingen, University of Minnesota, CAB International, University of California, Irvine, University of Aberdeen, McGill University, Université Paris-Est Créteil, Oregon State University, Wake Forest University, Institute of Ecology and Biodiversity, Universidad Rey Juan Carlos, Tufts University, Université de Rennes, Bielefeld University, University of Maine, Boise State University, Agricultural Research Service, Baruch College, City University of New York, Binghamton University, University of Rome Tor Vergata, University of Michigan, University of Antwerp, Wesleyan University, Yale University, St. Olaf College, Ordu University, KU Leuven, Wilfrid Laurier University, Iowa State University, The University of Western Australia, Middle East Technical University, The University of Texas at Austin, Universidad de Los Andes, University of Turku, Austral University of Chile, University of Wyoming, University of Bristol, International Institute for Applied Systems Analysis, École Pratique des Hautes Études, University of California, Davis, Université du Québec à Montréal, University of Wisconsin–Madison, Florida International University, Aberystwyth University, Université de Montréal, Universität Ulm, Memorial University of Newfoundland, The Nature Conservancy, University of Arizona, University of Stirling, University of Konstanz, Universidade Federal da Fronteira Sul, University of Hull, Université de Pau et des Pays de l'Adour, Ecologie Comportementale et Biologie des Populations de Poissons, Liverpool John Moores University, University of Bayreuth, Umeå University, Water Research Institute, National Research Council, Norwegian University of Science and Technology, Rutgers, The State University of New Jersey, Université du Québec en Abitibi-Témiscamingue, Université du Québec à Trois-Rivières, University of Bern, Eastern Illinois University, Inserm, Centre Hospitalier Régional et Universitaire de Nancy, Centre d'Investigation Clinique Pierre Drouin, Université de Lorraine, Consejo Superior de Investigaciones Científicas, Centre d'Investigacions sobre Desertificació, University of Arkansas at Fayetteville, The University of Queensland, University of Helsinki, University of Iceland, Karolinska Institutet, University of Würzburg, Hiram College, Montana State University, University of California, Riverside, Universidad Nacional de La Plata, Bangor University, MRC Biostatistics Unit, Harvard University, Universidad Complutense de Madrid, Technische Universität Darmstadt, Centre de Coopération Internationale en Recherche Agronomique pour le Développement, Animal, Santé, Territoires, Risques et Ecosystèmes, The Ohio State University, University of Bath, University of Southern Mississippi, University of California, Berkeley, University of Antananarivo, University of California San Diego, Vrije Universiteit Amsterdam, University of Vienna, University of Veterinary Medicine Vienna, University of Hong Kong, University of Neuchâtel, Virginia Commonwealth University, Missouri State University, University of Jyväskylä, Austrian Academy of Sciences, Institute for Interdisciplinary Mountain Research, Agricultural Research Organization, Office for National Statistics, Institute of Avian Research, Universidade Federal de Goiás, University of Exeter, New South Wales Department of Primary Industries, Leibniz Centre for Agricultural Landscape Research, University of Wisconsin–La Crosse, Duke University, UCLouvain, Federation University, United States Department of Agriculture, Arthur Rylah Institute for Environmental Research, Agence Nationale de Sécurité Sanitaire de l’Alimentation, de l’Environnement et du Travail, University of California, Los Angeles, Universidad de la República de Uruguay, Universidade Federal do Rio de Janeiro, British Trust for Ornithology, Rochester Institute of Technology, Utah State University, Rhodes University, University of St Andrews, Cornell University, Helmholtz Centre for Environmental Research, Trent University, Universidad de Valladolid, University of Bergen, University of Rhode Island, University of Plymouth, North Carolina State University, Helmholtz Munich, University of Cape Town, FitzPatrick Institute of African Ornithology, University College London, University of Saskatchewan, University of Regina, University of Waterloo, Public Health Scotland, University of Münster, Western Norway University of Applied Sciences, University of Southern Maine, University of Southern Queensland, Virginia Tech, University of Brighton, University of Oxford, University of Colorado Denver, Michigan State University, Université Sorbonne Paris Nord, Laboratoire d’Ethologie Expérimentale et Comparée, Lake Superior State University
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
SDG: Life on land
Open Access status: gold
Estimated APC paid: $3,690
Funders: Virginia Polytechnic Institute and State University, Michigan State University, Iowa State University, Australian Government, University of Oxford, University of Brighton, University of Southern Queensland, Centre National de la Recherche Scientifique, Université de Montpellier, College of Engineering, Michigan State University
OpenAlex ID: W4407194079
