Year: 2016
Type: article
Abstract: where E and Var denote expectation and variance over x, and by convention, we use squared Euclidean distance. These entities are related as follows: Inaccuracy = Imprecision + Bias2 An unbiased estimator may still be imprecise and in expectation far from the true value, and a precise estimator may still be biased and in expectation far from the true value. Morrissey (2016) shows that current practice of ‘informal’ meta-analysis often results in biased estimates because imprecision in the original statistics is converted into bias in the meta-analysis due to nonlinear transformations (e.g. deleting sign to get magnitude). He then shows how such bias can be corrected by use of mixed-model statistics taking account of the precision of individual studies, and he shows by examples that such corrections can have large effects and even reverse the conclusions of a study (see also Morrissey & Hadfield, 2012). While supporting Morrissey's mixed-model approach and formal meta-analysis more generally, not just to correct bias, but even more for their ability to weigh studies according to precision, I would like to comment on a crucial premise: the need to quantify the accuracy of the individual studies. Even when available, the standard errors needed for this purpose are themselves inaccurate as estimators of precision, and even more so as estimators of accuracy. While publication bias is a well-known problem (Palmer, 2000), there are many other sources of inaccuracy in individual studies relative to the goal of a meta-analysis. Selection gradients vary in what organisms, traits, environments and fitness components they pertain too, and in what methods and controls are used. Of course, one task of meta-analysis is to describe the effects of such differences, but heterogenous collections of estimates bring into question the meaning of meta-analytical statistics influenced by the relative frequency of different types of studies. As an example, in informal meta-analyses of evolvabilities, Houle (1992) and Hansen et al. (2011) found median evolvabilities of 0.25% and 0.26%, but Garcia-Gonzalez et al. (2012) got 1.28% (Hansen, 2016). This dramatic difference is largely due to a lower proportion of linear morphological traits with low evolvability in the latter study. Clearly, an average evolvability over all traits is not particularly meaningful, but the question remains as to how finely we need to divide the data before we can claim a meaningful average. A key operation in meta-analysis is to bring estimates from different studies onto a common scale, and much discussion has gone into the development of general measures of effect size, etc. (e.g. Nakagawa et al., 2015). This is not trivial, because parameters have biological meaning deriving from theoretical context that may be obscured by transformations and adjustments of scale (Houle et al., 2011). For example, Hereford et al. (2004) argued that standardizing selection gradients with trait standard deviations obscures the strength of the fitness–trait relation. Morrissey (2016) criticizes Murren et al. (2014) for not taking account of differences in precision when comparing intercepts and slopes of reaction norms, but we may also ask whether slopes and intercepts are on comparable scales in the first place. Is a unit change in slope comparable to a unit change in intercept? Meta-analyses are vulnerable to outright errors and incomplete reporting in the original studies. Many systemic errors of reporting evolutionary parameters have been revealed. Stinchcombe et al. (2008) estimated that 78% of all reported quadratic selection gradients are off with a factor of two due to reporting regression coefficients in place of second derivatives. Garcia-Gonzalez et al. (2012) revealed a number of errors in reporting measures of additive genetic variation, including a common confusion of additive variance with sire variance, which renders the estimates off with a factor of four. Voje et al. (2014) documented how many estimates of allometric exponents are in fact slopes of linear regression conducted on arithmetic and not logarithmic scales. Allometric regressions are also often based on reduced major axis or other curve-fitting techniques that simply do not estimate the correct parameter and can lead to huge inaccuracies (Hansen & Bartoszek, 2012; Voje et al., 2014). Standard errors also have sampling errors. Even leaving aside biases due to sampling from structured populations, imprecise measures of precision should be a concern for many meta-analyses. This problem is particularly acute in comparative studies, which is a form of meta-analysis across species (e.g. Lajeunesse, 2009; Hadfield & Nakagawa, 2010). Here, trait means from different species are often based on small (< 10) and uneven sample sizes. The standard error of a mean is calculated as follows: With a sample size of 10, the relative error is 47%, leaving the standard error too imprecise to be of much use, and even with n = 50, the relative error will be a non-negligible 20%. This raises the question of whether it is advisable to use published standard errors from small to moderately sampled studies to weigh and correct bias. The weighting of studies according to estimated precision is particularly problematic because the most unreliable estimates are also those with the least-precise standard errors, and some of them will be treated as reliable simply due to error in their standard errors. Methods for weighting and bias correction with phylogenetically correlated data exist (e.g. Hadfield & Nakagawa, 2010; Hansen & Bartoszek, 2012; Garamszegi, 2014), but many available standard errors are too imprecise for use. One solution is to assume that all estimates have the same sampling variance and then obtain a precise estimate of this across studies. The precision of individual estimates can then be given as s2/ni, where s2 is the global variance and ni is the sample sizes of the individual estimates. Grabowski et al. (2016) developed an improvement on this using a weighted average of a species’ own sampling variance and a sampling variance estimated from other species. Meta-analyses of selection gradients are less vulnerable to this problem, because sample sizes are usually larger. The median sample size in the Kingsolver et al. (2001) data set was 134 with a range from 10 to many thousands. The squared standard error of a simple regression slope is , where is residual variance and is predictor variance. Assuming no error in the predictor variance, the relative error is approximately . An n = 134 study would then have a relative error in SE2 of about 12%, which may be acceptable, but the weighting of smaller studies is unreliable, and it may be advisable to estimate their precision by other means. The standard errors needed for formal meta-analysis and bias correction are often not reported. In the Hereford et al. (2004) database, only 20% of univariate selection gradients came with standard errors, and in Thurman & Barrett's (2016) recent meta-analysis of selection coefficients on genetic variants, 97% of estimates were given without standard error. These numbers are even from vetted minorities of studies usable for informal meta-analysis. In all meta-analyses with which I have been involved, we have not been able to use a majority of studies due to inconsistencies, transformations and lack of reporting basic statistics. As so often in evolution and ecology, haphazard standards of reporting and the lack of attention to the meaning of estimates greatly impair our ability to generalize on the basis of past work. I thank the editors for the invitation, and Michael Morrissey for illuminating discussions.
Source: Journal of Evolutionary Biology
DOI: 10.1111/jeb.12947
Author: T. F. Hansen
Institution: University of Oslo
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: bronze
OpenAlex ID: W2473983204