Rejoinder to discussion on Is group testing ready for prime‐time in disease identification?
Year: 2021
Type: article
Abstract: We want to thank the authors of the three discussions to our article for their thoughtful comments, suggestions, and critiques. We enjoyed many of the intellectually stimulating points and believe that these comments, along with our response, will push the area of group testing for disease identification forward. One major point of our article was to emphasize that group testing designs require reliable estimates of test-misclassification to ensure that a screening program will not perform poorly in practice. Specifically, such estimates should be specific to the population where the screening will be conducted. It is well recognized in diagnostic medicine that the sensitivity and specificity of a test can be highly dependent on the population being considered (spectrum bias). In the article, we considered a validation procedure which randomly forms groups from a random sample in the population of interest to closely mimic the final pooling procedure. This approach minimizes the necessary assumptions and prior knowledge required for undertaking a validation study. In our experience, researchers are very skeptical of making any a priori assumptions about how testing will perform in a population. Still, in the Discussion section we acknowledged that in some cases, due to prior knowledge or necessity, other approaches may be considered. For example, for COVID-19 screening in many countries, there is no alternative to group testing and very large validation studies may not be possible. In light of this, we were happy to see Biggerstaff (B) and Bilder, Tebbs, and McMahan (BTM) address and expand upon the popular spiked designs as an alternative to our validation procedure. Such studies are inexpensive and easy to implement, particularly since they can utilize precollected samples. However, spiked designs are not assumption free and, depending on how the samples are ascertained, are sensitive to spectrum bias. Furthermore, obtaining population-based estimates of test performance from spiked experiments would require outside knowledge of the distributions of disease prevalence and viral load among cases, information that is unlikely to be obtainable. We think it is important for researchers planning to use group testing to be aware of these, potentially, serious biases. B's recommendation to develop alternative hybrid designs that combine different types of validation samples may be a compromise between validity and feasibility. We agree with B's comment that many of the lessons learned on group testing for identification could be applied to group testing for estimation. His discussion about using group testing for estimation in the context of virus estimation in mosquito populations was interesting and informative. We appreciate the discussion of laboratory validation by BTM. However, we do not think that this explanation detracts from our concern about how samples are selected. Convenience laboratory-based samples may not be representative of the actual pooling that occurs in the population where screening occurs. We find it hard to believe that any laboratory can "ensure there is little if any loss of accuracy when compared to individual testing." That may be true for certain assays, but clearly cannot be a general statement. It has to depend on the particular assay under consideration. We respectively disagree with BTM's argument that the expected number of tests in itself is a reasonable objective function for optimal design when tests are subject to misclassification. For example, when tests are subject to differential misclassification such as a function of group size (ie, dilution function), there is an implicit trade-off between choosing a design with a larger group size and the cost of misclassifying individuals. This trade-off needs to be considered, possibly with cost considerations as mentioned by B. BTM rely heavily on the Hitt et al paper to bolster their argument that the expected number of tests is sufficient as an objective function. However, that paper does not consider that misclassification may depend on the group size and is therefore irrelevant to the considerations in our current paper. Furthermore, our numerical calculations do show in the differential error setting that failure to constrain the expected number of tests can lead to very different design choices. For constructing an optimal design, BTM suggest first minimizing the expected number of tests and then calculating the resulting misclassification, adjusting the group size if the results are not satisfactory. This can certainly be done, but it seems more sensible and transparent to include this revaluation of the group size as a part of the objective function from the start. We do this by putting a constraint directly on the objective function being minimized. Furthermore, since the expected number of tests depends on assumed misclassification parameters, it is essential to first get accurate estimates of the misclassification parameters as recommended in our paper. Otherwise, it is impossible to guarantee that the optimal design is constructed based on the correct objective function. BTM raises concerns with the dilution function used and mentioned that "This is especially concerning because nucleic acid amplification tests were not developed when Hwang was published in 1976." First, the complaint about our use of the Hwang function is misleading since we recommend a nonparametric approach for estimating the dilution function (see algorithm 1) and do not rely on a particular model. Second, we use the Hwang dilution function (in figure 1) to simply show how the design characteristics are sensitive to the correct formulation of the dilution function. In fact, we use the nonparametric dilution function because we do not want to rely on any dilution function (Hwang or any others). Given how we use the Hwang function, we do not understand the relevance of the timing of its development relative to the development of NAAT testing. BTM mentioned that the parameters of the dilution function are not realistic given that no one would use group testing if the sensitivities drop below 0.9 with increasing group size. But, how would you know this unless you estimate the sensitivity relative to group size? Here is where the suggestion of B to develop a hybrid approach between population-based sampling and laboratory-contrived samples might be a nice alternative. A small laboratory-based study may rule out the use of group testing when it is clearly problematic due to dilution. However, if the laboratory-based verification is good, we still recommend the use of population-based validation before widespread screening with a group testing procedure. Johnson (J) raises the very important issue of subjectivity in the context of estimating pooling misclassification. We took a rather strict perspective in that we do not incorporate any prior information about dilution error and argued that one needs to confine estimation to the population under consideration. As J nicely summarized, we took a completely nonparametric model to avoid any assumptions. We showed that from this perspective, group testing may require very large validation samples in certain cases. We did not intend to argue that group testing should never be done in these cases, but that we need to understand the potential weaknesses inherent in making assumptions for tractability. (In the discussion, we mention this in the context of COVID-19 testing and the fact that there may be no alternative other than group testing for disease screening.) This could include Bayesian estimation in situations where one could elicit objective priors from validation samples measured in similar but different contexts. Following along with the recommendation of hybrid designs by B and the emphasis on laboratory-based screening by BTM, one interesting approach would be to develop priors from laboratory-based verification and then use them in population-based verification. Such an approach would allow for smaller population-based validation samples than those presented in our article. J's contrast in subjectivity with our approach raises a very good point. In some cases, using expert prior knowledge or subject-related knowledge might make sense. To us, it depends on what is the use of the final procedure. In situations such as large-scale screening we believe that physicians may be skeptical of incorporating expert knowledge if it relies on modeling assumptions. However, in other setting such as estimating disease prevalence (as mentioned by B), we might feel more comfortable with this approach. J raises the interesting and somewhat controversial area of diagnostic testing without a gold standard. He mentions that Bayesian estimation can get around identifiability by incorporating prior information on relevant model parameters. These methods are very novel and valuable, but we would emphasize that there is no free lunch here. Bayesian inference in this setting will require informative priors and be sensitive to their correct specification (ie, we better believe the experts). Others1 have proposed approaches where collecting a small amount of gold-standard information improves statistical inferences without the need to incorporate prior information. We truly thank each of the discussants for their thoughtful commentaries. We all would agree that group testing is an important technique that shows strong promise as a tool for disease screening into the future. At the same time, it is important to weigh the strengths and weakness of using this approach in particular applications. We hope that our article, the discussions, and rebuttal, will play a role in moving this area further.
Source: Statistics in Medicine
DOI: 10.1002/sim.9033
Language: English
Subfield: Infectious Diseases
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W3179877266