Arguing Practical Significance in Software Engineering Using Bayesian Data Analysis.
Year: 2018
Type: preprint
Abstract: This paper provides a case for using Bayesian data analysis (BDA) to makemore informed claims regarding practical significance of software engineering
research. We show that using BDA, here combined with cumulative prospect theory
(CPT), is appropriate when a researcher or practitioner wants to make more
explicit connections between statistical findings and practical significance in
empirical software engineering research. To illustrate our point, we provide an
example case using previously published data. We build a multilevel Bayesian
model for this data, for which we compare the out of sample predictive power.
Finally, we use our model to make out of sample predictions while, ultimately,
connecting this to practical significance using CPT. Throughout the case that
we present, we argue that a Bayesian approach is a natural, theoretically
well-grounded, practical work-flow for data analysis in empirical software
engineering. By including prior beliefs, assuming parameters are drawn from a
probability distribution, assuming the true value is a random variable for
uncertainty intervals, using counter-factual plots for sanity checks,
conducting predictive posterior checks, and out of sample predictions, we will
better understand the phenomenon being studied, while at the same time avoid
the obsession with $p$-values.
Source: arXiv (Cornell University)
Authors: Richard Torkar, Robert Feldt, Carlo A. Furia
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2892433920