Get started
Work

Van theorie naar observaties en terug: Inzichten in preregistratie, het delen van gegevens, en test severity

Year: 2024
Type: dissertation
Abstract: In the past decade, the replication crisis has cast doubt on many findings in psychology and other empirical sciences. As a response to this crisis, the field of meta-science (i.e., the scientific study of science itself) has grown. Within the meta-scientific discipline, various tools and methods have been suggested to tackle the issues that lead to the replication crisis. One prominent proposal is open science, which is an emphasis on transparent research workflow which allows the (scientific) community to scrutinize, reproduce, replicate, or build on published findings. These open science practices seem like an obvious step in the right direction. However, as with any new tool and method, they must be evaluated and, if necessary, improved. The first chapter of this dissertation focuses on preregistration. Preregistration involves freezing decisions regarding a study (e.g., study design, data collection, and data analysis) on a public third-party repository prior to seeing, or ideally prior to collecting, the data. Ideally, researchers stick to the preregistration plan but they can also deviate from it, as long they disclose it. Undisclosed deviations from preregistered plans undermine transparency because they make an evaluation the results more difficult. In Chapter 1, we examine the first 23 papers published with a Preregistered badge in Psychological Science, and compare them to their respective preregistration plans. Two out of 27 preregistered studies contained no deviations from the preregistration plan. In one study, all deviations were disclosed. Nine studies disclosed none of the deviations. Based on our findings, we offer recommendations to improve preregistration practices. The second and the third chapter focus on another open science practice - data sharing. With open data, the reproducibility of the results can be verified, and it facilitates building upon previous work. However, there are barriers to sharing research data, such as concerns that errors will be exposed. In Chapter 2, we replicate a study by Wicherts et al. (2011), who found a relation between reluctance to share data and reporting inconsistencies in test statistics, degrees of freedom, and p-values. Alongside repeating the original analysis on the original data and replication data, we conducted a multiverse analysis to assess the robustness of the results. Contrary to the study by Wicherts et al. (2011), we found no evidence supporting a strong relation between data sharing and consistency errors. When it comes to open data, participants are stakeholders as well. They may be less willing to participate if their data are openly shared. In Chapter 3, we investigated the impact of data sharing policies on participants' willingness to take part in a survey with sensitive questions by manipulating the data sharing policy. Our findings suggest that an open data policy might not significantly impact dropout, item skipping, data quality, or the diversity of the sample, however, a substantial number of participants would prefer their anonymized data to remain private. Finally, in Chapter 4, we discuss three other crises in psychology that call attention to the connection between statistical models and scientific theories. According to the falsificationist tradition in philosophy of science, theory corroboration crucially depends on the passing of severe tests (i.e., tests that are likely to be failed if the theory does not hold). Therefore, we investigate how proposed solutions to the theory, measurement and generalizability crisis are connected to test severity.
Language: English
Open Access status: green
OpenAlex ID: W7015839009
Cites:
Cited by:
Related to: