A Prospectively Frozen Zero-Human LLM Measurement Design, and the Confirmatory Checkpoint That Rejected Its Instrument
Year: 2026
Type: article
Abstract: This is a methods report of a rejected instrument: a prospectively frozen, zero-human, multi-family LLM content-analysis design whose fresh-data reliability checkpoint failed on all four coding dimensions at the start of the confirmatory run, halting the study before any hypothesis was tested. The case was to be a corpus-scale audit of natural theology's inferential chain — from a bare first cause to the God of Nicene Christianity — long discussed piecewise (the "gap problem," "ramified natural theology," "dwindling probabilities") but, so far as a structured prior-art sweep could determine, never measured as a whole. We introduce a witness-based criterion individuating the chain's dialectically separable steps, yielding a precommitted eight-step working partition whose factor count, for a fixed partition, is invariant under notational bundling; reading those factors as separately payable evidential burdens is defended, not derived. We designed, and prospectively froze under a hash-sealed local protocol, an audit of twenty-one issue-years (2004-2024) of the peer-reviewed philosophy-of-religion literature: 3,536 articles across eleven journals in three venue tiers, each article's step, direction, claim strength, and epistemic type of evidence to be coded by a fixed multi-family LLM instrument with zero human coding of corpus or checkpoint items, gated by a precommitted contamination-control battery. Two hypotheses were fixed in the sealed protocol: (1) among positive evidential claims, pro-side items are predominantly a priori and contra-side items predominantly empirical or historical (precommitted odds ratio >= 3); (2) positive evidential pro-side claims are more prevalent in confessionally affiliated venues than in mainstream ones. A precommitted kill criterion binds the symmetric case: a two-null result downgrades the project to a descriptive preprint with no journal claim. Neither hypothesis was tested. The precommitted checkpoint — 150 fresh, seed-drawn items, coded by three model families with three independent runs each — failed its dual statistic on all four dimensions: the point criterion on two (minimum pairwise Krippendorff's alpha .676-.736 against the .70 floor) and the bootstrap lower-bound criterion on all four (95% lower bounds .390-.650 against .667), with run-level determinism audits clean and the family-correlation watch untriggered. Under the frozen decision tree this halted all coding: the pilot reliability that licensed the instrument — flagged in the sealed plan itself as a selected estimate under optional stopping — did not survive contact with fresh data. The paper reports the design, the validation battery, the locked corpus infrastructure, and the complete precommitted trajectory ending in instrument rejection, as a methods contribution and a cautionary result for LLM-based content analysis: a precommitted fresh-data reliability checkpoint is cheap relative to a full run, and it does exactly what it is designed to do.
Author: Kang, Jaehoon
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7170076859
FWCI:
Cites:
Cited by:
Related to: