Strategic Reasoning Measurement
Year: 2026
Type: preprint
Abstract: PRE-COMMITTED ANALYSIS PLAN — STRATEGIC THEATRE CORPUS Dated 2026-06-12, written before any confirmatory cross-rung or human-vs-AI comparison was conducted. The corpus (300 AI-generated strategy analyses across 75 cells of 4 elicitation rungs, plus 25 matched human reports) was finalised before this plan was registered, but no confirmatory analysis was run before registration. This is a characterising-study pre-registration: it fixes the measures, the aggregation, the reporting discipline, and the order of analysis. It deliberately does not pre-set numerical thresholds or predict directions, because the study characterises the architecture of strategic reasoning across conditions rather than testing a point hypothesis against a cutoff. The protection it provides is against selective reporting and post-hoc analysis choice, not against a predicted effect failing to reach a threshold. PURPOSE The study compares the structural signatures of human- and AI-generated strategy documents. AI documents were generated across a four-level instruction-intensity elicitation ladder (TMBA, E1, E2, E3), where instruction binding-ness increases while content held constant, and matched to human-authored strategy documents on the same companies. The aim is to characterise how the structural signature of reasoning behaves across increasing elicitation, and how AI signatures compare to human ones. CONFIRMATORY MEASURES (the complete set) No measure outside this set is computed for the confirmatory analysis: Strict-Ext rate — claim-level attributable citation. Strict/lenient citation delta — the detectability gap between apparent and actual grounding. Node-type distribution (F / M / V / P). Edge-type distribution (S / W / J / E). These are computed across the four rungs and for human-vs-AI. WHY THESE FOUR (RELIABILITY-COMPLETE) The confirmatory set is the complete set of measures that are both reliable (cleared or characterised against the pre-committed inter-coder reliability bar) and coded and available now. Other measures are excluded from confirmatory status for principled reasons, not convenience: inferential structure (edge combinations, reasoning validity/depth) is reserved because the edge dimension had the weakest reliability; architectural connectivity and motifs are reserved because connectivity-coding reliability is not yet established; source validation is reserved because it requires an additional coding pass; and the qualifier dimension is reserved because its reliability fell below the pre-committed bar. The confirmatory four are thus the measures the instrument can confirmatorily support today; everything richer is exploratory or future work, by reliability and readiness, not by post-hoc choice. REPORTING DISCIPLINE All four measures are reported for all comparisons, whether or not each shows a pattern. No measure is dropped for being flat. Flat results are part of the finding. AGGREGATION The three seeds are averaged within a cell (entity x rung); the entity is the replication unit. Seed-level spread is shown alongside every cell and rung mean, so within-rung variation is always visible next to between-rung variation. No threshold for "real difference" is pre-set; the comparison is displayed (between-rung spread against within-rung spread) and characterised descriptively in light of the measured noise structure. ORDER OF ANALYSIS The four measures are computed and recorded first, as the confirmatory pass, before any exploratory pattern-hunting. Exploratory analyses are run only after the confirmatory pass is recorded, and are labelled exploratory, never presented as predicted. RESEARCH-QUESTION PROCEDURES (commitments to run and report honestly, not predicted results) RQ1 (consistency across the ladder): for each measure, report rung means with seed spread; describe whether between-rung variation exceeds within-rung, measure by measure; report all four whatever they show. RQ2 (detectability): report the strict/lenient delta per rung and per dimension; rank; describe the pattern; no pre-set cutoff. RQ3 (the coincidence): identify which signatures vary least across the ladder, and which have the largest strict/lenient gap, then report whether they are the same signatures — reporting the overlap whatever it is, including no relationship. HUMAN-VS-AI The same four measures are applied identically to human and AI documents (the symmetry principle: any measure of reasoning weakness applies to both). Differences are described, not tested against a cutoff. PRE-IDENTIFIED WIDE-VARIANCE RUNGS Two rungs are pre-identified, blind to results, as carrying wider variance and therefore wider error bars: E2 (coding-dense variance) and TMBA (generative-latitude variance, from its minimal prompt producing the widest output-length spread). If either proves pivotal to a finding and its error bars are too wide to resolve, seed augmentation for that rung is a pre-identified rather than post-hoc move. EXCLUSIONS Documents are excluded only as content-less stubs (under 500 words with no analysis) or as tagged duplicates / superseded generations. Short-but-complete documents are included. Any cell dropped for unrecoverable generation failure is documented by ID with reason. No exclusion is made on the basis of results. Final canonical corpus: 300 AI analyses and 25 human reports; audit-only artefacts are excluded with logged reasons. EXPECTED OUTCOMES AND EXPLORATORY EXTENSIONS The study expects to characterise whether and how the four structural signatures shift across increasing elicitation, how detectable any grounding gaps are, whether prompt-resistant signatures coincide with low-detectability ones, and how AI signatures compare to human ones — reporting all outcomes whether or not they show clear patterns. Anticipated exploratory extensions, named in advance so the exploratory pass is pre-anticipated rather than exploratory fishing, include: source validation (conditional on a confirmatory grounding finding — an added coding pass checking whether attributed sources are real and support their claims); architectural connectivity and motifs; inferential reasoning structure (kept structural, not evaluative); internal-reference patterns; qualifier behaviour across rungs; output-volume effects; and entity-level interactions. These are run only after the confirmatory pass and labelled exploratory. A note on length versus rate: the headline measures are rates and proportions, which are length-normalised, so output-length variance (notably in TMBA) is expected to be largely orthogonal to the rate-based signal.
Source: OSF Preprints (OSF Preprints)
Author: Adrian Tate
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7164776971