Work

PC's Test Log Can Overstate Type-I Error: A Selection Pitfall in Evaluating Conditional-Independence Tests

Year: 2026
Type: preprint
Abstract: Constraint-based causal discovery runs many conditional-independence (CI) tests. With a known true graph, it is tempting to judge a test's calibration from the tests that an algorithm such as PC actually ran. Pooled over tests with non-empty conditioning set S, this estimate is inflated in our runs. PC tests a pair given a larger S only if every earlier test of that pair rejected, so a truly independent pair that reaches a deeper test has often been falsely rejected already, on the same sample. With causal-learn's PC and Fisher's z test on linear-Gaussian data, logged null tests reject at 4.6–5.3% at |S| = 0 but at 14.5–28.5% at |S| = 1 (α = 0.05). The pattern holds for α ∈ {0.01, 0.10}, in denser graphs (where the excess is smaller), in FCI, and in our own PC with kernel conditional-independence (KCI) and generalised covariance measure (GCM) tests. Null pairs without an earlier false rejection stay near α. For Gaussian data we derive the asymptotic correlation of Fisher-z statistics across nested conditioning sets, which predicts the repeat-rejection rate in three-variable simulations; in causal-learn's PC runs it overpredicts by 6–9 percentage points in every bin of c. A fixed audit of independence queries avoids the bias. As a secondary question we ask when regression-induced miscalibration of single tests damages PC graphs. On the audit, single-test type-I error is rank-correlated with false adjacencies across six regressors (Spearman ρ = 0.66 for KCI and 0.73 for GCM), but this link comes from two clearly misspecified regressors; among four flexible regressors it vanishes (ρ = 0.04 and 0.17). In single-test experiments, cross-fitting did not remove type-I inflation.
Language: English
Open Access status: green
OpenAlex ID: W7236643527
Cites:
Cited by:
Related to: 10