Work

Factual2: Extending a Romanian Fact-Checking Corpus with Speaker Credibility Histories for Leak-Aware Veracity Classification

Year: 2026
Type: article
Abstract: Credibility history, the per-speaker count of past verdicts, is an important metadata feature in fact-checking benchmarks. This feature can be misused: including credibility history for all examples in the corpus embeds the truthfulness label in the feature vector. We present Factual2, a Romanian fact-checking corpus of 1958 political statements by 417 speakers from factual.ro, annotated with five veracity classes. To our knowledge, Factual2 is the first Romanian corpus to deal with credibility history in a controlled leakage setting. The count vectors are created using Leak-Free Aggregation (LFA), with self-exclusion, cold-start policy, and a stricter causal variant (LFA-temporal). Two leaky approaches are presented for diagnostic purposes; four label schemes and three segmentation protocols are available. The key results are: (i) Pre-Split Aggregation (PSA) overestimates the classification performance of LFA counts by 0.20 macro-F1 in the 5-class setting (0.484 vs. 0.285). (ii) In the hard binary setup, the leaky classifier achieves a macro-F1 of 0.798 in the speaker-disjoint setting, while the honest classifier achieves 0.393, which implies that the gap is due to leakage. (iii) Using the temporal protocol, the honest counts achieve 0.784 macro-F1, equal to the speaker-identity baseline; the maximum gain of the honest approach across the grid is 0.051.
Source: Electronics
Language: English
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W7225906728
FWCI:
Cites: 28
Cited by:
Related to: 10