Adjusted Positive Agreement: A Chance-Corrected Agreement Measure for Unconstrained Pile Sorts
Year: 2026
Type: article
Abstract: Unconstrained pile sorting, also known as free pile sorting or card sorting, is an important method for studying how people organize items within a domain. Yet commonly used measures for estimating respondent agreement struggle with the lumper–splitter problem. As a result, apparent similarity between respondents may reflect propensities to lump or split rather than shared conceptual agreement. This article introduces the adjusted positive agreement (APA), a pair-counting coefficient designed to address this problem. APA compares the item pairs that two respondents place together, adjusts this overlap for the total number of co-placed pairs produced by each respondent, and scales the result relative to the smaller of the two respondents’ co-placement totals. Across two simulation conditions, APA preserved agreement with less influence from lumping and splitting than Pearson correlation and the adjusted Rand index. The findings support APA as a basis for analyzing how cultural models are distributed across respondents.
Source: Field Methods
Author: Jonas Idevall Hagren
Institutions: Uppsala University, Karolinska Institutet
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $3,650
OpenAlex ID: W7222829862