Year: 2026
Type: conference-paper
Abstract: Embedding-based topic models such as BERTopic routinely produce hundreds of finegrained, overlapping topics that must be reduced before downstream use.Existing approaches couple reduction with outlier rejection, rely on local geometric linkage, or use costly and stochastic LLM prompting.We introduce Vendi Clustering (VC), a topic reduction method that reframes the problem as global diversity-preserving compression: at each step, VC selects the merge that induces the smallest loss in the Vendi Score, a spectral measure of diversity.A naive implementation of this global criterion is prohibitive at O(n 6 ); we derive algebraic update rules at order q = 2 that evaluate each candidate merge in O(1) via cached row-square sums and a lookahead Gram matrix, reducing total complexity to a tractable O(n 3 ).On 20 Newsgroups and AG News, VC outperforms standard reduction baselines on coherence and diversity, surpasses LLM-prompted reduction on the majority of metrics against both an open-weight model and a frontier API model, with topic diversity gains of up to 35% while running over 40× faster.
Authors: Sean Kim, Adji Bousso Dieng
Institutions: Brown University, Princeton University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7224426150