Year: 2024
Type: conference-paper
Abstract: Topic modeling is a text analysis technique for automatically discovering common themes in a collection of documents. “Human-in-the-loop” topic modeling (HLTM) allows domain experts to steer and adjust the creation of topic models. In this case study, we use a custom-built HLTM interface to assess the impact of human refinement on model interpretability and predictive performance in collaboration with an analytics team within our organization. Using a small dataset (≈ 12k documents) of responses drawn from an organizational employee satisfaction survey, we compare the pre- and post-refinement models using both human judgments and automated metrics. We find that human refinement can enhance interpretability and predictive performance in some cases, but may lead to overfitting on the training data, which negatively impacts model quality. Furthermore, we observe that existing evaluation methods don’t sufficiently and clearly capture topic model quality in applied settings, and propose guidance for further HLTM tool development.
Authors: Sally Gao, Milda Norkute, Abhinav Agrawal
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: bronze
OpenAlex ID: W4396833069