Work

Decomposing Video Activity Labeling into Human-Inspectable Steps: A Gray-Box Workflow via Narration-Derived Intermediate Outputs

Year: 2026
Type: article
Abstract: Video activity labeling divides a video into activity segments and assigns a category to each. Manual annotation requires sequential review, while end-to-end AI models may require retraining when the coding task or taxonomy changes. We propose Narration-Mediated Activity Annotation (NMAA), a gray-box workflow that organizes transcript sentences into activity units and compares the resulting annotations with a human reference to evaluate where one activity ends and another begins (B), how activities are classified (T), and which transcript words support each classification (E). Researchers revise the coding protocol based on these results and observed disagreements, reevaluate it, and retain a revision only when its stated purpose is achieved without a confirmed loss in any of the three measures. We evaluated NMAA on 35 Roblox Restaurant Tycoon videos (13.8 h, 16,707 sentences) using a 1200-sentence human reference. The largest stepwise change came from assigning categories sentence by sentence and merging adjacent same-category sentences afterwards, which produced the largest increase in evidence agreement and brought mean activity-unit length close to the human reference. On the held-out test set, the final protocol improved B, T, and E by 0.100, 0.163, and 0.202, respectively, relative to the baseline. The same revision reduced activity-unit length for all three large language models from different providers. When the full corpus was analyzed with the final protocol, Facility and Interior Management accounted for a larger share of the creators’ narrated management time than of textbook topic coverage, whereas Marketing and Promotion accounted for a smaller share.
Institution: University of Suwon
Language: English
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W7235199161
FWCI:
Cites: 33
Cited by:
Related to: 10