Work

The Trouble with ‘Seeing’:Two Ethnomethodological Challenges with AI Scenic Descriptions

Year: 2026
Type: article
Abstract: With the development of computer vision technology, multimodal conversational systems are now being built with the explicit goal of providing spoken ‘descriptions’ of their users’ surroundings (Chang et al., 2024; Cheema et al., 2025), at any moment, in response to these users’ requests (Chang et al., 2025; Čujdíková et al., 2024). We explore two ethnomethodologically significant challenges encountered by computer scientists in producing such systems: 1. The et cetera principle, notably investigated by Sacks (1963) and Garfinkel (1967). In computer vision, this principle is treated as a problem: namely, the impossibility of exhaustively formulating a situation in-so-many-words (Garfinkel & Sacks, 1970; Waismann, 1951)—which can be elaborated indefinitely—and, consequently, the fact that ‘descriptions’ (their length, level of detail, etc.; Stangl et al., 2021) are only relevant in relation to participants’ practical purposes in interaction. 2. The sequential placement of so-called ‘descriptions’ within the temporal order of users’ courses of action. Descriptions, accounts, glosses, formulations, etc., emerge as such (i.e., as practical resources for participants) in being finely timed in relation to a specific ecology of action (Goodwin, 2018; Mondada, 2007). We ground this analysis in a heuristic contrast between excerpts of blind users interacting, on the one hand, with sighted humans and, on the other hand, with artificial ‘assistants’. Through this comparison, we examine the conditions under which these co-participants’ contributions emerge as resources in relation to blind participants’ ongoing activities (e.g., crossing a road), and when they instead remain “detached” (Dreyfus, 1990) accounts, removed from the rapidly shifting temporal order of these participants’ embodied conduct. Notably, we detail how, in being accountably produced ‘too early’ or ‘too late’ relative to the practical expectancies of an unfolding activity, current multimodal agents’ contributions come to be decoupled from the “vivid present” (Au-Yeung & Fitzgerald, 2023; Garfinkel, 2006) of human participants. References Au-Yeung, T. S. H., & Fitzgerald, R. (2023). Time structures in ethnomethodological and conversation analysis studies of practical activity. The Sociological Review, 71(1), 221–242. https://doi.org/10.1177/00380261221103018 Chang, R.-C., Liu, Y., & Guo, A. (2024). WorldScribe: Towards Context-Aware Live Visual Descriptions. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, 1–18. https://doi.org/10.1145/3654777.3676375 Chang, R.-C., Natalie, R., Xu, W., Yap, J. Z. F., & Guo, A. (2025). Probing the Gaps in ChatGPT Live Video Chat for Real-World Assistance for People who are Blind or Visually Impaired. Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility. ASSETS ’25, Denver, Colorado, USA. https://doi.org/10.1145/3663547.3746319 Cheema, M., Seifi, H., & Fazli, P. (2025). Describe Now: User-Driven Audio Description for Blind and Low Vision Individuals. Proceedings of the 2025 ACM Designing Interactive Systems Conference, 458–474. https://doi.org/10.1145/3715336.3735685 Čujdíková, M., Stankovičová, M., & Vankúš, P. (2024). How The Blind See Chatgpt And Gemini: A Case Study From A Summer School For Visually Impaired Students. 7507–7512. https://doi.org/10.21125/iceri.2024.1813 Dreyfus, H. L. (1990). Being-in-the-World: A Commentary on Heidegger’s Being in Time, Division I (Vol. 102, Issue 2, pp. 290–293). Bradford. Garfinkel, H. (1967). Studies in Ethnomethodology. Polity Press. Garfinkel, H. (2006). Seeing sociologically: The routine grounds of social action. Paradigm. Garfinkel, H., & Sacks, H. (1970). On formal structures of practical action. In J. C. McKinney & E. A. Tiryakian (Eds.), Theoretical Sociology: Perspectives and Developments (pp. 338–366). Appleton-Century-Crofts. Goodwin, C. (2018). Why Multimodality? Why Co-Operative Action? (transcribed by J. Philipsen). Social Interaction. Video-Based Studies of Human Sociality, 1(2). https://doi.org/10.7146/si.v1i2.110039 Mondada, L. (2007). Multimodal resources for turn-taking: Pointing and the emergence of possible next speakers. Discourse Studies, 9(2), 194–225. https://doi.org/10.1177/1461445607075346 Sacks, H. (1963). Sociological description. Berkeley Journal of Sociology, 8, 1–16. Stangl, A., Verma, N., Fleischmann, K. R., Morris, M. R., & Gurari, D. (2021). Going Beyond One-Size-Fits-All Image Descriptions to Satisfy the Information Wants of People Who are Blind or Have Low Vision. Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility. https://doi.org/10.1145/3441852.3471233 Waismann, F. (1951). Verifiability. In G. Ryle & A. Flew (Eds.), Logic And Language (pp. 35--68). Blackwell.
Language: English
Open Access status: green
OpenAlex ID: W7163728960
FWCI:
Cites: 97
Cited by:
Related to: 10