Work

Effects of culture on think out loud usability evaluation

Year: 2012
Type: dissertation
Study design: Clinical Trial
Abstract: The Think Out Loud (TOL) method has been shown to be a valid and valuable usability evaluation technique in Western cultures. To determine whether TOL affects workload, performance and satisfaction, 45 Chinese, 30 Indian and 81 American students were asked to perform 3 scanning and 1 procedural task. All international participants (Chinese and Indian) had lived outside of their birth country for less than 2 years and all American participants had been born in the United States of America. Participants were randomly assigned to complete tasks using classic TOL, as described by Ericsson and Simon (1993), in either their native language, English, or in a control condition where no mention of TOL or silence was made. All participants were timed and their correct and incorrect solutions recorded. After each task participants were asked to complete a modified NASA Task Load Index (TLX) and 7 point Likert scale relating to satisfaction and performance. A Pearson product-moment correlation coefficient revealed that when participants self reports of workload increased, their performance and satisfaction decreased, and that satisfaction rating rose when performance increased. This discovery provided greater value to the utilized variables and the results which were analyzed through a one-way analysis of variance. The Indian participants reported no difference in workload rating, performance and satisfaction among TOL, TOL native and silent control groups. This result may be attributed to the considerable participant reported levels of acclimation, to the Western culture, experienced in both India and America. The American participants did not report a significant difference in workload rating, performance or satisfaction between the TOL and silent control groups. This result validates and mirrors previous research and emphasizes the differences found among Chinese participants. In the TOL native condition, Chinese participants experienced a greater level of mental workload and lower levels of both performance and satisfaction than their counterparts in the silent control condition. Those in the TOL condition performed worse and reported lower satisfaction levels than those in the silent control condition as well. Chinese participants also rated TOL as significantly more detrimental to their performance then American users in scanning tasks, and TOL more detrimental then TOL native in procedural tasks. This latest result underscores the potential of TOL performance to confound usability results as Chinese participants actually experienced higher workload levels in TOL native conditions then in TOL. These findings of TOL produced modifications in usability's core variables (efficiency, effectiveness and satisfaction) in Eastern participants combined with the lack of variability in Western participants, further endorses TOL as effective for Western cultures and indicates evaluation methods other than TOL may need to be employed in Eastern cultures.
Author: Simon Psavko
Language: English
Open Access status: green
OpenAlex ID: W2288766650
Cites: 38
Cited by: 1
Related to: 10