Files
Abstract
Speech perception is a multimodal process that involves both auditory and visual cues. Visual elements, such as lip and jaw movements, can significantly enhance comprehension, especially for those learning a second language (L2 learners). The McGurk effect, a perceptual illusion where mismatched auditory and visual signals alter speech perception, highlights the important role of visual information in this process. Moreover, visual cues like images have been shown to improve language comprehension in L2 learners, particularly when the visuals are directly related to the content. These findings underscore the significant role of visual cues in both speech perception and language comprehension.Despite strong evidence supporting the importance of visual cues, individual variations in experiencing the McGurk effect remain unexplained. Factors such as lipreading skills and cognitive functions may contribute to this variability, but further research is needed to fully understand these differences. In addition, a recent study failed to find a positive effect of visual cues on language comprehension. This discrepancy may be attributed to the low effectiveness of the visual cues used in the study or the difficulty of the test materials. A more robust experimental design is necessary to definitively answer this question.
In my dissertation, I designed experiments to investigate the factors that may contribute to the McGurk effect susceptibility. By analyzing the correlation between McGurk effect susceptibility and working memory, as well as between McGurk effect susceptibility and lipreading skills, I found that lipreading skills were weakly correlated with individual differences in McGurk effect susceptibility. In contrast, working memory had no significant impact. Furthermore, by analyzing the effects of visual cues on language comprehension in L2 learners, I found that content-related visuals improved comprehension in language tasks of medium difficulty. This enhancement was observed in L2 learners with intermediate English proficiency, whereas participants with low or high English proficiency levels did not show significant benefits. My study supports the complex role of visual cues in language acquisition for L2 learners.