5.8.2. How It Works
The illusion itself can’t happen in real life. McGurk made it by splicing the sound of someone saying “ba ba” over a video of him making a different sound, “ga ga.” When you’re not watching the video, you hear what’s actually being spoken. But when you see the speaker too, the two bits of information clash. The position of the lips is key in telling what sound someone’s making, especially for distinguishing between speech sounds (called phonemes) like “ba,” “ga,” “pa,” and “da” (those which you make by popping air out).
Visual information is really important for listening to people speak. It’s a cliché, but I know I can’t understand people as well when I don’t have my glasses on.
M.W.
We use both visual and auditory information when figuring out what sound a person is making and they usually reinforce each other, but when the two conflict, the brain has to find a resolution. In the world the brain’s used to, objects don’t usually look as if they’re doing one thing but sound as if they’re doing another.
Since visually you’re seeing “ga ga” and audition is hearing “ba ba,” these are averaged out and you perceive “da da” instead, a sound that sits equally well with both information cues. In other situations, visual information will dominate completely and change a heard syllable to the one seen in the lip movements.2
Remarkably, you don’t notice the confusion. Sensory information is combined before language processing is reached, and language processing tunes into only certain phonemes [Hack #49] . The decision as to what you hear is outside your voluntary control. The McGurk Effect shows integration of information across the senses at a completely preconscious level. You don’t get to make any decisions about this; what you hear is affected by what goes in through your eyes. It’s a good thing that in most circumstances the visual information you get matches what you need to hear.
5.8.3. End Notes
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-747.
Fusion of the sound and sight information is possible only when you have experience with a suitable compromise phoneme. One of the interesting things about phonemes is that they are perceived as either one thing or the other, but not as in-between values. So although there exists a continuum of physical sounds in between “ba” and “da,” all positions along this spectrum will be perceived as either “ba” or “da,” not as in-between sounds (unlike, say, colors, which have continuous physical values that you can also perceive). This is called categorical perception.
5.8.4. See Also
Hearing with Your Eyes (http://ccms.ntu.edu.tw/~karchung/Phonetics%20II%20page%20seventeen.htm; QuickTime) has a collection of McGurk Effect movies.
Taken from : Mind Hacks
