The McGurk Effect: When We Hear With Our Eyes

We have all seen a badly dubbed movie in which to know what the actor is saying we have read his lips. Suddenly, we find ourselves “listening with our eyes”, that is, we are deciphering sounds through another channel that is not the auditory one. This phenomenon in which there is a mismatch between sound and visual signals is known as the McGurk effect.

Although this experience sometimes leads us to make some mistakes that we will analyze below, it also stands as a very efficient strategy of the brain. For example, we can be at a party with very loud music and when trying to have a conversation with someone, what we do is look at their mouth to try to “read” their words.

There is something important that this effect shows us. People do not use each of our senses separately. In reality, all perceptual experience happens as a result of the interaction of more than one sense. Sight and hearing work together, so that although sometimes small errors may occur, they are our best allies …

What is the McGurk effect?

Much of our communication processes occur in situations where we can hear each other clearly. However, in recent times in the wake of the rise in video calls, we are experiencing the McGurk effect more frequently.

Sometimes, and due to bad connections, we do not listen well to the other and then we go from the auditory to the visual canal to try to read lips, but it is not easy. Sometimes, even with the person’s face and mouth as a reference, we make mistakes.

This fact was studied in the 1970s by cognitive psychologists Harry McGurk and John MacDonald. It was as a result of the 1976 research work that this term was first coined in the article Hearing Lips and Seeing Voices.

They did it after checking something very specific in a famous experiment. A fact that has been proven for decades and that never fails. If someone moves their lips pronouncing the syllable “ga”, but at the same time aloud what they say is “ba”, the brain will hear “da”. In other words, sometimes what the eyes read is not the same as what the ear perceives.

Why does the McGurk effect occur?

We already know that the McGurk effect happens when we try to “read” with our eyes and what we see (movement of the lips) does not match the actual sound. Well, it is interesting to note that the brain region where this phenomenon occurs is in the superior temporal sulcus. When trying to decode two sensory modalities (sight and sound) it is common for errors to appear.

Now, from the Department of Neurosurgery of the Houston Medical College (Texas) they point out something important in a study: when we have a face-to-face conversation, the brain listens through the ear and also deciphers words with its eyes. We do it unconsciously.

Attending to the movement of the lips allows us to reinforce communication in the event that a difficulty appears, such as environmental noise. However, there are sounds and syllables that are misinterpreted. Small inferences occur, small errors.

There are people who suffer this experience to a greater degree

We pointed it out at the beginning. People with Alzheimer’s disease are severely affected by the McGurk effect. If at some point they have to read lips to understand a message, they will not be able to decipher it.

Research works such as those carried out by Delbeuck, Collette and Van der Linden, 2007) indicate that the cause may be in the deterioration of brain connectivity.

Likewise, it has also been seen that children with expressive language disorder do not pay attention to the articulation of words, they do not use the visual channel to imitate or understand what is expressed to them. This hinders the development of the language itself.

The visual system helps us to discriminate sounds

The McGurk effect appears when we read other people’s lips and confuse some syllables or words. It is something completely normal that shows something interesting. The human being always uses the visual system during communication. It is not something that people with hearing impairments resort to exclusively.

The brain needs to make sure that the communication process is carried out efficiently. For this, it is not only limited to listening, to making use of the sense of hearing: we also use sight.

Looking at the mouth of our interlocutor facilitates understanding in any context, hence the importance of face-to-face conversations. Reading with the eyes is one more skill that we can be proud of.