What You See Is What You Hear

How Many 2s?

Although you certainly know the difference between a 2 and a 5, identifying the number of times that the figure "2" is printed onto the tile to your left is not necessarily the easiest of tasks. If you are like me, you will have to concentrate and look at the picture line-by-line to distinguish the essentially quite similar symbols that we use to denote twos and fives.

The task would be easier of course, if the 2's and 5's came to us in different colors as they do on the tile below. Or as they do for people who experience synesthesia, a particular neurological condition in which one sensory input is involuntarily linked to another sensory experience. Hearing color, seeing sounds as shapes, or - as in the opening example - numbers as colors are all examples of different kinds of synesthetic experiences.

Much has been written about synesthesia, and research on this perplexing phenomena has the potential to offer many exciting insights into how the human brain makes sense of the world; even for those of us who have never tasted yellow, seen a high-C or don't know what the solution to three times four feels like.

It Could Be This Easy...

 

I find synesthesia fascinating although - or maybe it is because - it seems so far removed from what most of us commonly experience.

Surely, I am capable of following the narrative that, let's say Kandinsky, was capable of hearing colors and that he therefore composed his famous "symphony pictures" to be melodically pleasing. And academically, I guess I could wrap my head around some of the neurological explanations that are offered to how synesthesia works, but still I feel that whenever I try to think more deeply about the synesthetic experience I run into a barrier that bars me eternally from true comprehension: If you have never shared the experience, how do you truly understand perceiving sound as color? And beyond the mere hearing sensation, what would the entire experience even feel like? It cannot feel exactly like it feels when I hear music? Or can it?

I do not have answers to any of these questions, and synesthesia seems to me like some sort of elusive superpower. It's like being Batman!

However, it is interesting to note that a very tiny element of the synesthetic experience is actually something all of us can, and actually do experience when we integrate information coming from different sensory stimuli:

The experience I am thinking of here, refers to an intelligent experiment from 1976 and a phenomena which is often referred to as the McGurk-illusion.

In the experiment, the researcher - Harry McGurk - , videotaped a person pronouncing the syllables "ga-ga", "ba-ba", "ka-ka" and "pa-pa" repeatedly, then showed the video to a group of participants. Each had no problem discerning the sounds and correctly identifying what the taped person was saying. Indeed, the recordings were so unambiguous that participants were able to correctly identify each sound by the audio recording alone.

In the next step, however, McGurk edited the tapes so that the audio recordings were no longer congruent with the visual image being shown on the screen. For example the visual information would show a person pronouncing "ga-ga" while the acoustic signal would sound "ba-ba".

Confronted with this incongruent information, the participant's brains came up with a rather astonishing interpretation of the sound that was being heard: For the majority of the participants, when asked to identify what they were hearing, their answer was a confident "da-da".

Given the two incongruent sensory inputs, the participant's brains had quickly transformed the acoustic signal into something else entirely. Neither "ba-ba", nor "ga-ga", but "da-da".

The foundation for this audio-visual illusion can be found in the brains propensity to construct coherent interpretations out of the multi-facetted information it receives. Here, the acoustic modalities of "ba" has similarities with the acoustic modalities of "da", while the lip movements for "ga" share some elements with the lip movements performed when pronouncing "da". Since the evolved brain by default operates on the (generally true) assumption that the movement of our lips is congruent with the sounds coming from them, it makes sense that the brain should figure "da" to be the most likely sound being produced, since the "da"-element is what both the videotrack and the soundtrack have in common. The point is, however, that participants don't just infer the "da"-sound, but that they truly hear it.

You can try it out here, and it should still work even though you now know what is going on. This robustness is part of what makes the effect a true illusion.

Now, I am of course perfectly aware that I am being a little childish in trying to connect the meager McGurk illusion to the grand experience of synesthesia. In the McGurk-illusion the brain takes an audio-sensory input and fuses it with a video-sensory input to produce a fused -but substantially different - acoustic experience. With synesthesia the brain is producing an acoustic sense-experience in the total absence of any auditory input.

Or back to my earlier metaphor, if experiencing synesthesia is like having Batman's powers, then the McGurk-illusion probably is like having dinner with Alfred, Batman's butler. The differences are obviously huge, but both can provide a little insight into what goes on in that Batcave we all occupy.

Main Reference: MCGURK, HARRY. (1976-12-23) Hearing lips and seeing voices. Nature, 65(5588), 497-748. DOI: 10.1038/264746a0

Thanks to Daniel R. Hawes / Psychology Today
http://www.psychologytoday.com/print/48507