Exploring Multimodal Interactions

In 1997, Allison Sekuler, Rene Lau and Robert Sekuler discovered a new form of interaction between audition and vision: A brief sound altered the direction in which objects were seen to move. Their basic results were reported in "Sound alters vision motion perception," Nature 385 308).

Since its initial report, this interaction has attracted considerable attention from other researchers. For example, it been studied with psychophysical methods by Watanabe & Shimojo 2001 and by Scheier and colleagues (Developmental Science 2003), and it has also been studied with functional neuroimaging Bushara and colleagues 2003. Feng Zhou, Victoria Wong and Robert Sekuler have now extended this work, examining how several simultaneously present multimodal cues combine in order to influence perceived motion, and developing a mathematical model of these interactions.

To demonstrate the basic phenomenon, we have prepared a QuickTime movie,which can be freely downloaded here.

Note that the discs' movement paths are ambiguous. Because the discs are identical to one another, after they coincide their movement could have continued in the same, original direction --causing the discs to seem to pass through one another-- or they could have bounced off one another. The probability of seeing these alternative percepts is strongly affected by the co-occurence of the click in the QT movie. The accompanying sound promotes seeing the discs as colliding and then bouncing off one other.

To see this, play the movie a few times with the sound on, and then with the sound muted. You may find that after having the seen the discs as bouncing off one another, several presentations are needed with sound muted in order to experience the alternative percept. Yes, there appears to be some carry over from one condition to the other. To make the sound even more potent, you can use your index finger to occclude the spot at which the two moving discs coincide.

By carefully decreasing the click's loudness you should also be able to see that the strength of the sound's effect is altered. This result is among the phenomena that Zhou, Wong and Sekuler have been studying.

The strength of the effect varies with a number of variables, including the speed of the moving objects. So for best effect, the window within which the display is seen should be adjusted to about 4 inches square.

For more information on any aspect of this research contact Robert Sekuler