Multi-sensory integration

Presentation for 2006 Meeting of the Society for Neuroscience

Multi-sensory integration of visual and auditory inputs:
One plus one does not always equal two

F. Zhou¹, V. Wong² & R. Sekuler² ¹Department of Psychology & Brain Science, The Johns Hopkins University, Baltimore MD
²Volen Center for Complex Systems, Brandeis University, Waltham MA

Two issues in multi-sensory processing by human observers are unresolved. First, simultaneous multisensory inputs could be processed independently at a sensory level, and then integrated probabilistically at a decision stage. Alternatively, these inputs could be integrated at a sensory level, which would produce a combined/holistic percept. Previous psychophysical studies have produced contradictory results. Second, when multisensory inputs are integrated at a sensory level, neural integration could be averaging, additive, or multiplicative as shown in single cell recording of cats. In contrast, behavioral responses to multisensory stimuli in humans have not been well quantified.
To address these issues, we measured the impact of two auditory and two visual stimuli on a bistable motion percept, in which two identical discs moved toward one another from opposite sides of a display, overlapped at the display's center, and proceeded to the opposite sides. After each trial, subjects indicated whether the discs appeared to "stream" through each other, or "bounce" off each other. The relative proportions of these two percepts, bouncing and streaming, were influenced by visual or auditory cues that accompanied the moving discs (Sekuler, Sekuler & Lau, 1997). The effect of two different visual cues and two different auditory cues were tested in isolation or in combination: luminance of the discs when they completely overlapped, duration of the overlap, intensity of a click sound accompanying the overlap, and the timing of the click sound relative to overlap. When multiple stimuli were presented in combination, we adjusted the intensity of the separate stimuli to reflect individual subject's responsiveness to each cue, such that the perceptual influences of these stimuli would have been equipotent in isolation.
Quantitative comparisons of a suite of alternative models showed that human observers did not perceive the sensory stimuli in isolation. Furthermore, the combined influence of multi-sensory stimuli did not track either the mean or the sum of individual sensory inputs. Instead, it assumed an intermediate value, so that the resulting state of the bistable visual percept could be predicted from a weighted sum of cues, with visual cues strongly dominating auditory cues.
Supported by NIH grants MH068404.