Multimodal Perception

Vision, touch, audition and all the other senses provide us with information about the environment. In this research area the main question we are interested in is, how the information from all these different senses is put together into a coherent picture of the world surrounding us. This concerns spatial as well as temporal information, but also the dynamic aspects of information gathering and processing. Most of the work is part of the European Project “IMMERSENCE” and the “Sonderforschungsbereich 550 (SFB550)”. Our goal within the IMMERSENCE Project is to establish the neuroscientific basis for haptic enhanced multimodal interaction.

Multimodal Integration

To perceive the external environment our brain uses multiple sources of sensory information derived from several different modalities including vision, touch and audition. All these different sources of information have to be efficiently merged to form a coherent and robust percept. Here we investigate some of the mechanisms that underlie this merging of the senses in the brain. We have shown that, depending on the type of information, different combination and integration strategies are used and that prior knowledge is often required for interpreting the sensory signals.

Dynamics of IntegrationDynamics of Integration

Cue integration has been demonstrated to be close to optimal under temporally constant stimulus conditions. That is, cues are assigned different weights according to their relative reliabilities. In real-world situations, however, stimulus conditions constantly change. For example, depending on the viewing situation the reliability of cues may change over time. Here we ask whether the system takes such continuous changes in the signal and its reliability into account by adjusting the cue weights online.

Multimodal Time Perception

Multisensory signals originate simultaneously from physical events. Temporal synchrony indicates that signals are likely to have been produced by the same event and that they should be combined into a unified percept. Perceived synchrony, however, differs from physical synchrony: sensorial and perceptual factors change perceived time. We investigate these factors to understand how they can influence temporal calibration and perceived synchrony, so that we can in turn comprehend how they affect sensory combination.

Kinesthetic Tactile Interactions

During tactile exploration of a surface both, tactile and kinesthetic information can be obtained by moving the hand actively over the surface of an object. The intended purpose is to gather detailed information of the object’s surface. Thus, one would expect intuitively that tactile sensitivity should be best during active exploration in order to perceive as much structural features as possible. However, it is known that the sense of touch degrades during active limb movements.

One of our research goals is to answer the question how precisely proprioceptive information influences tactile perception. Therefore we use novel tactile and kinesthetic displays to investigate the interaction of multimodal information. Results can be taken into account to develop new tactile and haptic devices in order to generate more realistic virtual environments.

Tactile & Haptic Motion

Today, human-machine interfaces are mostly restricted to the visual and auditory modality. Depending on the task, e.g., in aviation these modalities are almost at their limits of information reception. In order to facilitate the perception of crucial information the tactile sense is being centered because of its low demanding and intuitive information transfer. The “tap on the shoulder principle” illustrates the opportunity of this sensory channel. By tapping someone’s shoulder one can immediately attract someone’s attention to a specific location. A tactile vest, which takes advantage of this principle, is already being used in several fields of application, for example in aerospace. However, there are many arising questions regarding the perception of moving stimuli on the skin. We are especially interested in the effect of visual and auditory input on the perception of moving tactile stimuli. Results of this research will shed light on the understanding of tactile perception and could enhance the development of new tactile devices.