Delaying the presentation of information to one modality relative to another (an intersensory temporal offset) impairs performance on a wide range of tasks. We have recently shown, however, that a few minutes exposure to delayed visual feedback induces sensorimotor temporal adaptation, returning performance to normal. Here, we examine whether adaptation to delayed vestibular feedback is possible. Subjects were placed on a motion platform, and were asked to perform a stabilization task. The task was similar to balancing a rod on the tip of your finger. Specifically, the platform acted as if it were on the end of an inverted pendulum, with subjects applying an acceleration to the platform via a joystick. The more difficulty one has in stabilizing the platform the more it will oscillate, increasing the variablilty in the platform's position. The experiment was divided into 3 sections. During the Baseline section (5 minutes), subjects performed the task with immediate vestibular feedback. They then were presented with a Training section, consisting of 4 sessions (5 minutes each) during which vestibular feedback was delayed by 500 ms. Finally, subjects were presented with a Post-test (two minutes) with no feedback delay. Subjects performed rather well in the Baseline section (average standard deviation of platform tilt was 1.37 degrees). The introduction of the delay greatly impaired performance (8.81 degrees standard deviation in the 1st Training session), but performance rapidly showed significant improvement (5.59 degrees standard deviation during the last training section, p<0.04). Subjects clearly learned to compensate, at least partially, for the delayed vestibular feedback. Performance during the Post-test was worse than during Baseline (2.48° standard deviation in tilt). This decrease suggests that the improvement seen during training might be the result of intersensory temporal adaptation.

When moving through space, we continuously update our egocentric mental spatial representation of our surroundings. We call this seemingly effortless, automatic, and obligatory (i.e., hard-to-suppress) process spatial updating. Our goal here is twofold: 1) To quantify spatial updating; 2) Investigate the importance and interaction of visual and vestibular cues for spatial updating.

In a learning phase (20 min) subjects learned the positions of twelve targets attached to the walls, 2.5m away. Subjects saw either the real environment or a photo-realistic copy presented via a head-mounted display (HMD). A motion platform was used for vestibular stimulation. In the test phase subjects were rotated to different orientations and asked to point as quickly and accurately as possible to four targets announced consecutively via headphones.

In general, subjects had no problem mentally updating their orientation in space and were as good as for rotations where they were immediately returned to the original orientation. Performance, quantified as response time, absolute pointing error and pointing variability, was best in the real world condition. However, when the field of view was limited via cardboard blinders to match that of the HMD (40x30 deg), performance decreased and was comparable to the HMD condition. Presenting turning information only visually (through the HMD) hardly altered those results. In both the real world and HMD conditions, spatial updating was obligatory in the sense that it was significantly more difficult to IGNORE ego-turns (i.e., point as if not having turned) than to UPDATE them as usual.

Speeded pointing tasks proved to be a viable method for quantifying spatial updating. We conclude that, at least for the limited turning angles used (<60 deg), the Virtual Reality simulation of ego-rotation was as effective and convincing (i.e., hard to ignore) as its real world counterpart, even when only visual information was presented.

ACKNOWLEDGEMENTS: This research was funded by the Max-Planck Society and the Deutsche Forschungsgemeinschaft (SFB 550)

In most virtual reality (VR) applications turns are misperceived, which leads to disorientation. Here we focus on two cues providing no absolute spatial reference: optic flow and vestibular cues. We asked whether: (a) both visual and vestibular information are stored and can be reproduced later; and (b) if those modalities are integrated into one coherent percept or if the memory is modality specific.

We used a VR setup including a motion simulator (Stewart platform) and a head-mounted display for presenting vestibular and visual stimuli, respectively. Subjects followed an invisible randomly generated path including heading changes between 8.5 and 17 degrees. Heading deviations from this path were presented as vestibular roll rotation. Hence the path was solely defined by vestibular (and proprioceptive) information. The subjects' task was to continuously adjust the roll axis of the platform to level position. They controlled their heading with a joystick and thereby maintained an upright position. After successfully following a vestibularly defined path twice, subjects were asked to reproduce it from memory. During the reproduction phase, the gain between the joystick control and the resulting visual and vestibular turns were independently varied.

Subjects learned and memorized curves of the vestibularly defined virtual path and were able to reproduce the amplitudes of the turns. This demonstrates that vestibular signals can be used for spatial orientation in virtual reality. Since the modality with the bigger gain factor had a dominant effect on the reproduced turns, the integration of visual and vestibular information seems to follow a max rule, in which the larger signal is responsible for the perceived and memorized heading change.

In order to rapdily and accurately interact with the world, we need to perceive the consequences of our actions. It should not be surprising, then, that delaying the consequences of our actions, or delaying feedback about our actions, impairs performance on a wide range of tasks. We have recently shown that a few minutes exposure to delayed visual feedback induces sensorimotor temporal adaptation, returning performance to near normal levels. While visual feedback plays a large role in many tasks, there are some tasks for which vestibular perception is more critical. Here, we examine whether adaptation to delayed vestibular feedback is possible.

To test for vestibular temporal adaptation, subjects were placed on a motion platform and were asked to perform a stabilization task. The task was similar to balancing a rod on the tip of your finger. Specifically, the platform acted as if it were on the end of an inverted pendulum. Subjects moved the platform by applying an acceleration to it via a joystick. The experiment was divided into 3 sections. During the Baseline section, which lasted 5 minutes, subjects performed the task with immediate vestibular feedback. They then were presented with a Training section, which consisted of 4 sessions (5 minutes each) during which vestibular feedback was delayed by 500 ms. Finally, subjects performance on the task with immediate feedback was remeasured during a 2 minute Post-test.

The more difficulty one has in stabilizing the platform the more it will oscillate, increasing the variablilty in the platform's position and orientation. Accordingly, positional variance served as the primary measure of the subjects' performance. Subjects did rather well in the Baseline section (average standard deviation of platform tilt was 1.37 degrees). The introduction of the delay greatly impaired performance (8.81 degrees standard deviation in the 1st Training session), but performance rapidly showed significant improvement (5.59 degrees standard deviation during the last training session).

Subjects clearly learned to compensate, at least partially, for the delayed vestibular feedback. Performance during the Post-test showed a negative aftereffect: The performance with a 500 ms delay worse during the Post-test than during Baseline (2.48 degrees versus 1.37 degreees), suggesting that the improvement seen during training was the result of intersensory temporal adaptation.

Although different sensory organs have quite different characteristics, humans have no problems in their combined evaluation. Particularly, when asked to stabilize their position in space, humans need the capability to integrate the different sensory inputs. For the following stabilization task humans mainly use the vestibular, visual, and proprioceptive senses. To study this sensor fusion in a body stabilization task, we used a motion platform with six degrees of freedom for the vestibular stimulus, a head mounted display (HMD) for the visual stimulus, and a joystick as an input device. The motion platform and the HMD simulated the physical model of an inverse pendulum. Using the joystick, the subject could exert a force (acceleration) on the pendulum and thereby control the state of the model. In our experiments, the subjects had to balance themselves on the pendulum against changes in roll, yaw, or both axes simultaneously. They had either vestibular information, visual information, or both. The visual stimulus was a random-dot cloud with limited life-time dots and an artificial horizon in order to match the character of the vestibular stimulus (absolute positional information for roll, but only information about changing of position for yaw).

Subject performed a pre-test, six training sessions, and a post-test. In the pre- and post-test sections, the subjects had to perform a stabilization task for all nine possible conditions (each lasting 200 seconds). For the training section, the four subjects were divided into two groups receiving visual or vestibular input (VISGroup and VESTGroup, respectively). During the training section, the performance of all subjects showed a large overall improvement. In the in pre- and post-test of the yaw stabilization task, subjects performance (mean absolute positional error) was much better with visual than with vestibular stimulus (pre-test: vestibular 6.00°, visual 2.99°, t(3)=7.32, p<0.005; post-test: vestibular 5.18°, visual 1.64°, t(3)=6.83, p<0.006). For the roll task, all subjects had a much higher increase in performance with the vestibular than with the visual stimulus (vestibular 2.73°, visual 0.98° decrease of the average absolute position from pre- to post-test, t(3)= 6.55, p<0.007). Finally, the VESTGroup showed a significant improvement in the visual roll task (pre-test 3.02°, post-test 1.73° standard deviation, t(1)=14.8, p<0.043). The VISGroup showed also a large but non-significant improvement in the vestibular roll task (pre test 4.26°, post test 2.19° standard deviation). This suggests that subjects are able to transfer their learned skill from one input modality to another.

In order to know where we are when moving through space, we constantly update our mental egocentric representation of the environment, matching it to our motion. This process, termed spatial updating, is mostly automatic, effortless, and obligatory (i.e., hard-to-suppress). Our goal here is twofold: 1) To quantify spatial updating; 2) To investigate the importance and interaction of visual and vestibular cues for spatial updating.

The stimuli consisted of twelve targets (the numbers from 1 to 12, arranged in a clockface manner) attached to the walls of a 5x5m room. Subjects saw either the real room or a photo-realistic 3D model of it presented via a head-mounted display (HMD). For vestibular stimulation, subjects were seated on a Stewart motion platform. After each rotation, the subjects' task was to point as quickly and accurately as possible to four targets announced consecutively via headphones. Spatial updating performance was quantified in terms of response time and pointing error (absolute error and variance) in three different spatial updating conditions: Subjects were (a) rotated to a different orientation (UPDATE condition); (b) rotated as in (a), but asked to ignore that rotation and point as if not having turned (IGNORE); (c) rotated to a new orientation and immediately back to the original orientation before being asked to point (CONTROL); Each of the twelve subjects was presented with six stimulus conditions (blocks A-F, 15 min. each) in balanced order, with different amount of visual and vestibular information available.

Performance, especially response times, varied considerably between subjects, but showed the same overall pattern: 1) Performance was best in the real world condition (block A). When the field of view was limited via cardboard blinders (block B) to match that of the HMD (40x30°), performance decreased and was comparable to the HMD condition (block C). Presenting only visual information for the turns (through the HMD, block D) decreased the performance slightly further. 2) In those four blocks where there was visual information available about the rotation, subjects performed equally well in the UPDATE and CONTROL conditions. Performance in the IGNORE condition, however, was significantly impaired, indicating that spatial updating was indeed obligatory in the sense of being hard-to-suppress. 3) Without visual information about the turns (i.e., when subjects were blindfolded (block E) or saw a constant image of the scene (block F)), IGNORE performance increased and was comparable to the UPDATE performance. This suggests that spatial updating was no longer obligatory when visual cues about the motion were removed.

Speeded pointing tasks proved to be a viable method for quantifying spatial updating. We conclude that, at least for the regular target arrangement and limited turning angles used (<60°), the Virtual Reality simulation of ego-rotation was as effective and convincing (i.e., hard to ignore) as its real world counterpart, even when only visual information was available.

ACKNOWLEDGEMENTS: This research was funded by the Max-Planck Society and the Deutsche Forschungsgemeinschaft (SFB 550 Erkennen, Lokalisieren, Handeln: neurokognitive Mechanismen und ihre Flexibilität).

Perception of ego turns is crucial for navigation and self-localization. Yet in most virtual reality (VR) applications turns are misperceived, which leads to disorientation. Here we focus on two cues providing no absolute spatial reference: optic flow and vestibular cues. We asked whether: (a) both visual and vestibular information are stored and can be reproduced later; and (b) if those modalities are integrated into one coherent percept or if the memory is modality specific. In the following experiment, subjects learned and memorized turns and were able to reproduce them even with different gain factors for the vestibular and visual feedback.

We used a VR setup including a motion simulator (Stewart platform) and a head-mounted display for presenting vestibular and visual stimuli, respectively. Subjects followed an invisible randomly generated path including heading changes between 8.5 and 17 degrees. Heading deviations from this path were presented as vestibular roll rotation. Hence the path was solely defined by vestibular (and proprioceptive) information. One group of subjects' continuously adjusted the roll axis of the platform to level position. They controlled their heading with a joystick and thereby maintained an upright position. The other group was passively guided through the sequence of heading turns without any roll signal. After successfully following a vestibularly defined path twice, subjects were asked to reproduce it from memory. During the reproduction phase, the gain between the joystick control and the resulting visual and vestibular turns were independently varied by a factor of 1/sqrt(2), 1 or sqrt(2).

Subjects from both groups learned and memorized curves of the vestibularly defined virtual path and were able to reproduce the amplitudes of the turns. This demonstrates that vestibular signals can be used for spatial orientation in virtual reality. Since the modality with the bigger gain factor had for both groups a dominant effect on the reproduced turns, the integration of visual and vestibular information seems to follow a max rule, in which the larger signal is responsible for the perceived and memorized heading change.

Introduction:

Over the course of evolution humans as well as other animals learned to navigate through complex environments. Such navigation had two main goals: to find food and to find the way back to shelter. For most moving organisms it is important to know their location in the world and maintain some internal representation of it. For higher species it is most likely that multiple sensory systems provide information to solve this task. Consequently, to study human behavior in a complex environment it is important that the experimenter has full control over the stimulus for multiple senses. Furthermore, it is crucial to guarantee the following:

A) The stimulus, and the information it conveys has to be precisely controllable; B) The experimental conditions have to be repeatable; and C) The stimulus conditions have to be independent of the individual characteristics of the observer.

Virtual Environments have to some degree offered a solution for these demands. Recently, it has become increasingly possible to conduct psychophysical experiments with more than one sensory modality at a time. In this thesis, Virtual Reality (VR) technology was used to design multi-sensory experiments which look into some aspects of the complex multi-modal interactions of human behavior.

Contents:

The first part of this PhD thesis describes a Virtual Reality laboratory which was built to allow the experimenter to stimulate four senses at the same time: vision, acoustics, touch, and the vestibular sense of the inner ear. Special purpose equipment is controlled by individual computers to guarantee optimal performance of the modality specific simulations. These computers are connected in a network functioning as a distributed system using asynchronous data communication. The second part of the thesis presents two experiments which investigate the ability of humans to perform spatial updating. These experiments contribute new scientific results to the field and serve, in addition, as proof of concept for the VR-lab. More specifically, the experiments focus on the following main questions:

A) Which information do humans use to orient in the environment and maintain an internal representation about the current location in space?; B) Do the different senses code their percept in a single spatial representation which is used across modalities, or is the representation modality specific?

Results and Conclusions:

The experimental results allow the following conclusions: A) Even without vision or acoustics, humans can verbally judge the distance traveled, peak velocity, and to some degree even maximum acceleration using relative scales. Therefore, they can maintain a good spatial orientation based on proprioception and vestibular signals; B) Learning the sequence of orientation changes with multiple modalities (vision, proprioception and vestibular input) enables humans to reconstruct their heading changes from memory. In situations with conflicting cues, the maximum percept from either of the modalities had a major influence on the reconstruction. Most of the naïve subjects did not notice any conflicts between modalities. In total, this seems to suggest that there is a single spatial reference frame used for spatial memory. One possible model for cue integration might be based on a dynamically weighted sum of all modalities which is used to come up with a coherent percept and memory for spatial location and orientation.

Rapid and accurate interaction with the world requires that proper spatial and temporal alignment between sensory modalities be maintained. The introduction of a misalignment (either spatial or temporal) impairs performance on most spatial tasks. For over a century, it has been known that a few minutes of exposure to a spatial misalignment can induce a recalibration of intersensory spatial relationships, a phenomenon called Spatial Adaptation. Here, we present evidence that the sensorimotor system can also adapt to intersensory temporal misalignments, a phenomena that we call Temporal Adaptation. Temporal Adaptation is striking parallel to Spatial Adaptation, and has strong implications for the understanding of spatial cognition and intersensory integration.

The main question addressed in the Motion-Lab is: How do we know where we are? Normally, humans know where they are with respect to the immediate surround. The overall perception of this environment results from the integration of multiple sensory modalities. Here we use Virtual Reality to study the interaction of visual, vestibular, and proprioceptive senses and explore the way these senses might be integrated into a coherent perception of spatial orientation and location. This Technical Report describes a Virtual Reality laboratory, its technical implementation as a distributed network of computers and discusses its usability for experiments designed to investigate questions of spatial orientation.

Purpose: Delaying visual feedback can greatly impair performance on a wide range of tasks. Last year at ARVO we showed that humans can learn to perform nearly as well with delayed and immediate visual feedback on a simple obstacle avoidance task with abstract stimuli. Here, we examine the effects of training under more realistic conditions, with particular focus on the generalization of training to novel paths. Method: Both experienced psychophysical observers and paid, naive volunteers were asked to maneuver a virtual car along a convoluted path in a high-fidelity virtual environment. The stimuli were projected via an SGI Onyx 2 IR onto a 180 screen. All experiments used constant speeds, with the subject controlling the direction of travel using a forced-feedback steering wheel. Exp 1 used a single path. In the pre-test, 7 speeds were presented in random order 5 times. Visual feedback was immediate. Subjects then trained with a 280 ms delay using increasingly faster speeds. The post-test for each subject consisted of 5 trials at the fastest speed they had successfully completed in the pre-test. In Exp 2, subjects were given 15 trials of practice using immediate feedback. Feedback was delayed in all subsequent trials. Following the practice, baseline performance with 5 paths at 3 speeds was measured, subjects were trained on a single path, and then performance on 5 new paths was measured. Results: A strong negative aftereffect was found in Exp 1: performance with immediate feedback dropped (roughly 80(approx. 60levels. Moreover, this performance increase was found with novel paths. Conclusions: These results are consistent with sensorimotor adaptation to the intersensory temporal differences, and are not consistent with over-training, cognitive or motoric memorization of the path, or simple behavioral strategies. The presence of adaptation is further supported by the fact that training seemed to reduce the perceptual magnitude of the delay. Regardless of the underlying mechanism, however, it is clear that subjects can learn to accurately maneuver vehicles at high speeds with delayed feedback.

The consequences of an action usually occur immediately. One of the more important ramifications of this is that delaying visual feedback greatly impairs performance on a wide range of tasks. Cunningham et al. (ARVO 1999) have demonstrated that with practice, humans can perform equally well with delayed and immediate visual feedback on a simple obstacle avoidance task with abstract stimuli. Here, we examine the effects training in more detail under more realistic conditions. Naive volunteers maneuvered a virtual car along a convoluted path in a high-fidelity virtual environment, which was projected onto a 180 deg. screen. Subjects drove at a constant speed, steering with a forced-feedback steering wheel. In Exp. 1, subjects were presented with 7 speeds in random order 5 times, using immediate visual feedback and a single path. Subsequently, subjects trained with a 280 ms delay, and then were presented with 5 trials at the fastest speed they had successfully completed in the first section. In Exp 2, subjects were given 15 trials of practice using immediate feedback. Following this, subjects' performance with 5 paths at 3 speeds was measured, then they trained on a new path, and finally they were presented with 5 new paths at the 3 speeds. In both experiments, training with delayed feedback improved performance accuracy with delayed feedback, and seemed to reduce the perceptual magnitude of the delay. In Exp. 1, the training also lowered performance with immediate feedback. In Exp. 2, the improved performance generalized to novel paths. These results are the main hallmarks for sensorimotor adaptation, and suggest that humans can adapt to intersensory temporal differences. Regardless of the underlying mechanism, however, it is clear that accurate control of vehicles at high speeds with delayed feedback can be learned.

Purpose: The vestibular system is known to measure accelerations for linear forward movements. Can humans integrate these vestibular signals to derive reliably distance and velocity estimates? Methods: Blindfolded naive volunteers participated in a psychophysical experiment using a Stewart-Platform motion simulator. The vestibular stimuli consisted of Gaussian-shaped translatory velocity profiles with a duration of less than 4 seconds. The full two-factorial design covered 6 peak accelerations above threshold and 5 distances up to 25cm with 4 repetitions. In three separate blocks, the subjects were asked to verbally judge on a scale from 1 to 100 traveled distance, maximum velocity and maximum acceleration. Results: Subjects perceived distance, velocity and acceleration quite consistently, but with systematic errors. The distance estimates showed a linear scaling towards the mean and were independent of accelerations. The correlation of perceived and real velocity was linear and showed no systematic influence of distances or accelerations. High accelerations were drastically underestimated and accelerations close to threshold were overestimated, showing a logarithmic dependency. Conclusions: Despite the fact that the vestibular system measures acceleration only, one can derive peak velocity and traveled distance from it. Interestingly, even though maximum acceleration was perceived non linear, velocity and distance was judged consistently linear.

Purpose: The vestibular system is known to measure changes in linear and angular position changes in terms of acceleration. Can humans judge these vestibular signals as acceleration and integrate them to reliably derive distance and velocity estimates?

Methods: Twelve blindfolded naive volunteers participated in a psychophysical experiment using a Stewart-Platform motion simulator. The vestibular stimuli consisted of Gaussian-shaped translatory or rotatory velocity profiles with a duration of less than 4 seconds. The full two-factorial design covered 6 peak accelerations above threshold and 5 distances with 4 repetitions. In three separate blocks, the subjects were asked to verbally judge on a scale from 1 to 100 the distance traveled or the angle turned, maximum velocity and maximum acceleration.

Results: Subjects judged the distance, velocity and acceleration quite consistently, but with systematic errors. The distance estimates showed a linear scaling towards the mean response and were independent of accelerations. The correlation of perceived and real velocity was linear and showed no systematic influence of distances or accelerations. High accelerations were drastically underestimated and accelerations close to threshold were overestimated, showing a logarithmic dependency. Therefore, the judged acceleration was close to the velocity judgment. There was no significant difference between translational and angular movements.

Conclusions: Despite the fact that the vestibular system measures acceleration only, one can derive peak velocity and traveled distance from it. Interestingly, even though maximum acceleration was perceived non-linearly, velocity and distance judgments were linear.

The study of human perception has evolved from examining simple tasks executed in reduced laboratory conditions to the examination of complex, real-world behaviors. Virtual environments represent the next evolutionary step by allowing full stimulus control and repeatability for human subjects, and a testbed for evaluating models of human behavior.

Visual resolution varies dramatically across the visual field, dropping orders of magnitude from central to peripheral vision. Humans move their gaze about a scene several times every second, projecting task-critical areas of the scene onto the central retina. These eye movements are made even when the immediate task does not require high spatial resolution. Such attentionally-driven eye movements are important because they provide an externally observable marker of the way subjects deploy their attention while performing complex, real-world tasks. Tracking subjects' eye movements while they perform complex tasks in virtual environments provides a window into perception. In addition to the ability to track subjects' eyes in virtual environments, concurrent EEG recording provides a further indicator of cognitive state.

We have developed a virtual reality laboratory in which head-mounted displays (HMDs) are instrumented with infrared video-based eyetrackers to monitor subjects' eye movements while they perform a range of complex tasks such as driving, and manual tasks requiring careful eye-hand coordination. A go-kart mounted on a 6DOF motion platform provides kinesthetic feedback to subjects as they drive through a virtual town; a dual-haptic interface consisting of two SensAble Phantom extended range devices allows free motion and realistic force-feedback within a 1 m3 volume

We are using two PHANToM 3.0 force feedback devices in one workspace in order to perform studies of one-hand precision grip tasks or two handed pointing tasks. The visual environment is rendered by an Onyx workstation and presented in a specialized stereo head mounted display that allows eye tracking. The head position and orientation is tracked with an electromagnetic system (Fastrak). Together, these systems allow the current gaze direction in world coordinates to be computed in real time. The artificial visual and haptic environment may contain free movable objects as well as stationary parts, whereas the objects can be complex or simple. The graphical user interface allows all object properties to be changed online. In addition, we are using free programmable force effects that depend on position or velocity information. Psychophysical experiments that simulate eye-hand coordination in complex 3D scenes demonstrate results that seem to be in line with previous research in real environments. Thus, we believe that the dual-PHANToM instrument is an experimental device that is well suited for various studies of visual motor coordination, with special reference to aspects like timing and adaptation.

The development of autonomous as well as situated robots is one of the great remaining challenges and involves a number of different scientific disciplines. In spite of recent dramatic progress, it remains worthwhile to examine natural systems, because their abilities are still out of reach. Motivated by research work done in the fields of cognitive systems, visual perception, and psychology of memory we designed and implemented a memory architecture for visual tasks. Structural and functional concepts of the memory architecture were modeled on the ones found in natural systems. We present an efficient implementation based on parallel programming techniques. The memory module is integrated into a distributed system for speech and image analysis, which is currently developed in the Sonderforschungsbereich (SFB) 360, Situated Artificial Communicators, where a hybrid vision system combining neural and semantic networks is used.