MachullaDFE2011 3 TK Machulla M Di Luca E Fröhlich MO Ernst 2012-03-00 1 217 89 97 Experimental Brain Research Recent studies show that repeated exposure to an asynchrony between auditory and visual stimuli shifts the point of subjective simultaneity. Usually, the measurement stimuli used to assess this aftereffect are interleaved with short re-exposures to the asynchrony. In a first experiment, we show that the aftereffect declines during measurement in spite of the use of re-exposures. In a second experiment, we investigate whether the observed decline is either due to a dissipation of the aftereffect with the passage of time, or the result of using measurement stimuli with a distribution of asynchronies different from the exposure stimulus. To this end, we introduced a delay before measuring the aftereffects and we compared the magnitude of the aftereffect with and without delay. We find that the aftereffect does not dissipate during the delay but instead is stored until new sensory information in the form of measurement stimuli is presented as counterevidence (i.e., stimuli with an asynchrony that differs from the one used during exposure). no notspecified http://www.kyb.tuebingen.mpg.de/ published 8 15017 18824 15017 15422 5923 3 M Di Luca B Knörlein MO Ernst M Harders 2011-06-00 5 85 245 259 Brain Research Bulletin Spring compliance is perceived by combining the sensed force exerted by the spring with the displacement caused by the action (sensed through vision and proprioception). We investigated the effect of delay of visual and force information with respect to proprioception to understand how visual–haptic perception of compliance is achieved. First, we confirm an earlier result that force delay increases perceived compliance. Furthermore, we find that perceived compliance decreases with a delay in the visual information. These effects of delay on perceived compliance would not be present if the perceptual system would utilize all force–displacement information available during the interaction. Both delays generate a bias in compliance which is opposite in the loading and unloading phases of the interaction. To explain these findings, we propose that information during the loading phase of the spring displacement is weighted more than information obtained during unloading. We confirm this hypothesis by showing that sensitivity to compliance during loading movements is much higher than during unloading movements. Moreover, we show that visual and proprioceptive information about the hand position are used for compliance perception depending on the sensitivity to compliance. Finally, by analyzing participants’ movements we show that these two factors (loading/unloading and reliability) account for the change in perceived compliance due to visual and force delays. no notspecified http://www.kyb.tuebingen.mpg.de/ published 14 15017 15422 15017 18824 6616 3 M Di Luca 2011-04-00 8 51 961 967 Vision Research Handling a compliant object using a pinch grasp provides sensory information about deformation and resistive force from both index finger and thumb. In this paper, an object with rigid surfaces and composed of two compliant materials fixed on a central position is used to address how information from the two fingers is integrated into a holistic percept of compliance. Results indicate that with small differences in material compliance there is a small tendency to rely more on the information at the index finger. With larger differences in material compliance participants adopt different movement patterns with the two fingers to explore the objects. Compliance judgments depend on the relative amount of motion and force exerted—the finger that presses more contributes more to the final estimate. This tendency is consistent with the utilization of a unique force signal for the two fingers. The uneven contribution of the sensory information in the pinch leads to predictable compliance discrimination performance from the performance obtained using the fingers independently. no notspecified http://www.kyb.tuebingen.mpg.de/ published 6 15017 18824 6313 3 JX Maier M Di Luca U Noppeney 2011-02-00 1 37 245 256 Journal of Experimental Psychology: Human Perception and Performance Combining information from the visual and auditory senses can greatly enhance intelligibility of natural speech. Integration of audiovisual speech signals is robust even when temporal offsets are present between the component signals. In the present study, we characterized the temporal integration window for speech and nonspeech stimuli with similar spectrotemporal structure to investigate to what extent humans have adapted to the specific characteristics of natural audiovisual speech. We manipulated spectrotemporal structure of the auditory signal, stimulus length, and task context. Results indicate that the temporal integration window is narrower and more asymmetric for speech than for nonspeech signals. When perceiving audiovisual speech, subjects tolerate visual leading asynchronies, but are nevertheless very sensitive to auditory leading asynchronies that are less likely to occur in natural speech. Thus, speech perception may be fine-tuned to the natural statistics of audiovisual speech, where facial movements always occur before acoustic speech articulation. no notspecified http://www.kyb.tuebingen.mpg.de/ published 11 15017 15422 15017 18824 15017 18826 6128 3 M Di Luca 2010-12-00 6 19 569 584 Presence: Teleoperators and Virtual Environments A virtual reality (VR) system tracks one or more objects to generate the depiction of a virtual environment from the user's vantage point. No system achieves this instantaneously: changes in the depicted virtual environment are delayed from changes in the position of the objects being tracked. In this paper, a method is proposed to quantify this time difference, the end-to-end delay of the VR system. Two light-sensing devices and two luminance gradients are used to simultaneously encode the position of one tracked object and its virtual counterpart. One light-sensing device is attached to the tracked object and it captures light from the gradient in the physical environment. The other device captures light from the gradient in the virtual environment. A measurement is obtained by moving the tracked object repetitively (by hand) across the gradient. The end-to-end delay is the asynchrony between the signals generated by the two light-sensing devices. The results collected with oscillatory movements performed at different frequencies indicate that for some VR systems, the end-to-end delay might not be constant but could vary as a function of the oscillation frequency. no notspecified http://www.kyb.tuebingen.mpg.de/fileadmin/user_upload/files/publications/DiLuca_presence_6128.pdf published 15 15017 15422 15017 18824 5625 3 M Kuschel M Di Luca M Buss RL Klatzky 2010-10-00 4 3 234 244 IEEE Transactions On Haptics The compliance of a material can be conveyed through mechanical interactions in a virtual environment and perceived through both visual and haptic cues.We investigated this basic aspect of perception. In two experiments subjects performed compliance discriminations, and the mean perceptual estimate (PSE) and and the perceptual standard deviation (proportional to JND) were derived from psychophysical functions. Experiment 1 supported a model in which each modality acted independently to produce a compliance estimate, and the two estimates were then integrated to produce an overall value. Experiment 2 tested three mathematical models of the integration process. The data ruled out exclusive reliance on the more reliable modality and stochastic selection of one modality. Instead the results supported an integration process that constitutes a weighted summation of two random variables, which are defined by the single modality estimates. The model subsumes optimal fusion but provided valid predictions also if the weights were not optimal. Weights were optimal (i.e., minimized variance) when vision and haptic inputs were congruent, but not when they were incongruent. no notspecified http://www.kyb.tuebingen.mpg.de/ published 10 15017 15422 15017 18824 6084 3 R Höver M Di Luca M Harders 2010-10-00 1:7 8 1 23 Transactions on Applied Perception In this article, the data-driven haptic rendering approach presented in our earlier work is assessed. The approach relies on recordings from real objects from which a data-driven model is derived that captures the haptic properties of the object. We conducted two studies. In the first study, the Just Noticeable Difference (JND) for small forces, as encountered in our set-up, was determined. JNDs were obtained both for active and passive user interaction. A conservative threshold curve was derived that was then used to guide the model generation in the second study. The second study examined the achievable rendering fidelity for two objects with different stiffnesses. Subjects directly compared data-driven virtual feedback with the real objects. Results indicated that it is crucial to include dynamic material effects to achieve haptic feedback that cannot be distinguished from real objects. Results also showed that the fidelity is considerably decreased for stiffer objects due to limits of the display hardware. no notspecified http://www.kyb.tuebingen.mpg.de/ published 22 15017 15422 15017 18824 6500 3 M Di Luca M Ernst B Backus 2010-09-00 20 20 1860 1863 Current Biology How does the brain construct a percept from sensory signals? One approach to this fundamental question is to investigate perceptual learning as induced by exposure to statistical regularities in sensory signals [1–7]. Recent studies showed that exposure to novel correlations between sensory signals can cause a signal to have new perceptual effects [2, 3]. In those studies, however, the signals were clearly visible. The automaticity of the learning was therefore difficult to determine. Here we investigate whether learning of this sort, which causes new effects on appearance, can be low level and automatic by employing a visual signal whose perceptual consequences were made invisible—a vertical disparity gradient masked by other depth cues. This approach excluded high-level influences such as attention or consciousness. Our stimulus for probing perceptual appearance was a rotating cylinder. During exposure, we introduced a new contingency between the invisible signal and the rotation direction of the cylinder. When subsequently presenting an ambiguously rotating version of the cylinder, we found that the invisible signal influenced the perceived rotation direction. This de monstrates that perception can rapidly undergo ‘‘structure learning’’ by automatically picking up novel contingencies between sensory signals, thus automatically recruiting signals for novel uses during the construction of a percept. no notspecified http://www.kyb.tuebingen.mpg.de/ published 3 15017 15422 15017 18824 5262 3 M Di Luca F Domini C Caudek 2010-05-00 16 50 1519 1531 Vision Research Perceived three-dimensional shape can be systematically distorted by altering viewing conditions. Here we investigate whether such distortions are inconsistent or congruent among different surface properties (relative depth, orientation, and curvature). Participants matched two surfaces defined by different types of visual information (motion parallax, texture, or shading). Judgments of different surface properties elicited qualitatively different patterns of distortions. Adding multiple sources of information did not eliminate inconsistencies. This result indicates that 3D shape perception is not based on Euclidean representation. Moreover, models of cue integration that assume interpretation of information prior to fusion cannot account for such findings. no notspecified http://www.kyb.tuebingen.mpg.de/ published 12 15017 15422 15017 18824 5774 3 PW Battaglia M Di Luca MO Ernst PR Schrater T Machulla D Kersten 2010-03-00 3 6 1 10 PLoS Computational Biology Perception is fundamentally underconstrained because different combinations of object properties can generate the same sensory information. To disambiguate sensory information into estimates of scene properties, our brains incorporate prior knowledge and additional “auxiliary” (i.e., not directly relevant to desired scene property) sensory information to constrain perceptual interpretations. For example, knowing the distance to an object helps in perceiving its size. The literature contains few demonstrations of the use of prior knowledge and auxiliary information in combined visual and haptic disambiguation and almost no examination of haptic disambiguation of vision beyond “bistable” stimuli. Previous studies have reported humans integrate multiple unambiguous sensations to perceive single, continuous object properties, like size or position. Here we test whether humans use visual and haptic information, individually and jointly, to disambiguate size from distance. We presented participants with a ball moving in depth with a changing diameter. Because no unambiguous distance information is available under monocular viewing, participants rely on prior assumptions about the ball's distance to disambiguate their -size percept. Presenting auxiliary binocular and/or haptic distance information augments participants' prior distance assumptions and improves their size judgment accuracy—though binocular cues were trusted more than haptic. Our results suggest both visual and haptic distance information disambiguate size perception, and we interpret these results in the context of probabilistic perceptual reasoning. no notspecified http://www.kyb.tuebingen.mpg.de/ published 9 15017 18824 5870 3 M Di Luca T-K Machulla MO Ernst 2009-12-00 12:7 9 1 16 Journal of Vision After exposure to asynchronous sound and light stimuli, perceived audio-visual synchrony changes to compensate for the asynchrony. Here we investigate to what extent this audio-visual recalibration effect transfers to visual-tactile and audiotactile simultaneity perception in order to infer the mechanisms responsible for temporal recalibration. Results indicate that audio-visual recalibration of simultaneity can transfer to audio-tactile and visual-tactile stimuli depending on the way in which the multisensory stimuli are presented. With presentation of co-located multisensory stimuli, we found a change in the perceptual latency of the visual stimuli. Presenting auditory stimuli through headphones, on the other hand, induced a change in the perceptual latency of the auditory stimuli. We argue that the difference in transfer depends on the relative trust in the auditory and visual estimates. Interestingly, these findings were confirmed by showing that audio-visual recalibration influences simple reaction time to visual and auditory stimuli. Presenting co-located stimuli during asynchronous exposure induced a change in reaction time to visual stimuli, while with headphones the change in reaction time occurred for the auditory stimuli. These results indicate that the perceptual latency is altered with repeated exposure to asynchronous audio-visual stimuli in order to compensate (at least in part) for the presented asynchrony. no notspecified http://www.kyb.tuebingen.mpg.de/ published 15 15017 18824 4189 3 M Di Luca F Domini C Caudek 2007-05-00 10 47 1335 1349 Vision Research In two experiments, observers were asked to judge the relative depth of a probe and one or two flanker dots. In Experiment 1, we found that such judgments were influenced by the properties of adjacent image regions, that is, by the amount of angular rotation of a surrounding cloud of dots. In Experiment 2, we found that the properties of the adjacent image regions affected the precision of the observers’ judgments. With only the probe and the flanker dots presented in isolation, the precision of observers’ judgments was much lower than when probe and the flanker dots were surrounded by a rigidly-connected cloud of dots. Conversely, a non-rigid rotation of the surrounding dots was detrimental to the precision of visual performance. These data can be accounted for by the Intrinsic Constraint model [Domini, F., Caudek, C., & Tassinari, H. (2006). Stereo and motion information are not independently processed by the visual system. Vision Research, 46, 1707–1723], which incorporates the mu tual constraints relating disparity and motion signals. The present investigation does not show that the rigidity constraint affects the visual interpretation of motion information alone. Rather, our results show that perceptual performance is affected by the linear relation between disparity and velocity signals, when both depth-cues are present and the distal object is, in fact, rigid. no notspecified http://www.kyb.tuebingen.mpg.de/ published 14 15017 15422 4127 3 M Di Luca F Domini C Caudek 2004-12-00 26 44 3001 3013 Vision Research In three experiments we investigated whether the perception of 3D structure from the optic-flow involves a process of spatial integration. The observer‘s task was to judge the 3D orientation of local velocity field patches. In two conditions, the patches were presented either in isolation, or as part of a global optic-flow. In Experiment 1, the global optic-flow was a linear velocity field. In Experiment 2, the patches were embedded in a randomly perturbed linear velocity field. In Experiment 3, the local patches belonged to a smoothly curved surface. The results of these three experiments lead to two main conclusions: (1) a process linking spatially separated patches into global entities does affect the perception of local surface orientation induced by the optic-flow, and (2) linearity or smoothness of the global velocity field are not necessary conditions for spatial integration. no notspecified http://www.kyb.tuebingen.mpg.de/ published 12 4128 3 C Caudek F Domini M Di Luca 2002-05-00 10 42 1213 1223 Vision Research Temporal integration was investigated in the minimal conditions necessary to perform a structure-from-motion (SFM) task. Observers were asked to discriminate three-dimensional (3D) surface orientations in conditions in which the stimulus displays simulated velocity fields providing, in each frame transition, either sufficient (3 moving dots) or insufficient information (1 or 2 moving dots) to perform the task. When only two moving dots were shown in each frame transition of the stimulus displays (Experiment 1), we found that performance decreased as dot-lifetime increased. A facilitation effect of the overall display duration was also found. The negative effect of dot-lifetime on performance contrasts with what found in Experiment 2 with three dots in each frame transition, where performance improved with increasing dot-lifetime up to 170 ms, and then reached a plateau. Finally, for an optimal dot-lifetime of 150 ms, we found that performance was still above chance when each frame transition specified the mot ion of only one dot (Experiment 3). These results indicate that temporal recruitment alone can support the recovery of 3D information from sparse motion signals, thus providing a strong indication for the importance of temporal integration in the perceptual analysis of the optic flow. Our results reveal, moreover, that temporal integration in SFM has different characteristics, depending on whether, in each frame transition, the stimulus displays provide either sufficient (3 or more moving dots) or insufficient information (1 or 2 moving dots) to specify the higher-order properties of the optic flow necessary for 3D surface recovery. FP6 EC project ImmerSence (IST-2006-02714) no notspecified http://www.kyb.tuebingen.mpg.de/ published 10 4129 3 C Caudek F Domini M Di Luca 2002-04-00 3 64 366 379 Perception and Psychophysics Observers‘ perceptions of the three-dimensional structure of smoothly curved surfaces defined by patterns of image shading were investigated under varying conditions of illumination. In five experiments, observers judged the global orientation and the motion of the simulated surfaces from both static and dynamic patterns of image shading. We found that perceptual performance was more accurate with static than with dynamic displays. Dynamic displays evoked systematic biases in perceptual performance when the surface and the illumination source were simulated as rotating in opposite directions. In these conditions, the surface was incorrectly perceived as rotating in the same direction as the illumination source. Conversely, the orientation of the simulated surfaces was perceived correctly when the frames making up the apparent-motion sequences of the dynamic displays were presented as static images. In Experiment 6, moreover, the results obtained with the computer-generated displays we re r eplicated with solid objects. no notspecified http://www.kyb.tuebingen.mpg.de/ published 13 5910 7 B Knörlein M Di Luca M Harders Orlando, FL, USA2009-10-00 49 52 8th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2009) Visual delays are unavoidable in augmented reality setups and occur in different steps of the rendering pipeline. In the context of haptic interaction with virtual objects, it has been shown that delayed force feedback can alter the perception of object stiffness. We hypothesize that delays in augmented reality systems can have similar consequences. To test this, we carried out a user study to investigate the effect of visual and haptic delays on the perception of stiffness. The experiment has been performed in an optimized visuo-haptic augmented reality setup, which allows to artificially manipulate delays during visual and haptic rendering. In line with previous results, delays for haptic feedback resulted in decreased perceived stiffness. In contrast, visual delays caused an increase in perceived stiffness. However, the simultaneous occurrence of delays in both sensory channels led to a partial compensation of these effects. This could potentially help to correct stiffness perception of virtual objects in visuo-haptic augmented reality systems. no notspecified http://www.kyb.tuebingen.mpg.de//fileadmin/user_upload/files/publications/Knoerlein_DiLuca_Harders_2009_5910[0].pdf published 3 15017 18824 5624 7 R Höver M Di Luca G Szekely M Harders Salt Lake City, UT, USA2009-03-00 39 44 Third Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC 2009) Data-driven haptic rendering requires processing of raw recorded signals, which leads to high computational effort for large datasets. To achieve real-time performance, one possibility is to reduce the parameter space of the employed interpolation technique, which generally decreases the accuracy in the rendering. In this paper, we propose a method for guiding this parameter reduction to maintain high accuracy with respect to the just noticeable difference for forces. To this end, we performed a user study to estimate this perception threshold. The threshold is used to assess the final error in the rendered forces as well as for the parameter reduction process. Comparison with measured data from real object interactions confirms the accuracy of our method and highlights the reduced computational effort. no notspecified http://www.kyb.tuebingen.mpg.de/ published 5 15017 1542215017 18824 5026 7 R Groten J Hölldampf M Di Luca M Ernst M Buss Madrid, Spain2008-06-00 838 843 6th International Conference EuroHaptics 2008 In this work, we analyze whether oscillatory motion between two extreme positions could be used to create a robotic dancing partner that provides natural haptic feedback. To this end, we compared the pattern of hand movements performed following a pacing signal while participants were instructed to either move rhythmically or to dance. Furthermore, we analyzed the influence of the frequency and type of pacing signal on the two kinds of movements. Trajectories were analyzed in terms of: frequency of movement, spatial and temporal synchronization, and jerk. Results indicate that it is easier to perform synchronized movements while dancing, even though these movements partially deviate from the pacing frequency. Dance movements are in fact more complex than the ones produced to keep the rhythm and for this reason they should be modeled accordingly in order to provide realistic haptic feedback. no notspecified http://www.kyb.tuebingen.mpg.de/ published 5 15017 1542215017 18824 MohlerdB2012 2 BJ Mohler M Di Luca HH Bülthoff American Psychological Association Washington, DC, USA 2012-10-00 81 97 Handbook of Spatial Cognition no notspecified http://www.kyb.tuebingen.mpg.de/ published 16 15017 15422 ErnstDM2011 2 M Ernst M Di Luca Oxford University Press New York, NY, USA 2011-10-00 225 250 Sensory Cue Integration The brain receives information about the environment from all the sensory modalities, including vision, touch and audition. To efficiently interact with the environment, this information must eventually converge in the brain in order to form a reliable and accurate multimodal percept. This process is often complicated by the existence of noise at every level of signal processing, which makes the sensory information derived from the world imprecise and potentially inaccurate. There are several ways in which the nervous system may minimize the negative consequences of noise in terms of precision and accuracy. Two key strategies are to combine redundant sensory estimates and to utilize acquired knowledge about the statistical regularities of different sensory signals. In this lecture, I elaborate on how these strategies may be used by the nervous system in order to obtain the best possible estimates from noisy sensory signals, such that we are able of efficiently interact with the environment. Particularly, I will focus on the learning aspects and how our perceptions are tuned to the statistical regularities of an ever changing environment. no notspecified http://www.kyb.tuebingen.mpg.de/ published 25 15017 1542215017 18824 diLucaMBE2011 7 M Di Luca T Machulla M Barnett-Cowan MO Ernst Tübingen, Germany2011-09-00 Bernstein Cluster D Symposium: Multisensory Perception and Action no notspecified http://www.kyb.tuebingen.mpg.de/ published 0 15017 1542215017 18824 6419 7 M Di Luca M Barnett-Cowan Lausanne, Switzerland2010-08-00 187 33rd European Conference on Visual Perception Prolonged exposure to asynchronous audiovisual stimulus pairs changes the perception of audiovisual simultaneity. It has been proposed that this change occurs by adjusting the perceptual latency of stimuli in order to minimize perceived audiovisual asynchrony. How is this adjustment achieved? For signals with a gradual onset, perceptual latency can be minimized by decreasing detection threshold (or vice-versa). Here we assess whether this occurs following recalibration of simultaneity. Participants were exposed for 5 minutes to asynchronous (150ms) audiovisual stimulus pairs with either light or sound leading. Auditory stimuli were presented via headphones, visual via an LED. Detection thresholds for visual and auditory stimuli were then measured with a 2IFC task interleaved with short re-exposures to the asynchrony. Results indicate that while the detection threshold for visual stimuli does not significantly vary, the detection threshold for auditory stimuli critically depends on which modality leads during asynchronous audiovisual exposure. All nine participants tested were more sensitive in detecting auditory stimuli after light-leading exposure than after sound-leading. We suggest that by becoming more or less sensitive to sound the brain is able to change the perceptual latency of auditory stimuli to minimize audiovisual asynchrony, while keeping the perceptual latency of visual stimuli relatively constant. no notspecified http://www.kyb.tuebingen.mpg.de/ published -187 15017 1542215017 18824 6498 7 M Di Luca MO Ernst BT Backus Brussels, Belgium2010-06-00 8 9 4th Annual Meeting of the Association for the Scientific Study of Consciousness (ASSC 4) The perceptual appearance of a visual stimulus can be changed by presenting stimuli that are similar, but that differ along specific dimensions, to the observer in advance. Many negative adaptation aftereffects are familiar to students of perception, for example. A different example is “cue recruitment” (Haijiang et al., 2006): a visual signal that has no effect on some attribute of appearance can often be made to affect that attribute through the use of classical (Pavlovian) conditioning procedures. In that case, the signal has come to be treated as a new cue by the visual system, insofar as it now participates in the construction of some new aspect of appearance that it previously did not. We asked whether this learning requires that the signal be visible, i.e. whether it must have a consciously accessible perceptual consequence, of any sort, during training. To do this we employed an invisible visual signal, namely, a vertical gradient of vertical disparity obtained by slightly magnifying the image in one eye. This signal is measured by the visual system, but it had no influence on any of the perceptual attributes that observers’ visual systems computed from the displays, in which horizontal lines depicted a rotating cylinder. During training we made the eye of vertical magnification (EVM) contingent on the rotation direction of the cylinder. After training we presented an ambiguous version of the cylinder and found that EVM influenced the perceived direction of rotation consistent with contingency during training. Thus, a signal need not be visible for the adult visual system to give it new use as a participant in the construction of visual appearances. Haijiang, Q., Saunders, J. A., Stone, R. W., & Backus, B. T. (2006). Demonstration of cue recruitment: Change in visual appearance by means of Pavlovian conditioning. PNAS, 103, 483–486. no notspecified http://www.kyb.tuebingen.mpg.de/ published 1 15017 1542215017 18824 6497 7 CV Parise M Di Luca MO Ernst Liverpool, UK2010-06-00 11th International Multisensory Research Forum (IMRF 2010) Perceptual judgments are classically regarded as involving both a sensory and a decisional component. An optimal observer engaged in a signal detection task should maximize the correct responses by setting an appropriate decisional criterion according to his/her sensitivity to sensory signals. A large body of literature supports the view that observers set a near-optimal criterion in the detection of a single signal. However, there seems to be a systematic deviation from optimality when observers are required to concurrently judge multiple signals within the visual modality. In this case, observers set only a single criterion for the joint presentation of the compound signals (Gorea & Sagi 2000). This result has been interpreted as an indication that humans are unable to simultaneously handle multiple signal representations within a sensory modality. Literature, however, remains silent as to the crossmodal case. In a 2IFC task we asked participants to concurrently report whether or not visual and/or auditory stimuli changed position within each trial. Our results show that with multisensory signals observers can simultaneously set separate criteria for each modality. It is still an open question whether separate criteria are also set when multisensory signals are integrated. no notspecified http://www.kyb.tuebingen.mpg.de/ published 0 15017 1882415017 15422 6660 7 J Hartcher-O‘Brien S Telgen M Di Luca MO Ernst Liverpool, UK2010-06-00 11th International Multisensory Research Forum (IMRF 2010) Perceived time is not veridical but distorted and differs across the senses. Here we ask, which points in the perception of a temporal event contribute most significantly to these multisensory distortions? To this end, we investigated perceptual estimates of temporal landmarks (onset, peak amplitude, and offset) for a Gaussian standard signal (s=150ms). Particularly we were interested in how the perception of these landmarks differ across vision and audition. Participants undertook a temporal order judgment task comparing the onset, peak and offset landmarks of the standard stimuli to short spike-like stimuli (s=5ms) in vision or audition. All four combinations were tested: V-v, A-a, V-a, A-v. Results demonstrate that the visual as compared to auditory standard stimuli were perceived shorter. More interestingly, we found a compression in the perceived duration for onset-peak intervals compared to peak-offset intervals. This compression effect was more pronounced in the visual modality. Discrimination threshol ds were worse for offset judgements compared to onset or peak judgements in both modalities. The differences in perceived duration can potentially be used to explain multisensory illusions such as the flash lag effect and perceived crossmodal asynchronies. We quantitatively explain these distortion effects using models of signal processing. no notspecified http://www.kyb.tuebingen.mpg.de/ published 0 15017 1542215017 18824 6490 7 J Hartcher-O'Brien M Di Luca MO Ernst Naples, FL, USA2010-05-00 1408 10th Annual Meeting of the Vision Sciences Society (VSS 2010) Despite continual temporal discrepancies between sensory inputs, signals arising from the same event are bound together into a coherent percept. It has been suggested that multiple timekeepers monitor the different sensory streams, producing differences in perceived duration of events. Given this, what is the integration strategy adopted for combining sensory information in the time domain? Specifically, if the brain has information about the duration of an event from more than one source, can the uncertainty of the duration estimate decrease, and can the Maximum Likelihood Estimate (MLE) model predict such a change? Using a 2AFC procedure, participants had to judge which interval was longer (1st or 2nd) for auditory, visual and audiovisual stimuli. Each trial contained 2 intervals: a standard stimulus (sampled from one of three durations), and a comparison interval whose duration changed randomly in relation to standard stimulus duration. The reliability of the auditory stimulus was manipulated to produce the unimodal weighting scheme. Data was fit with a cumulative Gaussian psychometric function from which PSE and JND were extracted. Results for unimodal trials showed JND changes that depended upon the duration of the standard, according to Weber's law. JND values also decreased with decreases in signal noise. Comparison of the present bimodal results with MLE predictions revealed optimal integration of auditory and visual duration cues. Additionally the results show that the integration of uncertain visual and auditory duration signals is a weighted average of these signals. That is, PSE shifts in perceived duration tended to reflect MLE predictions with shifts following the more reliable unimodal signal. These results are the first to demonstrate ‘optimal’ integration of sensory information in the time domain and contradict other studies applying MLE to this stimulus feature. no notspecified http://www.kyb.tuebingen.mpg.de/ published -1408 15017 1542215017 18824 6488 7 T Machulla M Di Luca MO Ernst Naples, FL, USA2010-05-00 1414 10th Annual Meeting of the Vision Sciences Society (VSS 2010) Recent studies have investigated adaptation to temporal discrepancies between different sensory modalities by first exposing participants to asynchronous multisensory signals, and subsequently assessing the magnitude of the adaptation effect (the size of the shift in subjective simultaneity). Although never reported, there is reason to assume that the strength of the adaptation effect declines during this measurement period. Usually, short re-exposures are interleaved with testing to prevent such declining. In the present study, we show that a decrease in the strength of adaptation still can take place, even when a common re-exposure procedure is used. In a second experiment, we investigated whether the observed decline is due to: (1) a dissipation of adaptation with the passage of time or, (2) a new adaptation induced by the test stimuli. We find that temporal adaptation does not dissipate with time but is stored until new sensory information, i.e., stimuli that differ from those used during the adaptation procedure, is presented. An alternative explanation, namely that adaptation decays over time but is re-established before the first test trial due to the experimental procedure we chose, is addressed in a control experiment. This finding is discussed in terms of Helson's adaptation level (AL) theory [1947, Adaptation-level as frame of reference for prediction of psychophysical data. The American Journal of Psychology, 60, 1–29], according to which the null point of any perceptual dimension, in our case the perception of simultaneity on the dimension of temporal order, is a summarizing statistic of all stimuli presented in the past. Any single stimulus pulls the AL toward its own value, and any single stimulus is judged as though it was being compared with the current AL. no notspecified http://www.kyb.tuebingen.mpg.de/ published -1414 15017 1542215017 18824 6489 7 M Di Luca B Knörlein M Harders MO Ernst Naples, FL, USA2010-05-00 852 10th Annual Meeting of the Vision Sciences Society (VSS 2010) Compliance of deformable materials is perceived through signals about resistive force and displacement. During visual-haptic interactions, visual and proprioceptive signals about material displacement are combined over time with the force signal. Here we asked whether multisensory compliance perception is affected by the timing of signals by introducing an asynchrony between the participant's movement (sensed proprioceptively) and force information or visual information. Visual-proprioceptive asynchronies are obtained by making participants see a delayed video of their haptic interaction with an object rather than the real interaction. Force-proprioceptive asynchronies are instead obtained by making participants compress a virtual object with their hand and sense the resistive force generated by a force feedback device. Results indicate that force-proprioceptive asynchronies can significantly alter the perception of object stiffness. Moreover, we find that perceived compliance changes also as a function of the delay of visual information. These effects of asynchrony on perceived compliance would not be present if all force-displacement information would be utilized equally over time, as both delays generate a bias in compliance which is opposite in the compression and release phases of the interaction. To explain these findings we hypothesized instead that information during object compression is weighted more than information obtained during object release and that visual and proprioceptive information about the hand position are used for compliance perception depending on the relative reliability of the estimate obtained. We confirm these hypotheses by showing that sensitivity to compliance is much higher during object compression and that degradation of visual and proprioceptive information can modify the weights assigned to the two sources. Moreover, by analyzing participants' movements and feedback forces we show that the two hypothesized factors (compression-release and visual-proprioceptive reliability) can account for the change in perceived compliance due to force-proprioceptive and force-displacement asynchronies. no notspecified http://www.kyb.tuebingen.mpg.de/ published -852 15017 1542215017 18824 5759 7 MO Ernst M Di Luca B Backus Naples, FL, USA2009-08-00 34 9th Annual Meeting of the Vision Sciences Society (VSS 2009) Cue recruitment occurs when a sensory signal is put into correlation with trusted cues and subsequently influences perceptual interpretation as the trusted cues do. In all cue recruitment experiments to date, the signal has been well above detection threshold and was easily visible. For example, it has been shown that object position and motion can be recruited as a cue to influence the interpretation of the ambiguous Necker Cube (Haijiang et al., 2006). Here we asked whether a signal that is not visible on its own could be recruited as a cue. Vertical size ratio (VSR, the ratio of vertical angles subtended by an object at the two eyes) is normally used to correct for relative position of the head when interpreting horizontal disparity (Gillam & Lawergren, 1983; Backus et al 1999) but it is not visible in displays consisting of horizontal lines only because there are no horizontal discontinuities. We manipulated VSR of displays simulating a cylinder composed of horizontal lines that rotated about a horizontal axis. On training trials, the rotation direction of the cylinder was unambiguously specified by horizontal disparity and occlusion cues and these trusted cues were correlated with the VSR cue to be recruited. On test trials, the display did not contain horizontal disparity or occlusion, so that the rotation direction specified by the trusted cues was ambiguous. If participants however made use of the VSR cue in the test display rotation direction could become unambiguous after training. For 8 out of 9 participants, apparent rotation on test trials became contingent on the value of VSR. We conclude that a signal need not have perceptual consequences by itself for the system to assign it a new use during the construction of appearances. no notspecified http://www.kyb.tuebingen.mpg.de/ published -34 15017 18824 5770 7 M Rohde M Di Luca MO Ernst Naples, FL, USA2009-08-00 712 9th Annual Meeting of the Vision Sciences Society (VSS 2009) In the well-known rubber hand illusion (RHI, Botvinick & Cohen, Nature, 1998), synchronous tactile stimulation of a subject's invisible hand and a visible rubber hand (placed at a fixed lateral distance) induces a drift of perceived location of the real hand towards the rubber hand. The purpose of this study was to investigate the spatio-temporal characteristics of this proprioceptive drift. To this end, we measured the perceived position of the participants' index finger in relation to a small visible probe using an adaptive two-staircase method and a forced-choice task. We determined perceived finger location by fitting a psychometric function to the responses. Prior to introducing the rubber hand at a distance of 17cm from the real hand, we determined the perceived finger location in darkness. The time course of the RHI was then determined in three phases: pre-test while only looking at the rubber hand, prolonged synchronous tactile stimulation, and post-test again without touch but with visible rubber hand. The synchronous stimulation was fully controlled using two PHANToM force-feedback devices. The perceived finger location immediately shifted 1.4cm towards the rubber hand when in view. This shift rose to an average of 6.3cm after 8min of tactile stimulation. The distribution of responses indicates that the proprioceptive drift is truly gradual. As previous work suggests (cf.Holmes et al., P&P, 2006; Tsakiris & Haggard, JEP, 2005), these findings show that the RHI involves both immediate effects that result from multisensory integration as well as gradual recalibration effects with a time constant of several minutes. After 5mins of post-test, a drift of 4.9cm remained, showing that recalibration produces an after-effect with respect to both the baseline measure recorded in darkness and the pre-test. We will further investigate the determinants of the gradual build up and decay of the drift. no notspecified http://www.kyb.tuebingen.mpg.de/ published -712 15017 18824 6492 7 J Hölldampf A Peers M Di Luca M Buss Barcelona, Spain2009-03-00 Real Actions in Virtual Environments (RAVE 2009) no notspecified http://www.kyb.tuebingen.mpg.de/ published 0 15017 18824 5697 7 M Di Luca D Knill M Ernst Rauischholzhausen, Germany2008-10-00 Workshop "Cue combination - Unifying Perceptual Theory" Multiple cues are integrated by weighting individual cue estimates according to their reliabilities. This has been shown repeatedly under static, prolonged viewing conditions, but in real-world situations viewing conditions may change and the reliability of cues can vary over time. Here we ask whether the perceptual system instantaneously detects reliability changes and adapts the cue weights accordingly. Subjects binocularly viewed a spinning surface slanted in depth. In the display there was a slight discrepancy between the slant specified by the stereoscopic cue and the one specified by motion. We modulated the reliability of the motion cue by changing the speed of rotation. Increasing the speed of rotation increased the reliability of the motion cue and consequently biased the perceived slant towards the slant defined by the motion cue. This manipulation, however, did not affect the magnitude of slant specified by the motion cue. By modulating the rotation speed at different frequencies the surface was perceived to oscillate in depth. Perceived oscillation amplitude decreased with higher modulation frequency and phase shift between rotation modulation and perceived oscillation increased. As a control, we simulated an actual oscillation of the surface and we did not obtain such response pattern. This result indicates that quick changes of reliability, defined by changes in rotation speed, have smaller perceptual consequences. Cue weights do not change instantaneously, likely because there is latency in updating the reliability of the motion cue when the spinning surface changes its speed of rotation no notspecified http://www.kyb.tuebingen.mpg.de/fileadmin/user_upload/files/publications/DiLucaKnillErnst.pdf published 0 15017 1542215017 18824 5024 7 M Di Luca M Ernst W Adams Naples, FL, USA2008-06-00 526 8th Annual Meeting of the Vision Sciences Society (VSS 2008) Recently it has been shown that congruent visual and haptic signals are integrated in a statistically optimal fashion. Spatial separation between the signals can preclude this integration. Here we investigate whether optimal integration occurs between an amodally completed visual stimulus and its haptic counterpart. Thus, we ask whether integration occurs despite the sensory information not being derived from the same spatial location. This may indicate that subjects inferred that the visually specified parts of the stimulus and the haptic information have a common cause and thus should be integrated. The visual stimulus was a disparity-defined bar that was partially occluded (amodal completion condition). The bar could also be touched behind the occluder using two fingers. Subjects' task was to discriminate the size of two successively presented bars using a 2-IFC paradigm, where one interval contained conflicting haptic and visual information. Performance in the amodal completion condition was not different from a condition in which the occluder was removed (visual-haptic condition). Both conditions were consistent with an optimal integration strategy. More interestingly, integration deviated from optimality when we introduced a slight modification to the visual stimulus - small gaps between the bar and the occluder (gap condition). This manipulation interfered with the amodal completion process and consequently subjects relied almost completely on the haptic information for discriminating the size of the bars. These findings suggest that visual and haptic information can be combined optimally even when visual information is not directly specified by sensory information, but results from amodal completion. In conclusion, it seems that the perceptual system determines when to combine visual and haptic information based on the likelihood the signals have of belonging to the same object (i.e. if there is a causal relationship between the signals) and not only on signal co-location. no notspecified http://www.kyb.tuebingen.mpg.de/ published -526 15017 1542215017 18824 5172 7 P Battaglia M Ernst P Schrater M Di Luca T Machulla D Kersten Naples, FL, USA2008-06-00 1090 8th Annual Meeting of the Vision Sciences Society (VSS 2008) An object's visual image size is an ambiguous cue to its physical size. But if the object's distance is known, the physical size may be disambiguated and more accurately estimated. We asked whether humans use distance cues to improve size judgments. We presented participants with a virtual ball that changed in physical size (imagine a balloon inflating or deflating) as well as distance simultaneously, and asked them to discriminate whether the physical size increased or decreased. With only visual image size information, size-change discrimination was poor. When additional haptic and/or stereo distance-change cues were provided, size-change judgments improved significantly. We conclude that the brain exploits its knowledge of how image size, physical size, and distance are related to improve perceptual size judgments. We compared participants' use of distance cues with predictions of an ideal observer that incorporates distance cues in proportion to their reliability to quantify human behavior. We independently measured participants' stereo and haptic distance discrimination performance, applied these empirical reliability measurements in the ideal model, and found participants use stereo information to a similar degree as the ideal observer, but use haptic information less than the ideal observer. This result was confirmed by an additional conflict condition in which haptic and stereo distance-change cues indicated different values and their relative use could be measured. Lastly, we ran a condition in which participants gripped the object with two fingers, so that a direct size-change cue was available, and found participants integrated direct and indirect size-change cues to improve performance. no notspecified http://www.kyb.tuebingen.mpg.de/ published -1090 15017 1542215017 18824 5025 7 R Groten J Hölldampf M Di Luca M Buss Barcelona, Spain2008-02-27 Real Action, Virtual Environments (RAVE-08) no notspecified http://www.kyb.tuebingen.mpg.de/ published 0 15017 1542215017 18824 5034 7 MO Ernst M Di Luca Arezzo, Italy2007-08-00 172 30th European Conference on Visual Perception Multimodal (visual & auditory) information is combined by weighting unimodal signals by their relative reliabilities. Reliability in real-world stimuli, however, does not remain constant. Here we ask whether the system adjusts the weights online in the McGurk effect. Subjects were presented with an audiovisual recording of an actor‘s face producing a series of syllables. Such syllables were composed either by consistent multimodal information, or by auditory /ba/ and lip movement of /ga/. Subjects‘ task was to continuously report the perceived syllable. We varied the reliability of the visual information by changing the visibility of the face. With inconsistent multimodal information, increased reliability of the visual signal biased perception towards the illusory /da/ percept whereas decreased reliability biased perception towards the auditory /ba/. However, changes in reliability had perceptual consequences only after 3 - 4 s. Therefore, the reliability estimate of the sensory signals is not instantaneous but continuously updated with a time-constant of a few seconds. no notspecified http://www.kyb.tuebingen.mpg.de/ published -172 15017 1542215017 18824 4522 7 T Machulla M Di Luca M Ernst Arezzo, Italy2007-08-00 209 30th European Conference on Visual Perception When signals in different modalities are physically asynchronous, observers may still perceive them as being simultaneous due to differential physical transmission and physiological conduction delays. If sensory signals in different modalities are processed independently of each other as assumed by independent-channels models, then the relative timings that lead to perceived simultaneity should be transitive across several modality pairs. For instance, if modality A has to be presented 20 ms before modality B to seem simultaneous with it and modality B 10 ms before modality C, then A should be presented 30 ms before C to seem simultaneous with it. Using temporal order judgments we measured the point of subjective simultaneity (PSS) in three different modality pairs (visual-auditory, tactile-auditory, visual-tactile). Our results indicate that PSS are not transitive. Thus, we infer that signals are not processed independently from each other. Perceived signal timing in one modality depends on which other modality it is paired with. Therefore, independent-channels models cannot account for processes underlying decisions about simultaneity of signals in different modalities. no notspecified http://www.kyb.tuebingen.mpg.de/ published -209 15017 1542215017 18824 4519 7 M Di Luca T Machulla M Ernst Arezzo, Italy2007-08-00 207 30th European Conference on Visual Perception Repeated presentations of asynchronous audiovisual signals recalibrate the point of subjective simultaneity. It is not clear whether this effect is the result of an adaptation mechanism specific to the audiovisual modality pair or whether it is due to a mechanism common to all modalities. Only in the latter case we expect that repeated asynchronous audiovisual information influences perceived simultaneity in other modality pairs (audiotactile or visuotactile). We presented a series of asynchronous audiovisual signals to the participants (SOA: 200ms, -200ms) and then estimated the point of subjective simultaneity for three modality pairs (audiovisual, audiotactile, visuotactile). Consistent with previous research, perceived simultaneity in the audiovisual modality pair changed for the two SOAs. Subjective simultaneity shifted also in the audiotactile modality pair. Hence, we conclude that the three tested modalities share a common adaptation mechanism. Moreover, since the visuotactile modality pair was not significantly affected by the manipulation, audiovisual adaptation is likely the result of a phenomenal shift of the auditory events in time. no notspecified http://www.kyb.tuebingen.mpg.de/ published -207 15017 1542215017 18824 4521 7 TK Machulla M Di Luca MO Ernst Tübingen, Germany2007-07-00 85 10th Tübinger Wahrnehmungskonferenz (TWK 2007) Sensory processing times can greatly differ between the senses. Hence, signals from different modalities that are presented with a delay corresponding to the processing time difference between these modalities appear simultaneous to the observer. We hypothesized that if processing is independent for each modality and if there is a common mechanism for the perception of simultaneity across modalities then subjective simultaneity should be transitive. For example, if modality A has to be presented 20ms before B and modality B 10ms before C to be perceived as synchronous, then modality A should be presented 30ms before C to seem synchronous with it. Observers judged the temporal order of three different modality pairs (visual-tactile, tactile-auditory, and visual-auditory) for eleven stimulus onset asynchronies. Stimuli from the three conditions were not blocked but presented randomly to prevent attentional prior-entry effects that might lead to artifactual intransitivity. From the responses, we determined the presentation delay leading to subjective simultaneity. To appear synchronous the visual signal has to be presented 34ms before the tactile, the tactile 55ms before the auditory, and the visual 28ms before the auditory. These results deviate significantly from transitivity. We conclude that either stimulus processing time in one modality depends on which other modality it is paired with, or the notion of a common mechanism for crossmodal simultaneity has to be rejected. no notspecified http://www.kyb.tuebingen.mpg.de/ published -85 15017 1542215017 18824 4563 7 KM Mayer M Di Luca MO Ernst Tübingen, Germany2007-07-00 68 10th Tübinger Wahrnehmungskonferenz (TWK 2007) Sensory modalities do not act in isolation; they can influence one another. One intriguing example is temporal ventriloquism, where auditory and visual signals presented asynchronously appear to be closer in time than they are generated. The characteristics of this perceptual phenomenon are still not completely defined. In particular, it is not clear whether the order of presentation has an influence on the effect, whether there are temporal grouping effects for other modalities, and whether this effect only occurs for certain asynchronies of the signals. This study aims to fill this lack of knowledge by investigating how crossmodal temporal grouping affects perceived duration when onset and offset of a stimulus are defined by signals of different modalities in all combinations of auditory, visual, and tactile stimuli. To test this, a two interval forced choice (2IFC) paradigm was used, where participants had to judge which of two intervals was shorter. One interval consisted of a lasting sound (filled interval), the other interval was defined by two signals of different modalities at the onset and offset (empty interval). These two signals could be auditory (beep), visual (flash of a LED) or tactile (vibration on the participants’ left index finger). Different stimulus onset asynchronies (SOA) between 100 and 900ms of the crossmodal interval were tested. Moreover, the order of the signals forming the empty interval was reversed. The duration of the filled interval was 30 to 170% of the duration of the empty interval. For each SOA of the empty interval, the point of subjective equality (PSE) for the duration was computed, which is the duration of the filled interval perceived to be equal to the empty interval. Results confirmed that for audiovisual intervals grouping effects were more salient with larger SOAs between the signals defining the empty interval. When the light preceded the sound grouping effects were also found for 500 and 700ms. For audiotactile intervals, instead, grouping effects occurred between 500 and 900ms but only when vibration preceded sound. Lastly, for visuotactile intervals, grouping occurred at SOA of 500ms but only when light preceded vibration. From these results, we can conclude that audiovisual temporal ventriloquist is only an instance of a more general crossmodal grouping effect that occurs with various modalities. Moreover, our data indicates that this effect acts differently on specific modality combinations. Temporal grouping is affected by both the time between the signals and their order of presentation. no notspecified http://www.kyb.tuebingen.mpg.de/ published -68 15017 1542215017 18824 4518 7 M Di Luca T Machulla M Ernst Sydney, Australia2007-07-00 15 International Intersensory Research Symposium 2007: Perception and Action Crossmodal stimuli can be perceived as being simultaneous even if they are not physically synchronous. This phenomenon has been attributed to different conduction delays. In this work we tested whether time in different modalities is processed independently or if crossmodal interaction influence the perception of synchrony. (1) If unimodal timing is processed independently, perceived simultaneity across modality pairs should be Transitive. For example, if modality A has to be presented 20ms before modality B to appear simultaneous and modality B 10ms before modality C, then A should be presented 30ms before C to appear simultaneous. Subjects made Temporal Order Judgments (TOJ) of asynchronous signals in three modality pairs (audio-visual, audio-tactile, visual-tactile). The Point of Subjective Simultaneity (PSS) calculated for each modality pair are not transitive, indicating that perceived time is not processed independently in each modality. (2) It has been shown that PSS of audio-visual signals can be recalibrated by the repeated presentation of asynchronous stimuli. It is not clear whether this effect is the result of an adaptation mechanism specific to the audio-visual modality pair or whether it is due to a common crossmodal mechanism. Using the same type of measurements, we show that PSS following presentation of an asynchronous audio-visual stimulus is not constant in the audio-tactile modality pair. Hence, crossmodal timing is also affected by a common adaptation mechanism. Since PSS for visual-tactile stimuli was not affected, audio-visual adaptation effects are likely the result of a phenomenal shift of the auditory events in time. Our results indicate that perceived timing in one modality depends on which other modality this is paired with and that perceived simultaneity changes also for non adapted modality pairs. These results are not consistent with independent-channels models of crossmodal timing, but they rather indicate that time perception is affected by crossmodal interactions. no notspecified http://www.kyb.tuebingen.mpg.de/fileadmin/user_upload/files/publications/Perception-Action-Symposium-2007-DiLuca.pdf published -15 15017 1542215017 18824 4684 7 T Machulla M Di Luca M Ernst Santorini, Greece2007-07-00 36 37 1st Peach Summer School Human observers acquire information about physical properties of the environment through different sensory modalities. For natural events, these sensory signals show a specific temporal, spatial and contextual configuration that aids the integration into a coherent multisensory percept. For multimodal virtual environments, however, signals have to be created and displayed separately for different modalities, which may result in a miscalibration of these signals. This, in turn, can greatly reduce the observer’s sense of immersion and presence. Using psychophysical methods, we investigate fundamental questions regarding how the temporal alignment of signals from the visual, auditory and tactile modalities is achieved. A first project examines the perception of subjective simultaneity of signals. Simultaneity detection poses a non-trivial matching problem to the human brain: physical and neural transmission times differ greatly between the senses. As there is only partial compensation for these differential delays, subjective simultaneity may result from presenting stimuli with a physical delay. Here, we are interested in whether this phenomenon reflects an amodal timing mechanism that works across all modalities in a uniform fashion. Further, we look at the sensitivity for asynchrony detection for different modality pairs as well as at interindividual differences. In a second project, we examine the ability of the human cognitive system to adapt to asynchronous information in different modalities. Adaptation may be used to reduce the disruptive effects of temporal miscalibration between signals in different modalities. We are interested in the strength of adaptation as well as the mechanism underlying this effect. Future projects aim at the investigation of - the precise relationship between the perception of synchrony and multimodal integration, - the influence of prior knowledge about a common origin of signals on the perception of synchrony - the influence of timing on the perception of cause and effect - the neural basis of the detection of synchrony In conclusion, our research seeks to understand the mechanisms underlying temporal calibration between different sensory modalities with the goal to identify factors that foster multimodal integration and, in turn, the sense of presence. no notspecified http://www.kyb.tuebingen.mpg.de/ published 1 15017 1542215017 18824 4516 7 M Ernst M Di Luca D Knill Sarasota, FL, USA2007-06-00 92 7th Annual Meeting of the Vision Sciences Society (VSS 2007) Cue integration has been demonstrated to be close to optimal under temporally constant stimulus conditions. That is, cues are assigned different weights according to their relative reliabilities. In real-world situations, however, stimulus conditions constantly change. For example, depending on the viewing situation the reliability of cues may change over time. Here we ask whether the system takes such continuous changes in reliability into account by adjusting the cue weights online. Subjects were binocularly presented with a spinning disk slanted in depth. Thus one cue was disparity, the other motion. There was a ±30 deg conflict between the slants defined by the two cues. We varied the reliability of the motion cue by sinusoidally changing the speed of rotation at different frequencies (0.067, 0.1, 0.2 Hz). Decreasing the speed of rotation decreases the reliability of the motion cue. However, it does not affect the magnitude of slant specified by the motion cue. Subjects task was to continuously adjust the angle of a two-lines probe according to the perceived slant. We found that increasing the motion cue reliability with faster rotations biased perceived slant towards the slant defined by the motion cue. The surface was therefore perceived to oscillate in depth according to the modulation of speed. The oscillation amplitude decreased with higher modulation frequency. The phase shift between rotation modulation and perceived oscillation increased with frequency. As a control, we repeated the task in order to estimate subject's reaction time for adjusting the probe. In this control the slant of the surface was actually oscillating in depth. By subtracting the reaction time from the phase shifts obtained in the experimental conditions we estimated that the time it takes to update the weights is less then a second. no notspecified http://www.kyb.tuebingen.mpg.de/ published -92 15017 1542215017 18824 4515 7 M Di Luca M Ernst Sarasota, FL, USA2007-06-00 90 7th Annual Meeting of the Vision Sciences Society (VSS 2007) Several studies showed that cue integration is close to optimal when two or more cues are available simultaneously. However, most of these studies consider only constant cues. Here we investigate how different depth cues interact when they are not presented simultaneously but they are alternating. We ask whether there is fusion of cues in time and how the interaction between cues depends on the frequency of alternation. To study this, we presented two surfaces in alternation at six different frequencies (from 0.8 to 15 Hz). One surface was defined by a random-dot pattern displayed in stereo (disparity-defined surface); the other was defined by a monocularly viewed regular texture (texture-defined surface). The angle between the two surfaces was always +20 or −20 degrees. Participants had to indicate whether the texture-defined surface was slanted to the left or to the right. The orientation of the two surfaces was varied jointly using a double staircase procedure to find the orientation at which the texture-defined surface appeared frontoparallel. Results indicate that there is a significant interaction between the cues depending on frequency. That is, the orientation of the stimulus needed to see the texture-defined surface as frontoparallel depended on the sign of conflict and the frequency of alternation. At high frequencies (above 6 Hz) there was a perceptual bias of the texture-defined surface in the direction of the disparity-defined surface, indicating integration of the signals. At low frequency (0.8 Hz), however, this interaction did not only disappear, it reversed in the opposite direction, indicating a contrast effect. This contrast may be explained as an aftereffect resulting from adaptation to the disparity-cue slant. We conclude that simultaneity between cues is not necessary for integration to occur. There seems to be a temporal window for integration in the order of 150 ms. no notspecified http://www.kyb.tuebingen.mpg.de/ published -90 15017 1542215017 18824 4633 7 M Di Luca F Domini C Caudek Sarasota, FL, USA2006-06-00 340 6th Annual Meeting of the Vision Sciences Society (VSS 2006) To test whether perceived shape from shading, texture and motion is affine, we asked participants to compare the curvature at the tip of two surfaces of revolution with quadratic profile. The first surface was defined by shading or motion and the second was defined by texture information. This match was obtained by keeping constant the texture surface, and by varying the illumination direction for the shading surface, and the angular rotation for the motion surface. If the 3D shapes perceived from shading, motion or texture are related to the simulated surface by an affine stretching, then our procedure should produce identical values of perceived curvature, depth and slant also for all other local patches of the three surfaces. Our empirical results, however, show that this is not the case. This implies that the recovered 3D shapes from shading, texture and motion are not related to the simulated 3D surface by an affine transformation. These results are compatible with the hypothesis that the local analysis of image signals specifies different 3D properties. Shading specifies only local curvature; texture local slant and curvature; motion local curvature, slant and depth. Slant and depth from shading, and depth from texture can only be computed through spatial integration, which necessarily introduces noise in the recovery process. Therefore we expected that the perceived values of slant and depth from shading, and depth from texture will be smaller and less reliable then those specified by motion information. Empirical results confirm this hypothesis. NSF: BCS 0345763 no notspecified http://www.kyb.tuebingen.mpg.de/ published -340 4634 7 MJ Tarr M Di Luca WD Zosh Sarasota, FL, USA2005-09-00 243 Fifth Annual Meeting of the Vision Sciences Society (VSS 2005) The human visual system can easily interpret patterns of shading and attribute them to 3D surfaces. However, perceived shape is rarely veridical. Here we introduce a new effect in which extreme deformations of shape are perceived relative to ground truth. Specifically, convex, specular surfaces illuminated by multiple point-light sources are interpreted in a manner more consistent with one light source illuminating a quite different shape. We hypothesize that the visual system is not able to correctly derive the shape of objects under multiple illuminations due to an overriding single light-source assumption. However, this assumption can be disregarded if there is sufficient evidence against it. For example, other cues to shape such as shadows or stereo disparity may provide information sufficient to support more accurate shape perception, regardless of inferences based on this assumption (although this does not mean that observers now interpret the image as arising from multiple light sources). On the other hand, even the presence of boundary contours may not be sufficient for a “correct” interpretation of the image. Along with psychophysical evidence that observers interpret multiple light-source images as the product of a single source, we developed an image-matching method that produces an image of a shape plus single illuminant that is nearly indistinguishable from the original image with multiple light sources. This method is based on local adjustments of slant in order to minimize the difference between the new image and the target. This method effectively predicts the perceived shape of multiply illuminated convex surfaces. In summary, observers appear to apply relatively simple assumptions in how they derive shape percepts from shading. Moreover, it is possible to capture these assumptions in an image-matching model that accurately predicts observer performance. Thus, our ability to derive accurate models of lighting in a scene may be severely restricted. no notspecified http://www.kyb.tuebingen.mpg.de/ published -243 4636 7 F Domini M Di Luca C Caudek Sarasota, FL, USA2005-09-00 992 Fifth Annual Meeting of the Vision Sciences Society (VSS 2005) Retinal velocities and horizontal disparities produced by the projection of a rigidly rotating object are linearly related. This relation identifies a one-dimensional subspace in the disparity-velocity space. A recent model proposed by Domini and Caudek (2004) predicts that the visual system derives 3D structure from stereo-motion signals by means of a two-stage process: a) in the first stage it estimates the direction of this subspace (defined as Intrinsic Constraint (IC) line) by performing a principal component analysis, and b) in the second stage it derives 3D properties from the IC line. Purpose of this work was to test the first hypothesized stage. In two experiments the observers binocularly viewed a rotating 3D cloud of dots and judged when a probe dot at the center of the structure appeared to be aligned in depth with two nearby target dots having both the same velocity and disparity (i.e. same simulated depth). In six conditions (four in the first experiment and three in the second experiment) we independently perturbed the disparity and velocity signals of the surrounding dots so to produce a noisy relationship among these signals (i.e. a noisy IC line). We reasoned that if the visual system recovers the IC line before estimating depth, then these perturbations should also influence the observers' task, even though it only concerns matching the velocity and disparity of the unperturbed probe and target dots. In fact, we found that different levels of noise (experiment 1) influenced the accuracy with which the observers' task was performed. Moreover, we also found that the noise distribution - and not only the noise level - in the velocity-disparity space influenced the observers' accuracy (experiment 2). These findings are compatible with a model that performs a Principle Component Analysis in the disparity-velocity space and recovers depth from the resulting lower-dimensional space. no notspecified http://www.kyb.tuebingen.mpg.de/ published -992 4635 7 M Di Luca C Fantoni Sarasota, FL, USA2005-09-00 989 Fifth Annual Meeting of the Vision Sciences Society (VSS 2005) Consider two vertically adjacent random dot surfaces twisted 20° around the vertical axis and undergoing a 10° oscillation around the same axis during a period of 1/2 second. The rotation of the surfaces is perceived as rigid when they amodally unified behind a horizontal occlusion. Conversely, when the surfaces are interpreted as separated objects, they are perceived as undergoing a rotation of a different magnitude (see Domini, Caudek and Proffitt 1997). This effect suggests that spatial unification may affect the process of recovering depth in structure from motion stimuli by maximizing rotation coherence between joinable surfaces. To test our claim we measured the threshold for perceived rigidity of twisted surfaces in 4 conditions resulting by the combination of 2 factors: occlusion (present vs. absent) and surface-similarity (congruent vs. incongruent). Both the simulated slant of the lower surface and the amount of rotation were kept constant at 10°. The simulated slant of the upper plane was varied in a constant stimuli fashion. Observers were required to judge if the twisted pair of surfaces appeared to rotate rigidly or not. The stimulus is perceived as being a rigid structure for a wider range of slant differences when an occlusion is present. The difference between occlusion-present and occlusion-absent conditions is noticeably reduced when the degree of connectivity between surfaces is lowered by introducing surface dissimilarity. These results support the idea that the surface interpolation process interferes with perceived rigidity of structure from motion stimuli. The amount of perceived rotation is modulated to maximize the coherence between joinable parts in order to achieve a smooth unification of misaligned surfaces (see Fantoni, Gerbino and Kellman 2004). Moreover, perception of surfaces that requires interpolation with torsion is possible with a limited range of twist angle. no notspecified http://www.kyb.tuebingen.mpg.de/ published -989 4637 7 M Di Luca F Domini C Caudek Minneapolis MN, USA2004-11-00 51 45th Annual Meeting of The Psychonomic Society This study explores the combination stereo and motion cues to depth by investigating whether the perceived depth order of two dots could be reversed as a function of surrounding stimulus information. Observers indicated which of two dots presented within a 3-D structure appears closer. The results indicate different depth order for different surrounding conditions. This finding is consistent with the predictions of the intrinsic constraint (IC) model, according to which the visual system uses the linear relation between horizontal disparity and motion of the points of a rigidly rotating object (identifying the IC line) to compute depth. Points that have a ratio between velocity and disparity that is different from the ratio of these signals from the surrounding points are considered as noisy perturbation of their projection on the IC line. Their depth is determined by projecting them on the IC line, and it is, therefore, affected by the surrounding structure. no notspecified http://www.kyb.tuebingen.mpg.de/ published -51 4638 7 M Di Luca F Domini C Caudek 2004-08-00 ISEP, 2004, Hartford (CT) no notspecified http://www.kyb.tuebingen.mpg.de/ published 0 4639 7 M Di Luca F Domini C Caudek 2004-08-00 466 Journal of Vision no notspecified http://www.kyb.tuebingen.mpg.de/ published -466 4641 7 M Di Luca F Domini C Caudek Sarasota, FL, USA2003-10-00 276 Third Annual Meeting of the Vision Sciences Society (VSS 2003) Structure-from-motion studies have shown that the perceived slant of a local planar patch is (1) an increasing function of deformation and (2) a decreasing function of the average velocity of the first-order optic flow (Domini Caudek 1999, Todd and Perotti, 1999). Recent findings, however, indicate that local judgments are subject to a process of spatial integration when local patches are embedded in a global optic flow (Di Luca, Domini and Caudek, 2002). To generalize the previous work, in this study we asked human observers to judge the perceived orientation of local patches of a rotating random-dot spherical surface. These patches could either be seen in isolation or as part of the whole surface. In two experimental conditions the axis of rotation was either in front or behind the simulated surface so to allow the same surface patch to project in each condition the same deformation but different average velocities. The results can be summarized as follow: (1) When the patches were judged in isolation perceived orientations were consistent with those in previous studies on local slant perception. (2) When the whole surface was visible, the local judgments were consistent with the perception of a quasi-spherical surface, suggesting a process of spatial integration. (3) These judgments were not consistent, however, with the veridical percept of a spherical surface and they also indicated that different structures were perceived when the position of the axis of rotation was changed. Surprisingly, we also found that when the same random-dot sphere rotates about different axes of rotation it is perceived as having two qualitatively different shapes (see demo on www.cog.brown.edu/~max/pub/sphere_integration/index.htm). no notspecified http://www.kyb.tuebingen.mpg.de/ published -276 4640 7 M Di Luca F Domini C Caudek Paris, France2003-09-00 74 26th European Conference on Visual Perception In this study we investigated whether motion and stereo information to 3-D structure are combined in a linear fashion as predicted by most of the theories of cue combination. Observers binoc- ularly viewed a motion and stereo specified random-dot planar surface. The observers' task was to place a probe dot on the perceived surface. The perceived position of the probe dot could be adjusted by modifying (i) its 2-D velocity, (ii) its binocular disparity, and (iii) a combination of the two, with respect to the planar surface. In different conditions we varied the relative intensity of the velocity and binocular disparity of the probe dot. If the hypothesis of linear combina- tion of motion and disparity information is correct, then settings derived from different relative intensities of motion and disparity information must lie on a straight line in the velocity - disparity space. The results, however, show that this is not the case. This suggests that motion and stereo modules of 3-D shape reconstruction are not independent. no notspecified http://www.kyb.tuebingen.mpg.de/ published -74 4642 7 M Di Luca F Domini C Caudek Sarasota, FL, USA2002-11-00 648 Second Annual Meeting of the Vision Sciences Society (VSS 2002) Recent investigations have shown that the perceived slant of the orthographic projection of a rotating random-dot planar surface is (1) an increasing function of deformation and (2) a decreasing function of tilt of the velocity field (e.g. Domini and Caudek, 1999). Deformation is defined as the sum of the squared horizontal and vertical gradients of the velocity field and tilt as the ratio between the vertical and horizontal gradients. Further findings have also indicated that perceived slant is (3) a decreasing function of the average velocity of the first-order optic flow (e.g. Todd and Perotti, 1999). If the perceptual interpretation of a linear velocity field is affected by the three factors listed above, then some form of spatial integration must also take place. In fact, observers report perceiving a planar rotating surface when viewing a linear velocity field regardless of the fact that, for each local patch of the velocity field, factors 1 and 2 are the same, but factor 3 can largely vary. According to the above hypothesis, we should then expect that perceived orientation of different patches of a rotating random-dot planar surface should be (i) different when they are viewed in isolation, and (ii) equal when they are viewed as part of the same surface. In our investigation, we asked human observers to judge the perceived slant and tilt of local patches of a rotating random-dot planar surface. Surprisingly, we found that the local patches were perceived as having different slants even if the surface was entirely visible. These differences, however, were significantly smaller than those found when the local patches were perceived in isolation, therefore indicating that spatial integration does in fact take place. Perceived tilt was also found to depend on the local mean velocity. Also this effect, however, was significantly reduced when the whole surface was visible. no notspecified http://www.kyb.tuebingen.mpg.de/ published -648 4643 7 C Fantoni M Di Luca T Agostini A Galmonte 2001-06-00 AIP, Congresso nazionale della sezione di psicologia sperimentale, Alghero (Italy) no notspecified http://www.kyb.tuebingen.mpg.de/ published 0 6486 15 M Di Luca 2007-02-00 no notspecified published 6487 14 M Di Luca 2001-00-00 no notspecified published diplom DiLuca2012 10 M Di Luca DiLuca2010_2 10 M Di Luca DiLuca2010 10 M Di Luca 6578 10 M Di Luca 6493 10 M Di Luca 5696 10 M Harders M Di Luca 5173 10 MO Ernst M Di Luca W Adams 5064 10 T Machulla M Di Luca M Ernst 4517 10 M Di Luca M Ernst 4632 10 M Di Luca M Ernst 6494 10 M Di Luca 6495 10 C Fantoni M Di Luca T Agostini A Galmonte 6496 10 T Agostini M Di Luca C Fantoni D Fantoma A Galmonte P Bruno