Mario Kleiner

As part of a PhD thesis in computer science, i am working on the development of tools and methods, mostly software, to facilitate research in cognitive science. One focus is on video and computer graphics methods for dynamic facial perception research, together with Cristobal Curio, Martin Breidt and Katharina Dobs. Another project is the improvement and maintenance of the Visual Psychophysics Toolbox Version 3, a cross-platform, free software toolkit for cognitive science (http://www.psychtoolbox.org). I also contribute to the development of the GNU/Linux operating system and related parts of the free software ecosystem with the goal of further improving its already excellent suitability for neuro-science research.

Psychophysics Toolbox 3: Developing a modern toolkit for Psychophysics

Mario Kleiner, Cristóbal Curio, Heinrich Bülthoff

Introduction

One of the pillars of both basic and clinical behavioral sciences is the testing of subjects in perceptual tasks. The increasing power and functionality of computer technology has greatly expanded the scope and range of stimuli available for the psychophysical study of both vision and audition, as well as the sophistication of response-contingent experimental designs that may be implemented as part of such study. General purpose computing systems are not designed for psychophysics, therefore deploying these systems in the laboratory requires the time-consuming and technically challenging development and verification of specialized software. Clearly there is much to be gained through the sharing of effective solutions in the form of software toolkits. A popular option is Psychophysics Toolbox 3, a cross-platform, free and open-source software toolkit for the Matlab and Octave programming environments [1].

Goals

Our goal is to improve Psychophysics Toolbox 3 for a large variety of testing paradigms. We develop novel or more efficient methods for stimulus presentation, response collection, and for coping with variations and flaws in general purpose computing platforms. A long term goal is improvement of the underlying infrastructure itself, for example, the third party toolkits employed by us, and specifically the Linux operating system, to promote the use of Linux for neuro-science applications which sometimes require extensive precision, robustness and flexibility more easily achievable by an open-source computing platform.

Methods

Our development efforts are motivated both by feedback from users of the toolkit via its internet forum, and even more so by exploring the capabilities and limitations of the toolkit as part of in-house [2][3] and external research collaborations [4] which require implementation of challenging stimulation paradigms on general purpose commodity hardware. We take advantage of the latest developments in hardware, exploring how paradigms can be implemented more efficiently by use of new features, e.g., the programmable shading hardware of modern graphics cards. We collaborate with other open-source software projects, e.g., the Linux graphics and real-time developer community, helping to improve the functionality and suitability of such systems for very demanding neuro-science applications. This is done by participating in the design process of new features and also by contributing code to those projects. We also occasionally consult vendors of specialized research equipment on optimal design of their hardware for good interoperability with Psychtoolbox 3.

Initial results

Improvements have been made in various areas. One unique feature is the ability for users with limited programming skills to make use of efficient graphics hardware accelerated image processing (Fig. 1) and rendering of procedural dynamic stimuli (Fig. 2). Advanced video-capture and playback functionality allows for implementation of closed-loop action-perception and social interaction studies [4]. The timing precision and robustness of visual and auditory stimulus presentation [5] and response collection has been improved [6]. Some of this work has also found its way into the graphics subsystem of the Linux operating system, which may benefit other toolkits and similar applications as well [7].

Figure 1: Illustration of Psychtoolbox GPU image processing pipeline vs. regular drawing toolkits. Top: Typical toolkits directly draw to the OpenGL backbuffer. Bottom: Example processing graph in Psychtoolbox. Blue are intermediate frame buffers. Red are GPU accelerated processing operators. In this stereo configuration, each image for each eye would be drawn separately, then exposed to some per-eye image processing. The resultant images would be merged together, e.g., into an anglyph stereo image, then subjected to post-processing, e.g., gamma correction and written to the final framebuffer. All processing would happen transparently for the usercode and fast due to GPU hardware acceleration.

Figure 2: Two examples of advanced stimuli. Left: A structure from motion stimulus, generated from a 3D shape in real-time via fast GPU based random sampling and tracking of 3D surface points (color added for illustration – this is a rotating earth globe). Right: Linear superposition of 100 gabor patches in real-time via use of GPU accelerated alpha-blending and floating point framebuffers. While such stimuli can also be generated offline in other toolkits, Psychtoolbox allows to generate them with minimal amount of code and in real-time.

References

1. Kleiner M, Brainard D, Pelli D (2007) What‘s new in psychtoolbox-3?, Perception 36(ECVP 2007 Abstract Supplement) 14.

2. Curio C, Kleiner M, Breidt M, Bülthoff HH (2010) The virtual face mirror project: revealing dynamic self-perception in humans, 4th International Conference on Cognitive Systems (CogSys 2010) 4 1.

3. Conrad V, Bartels A, Kleiner M, Noppeney U (2010) Audiovisual interactions in binocular rivalry, Journal of Vision 10(10:27) 1-15.

4. Redcay E, Dodell-Feder D, Pearrow MJ, Mavros PL, Kleiner M, Gabrieli JDE, Saxe R (2010) Live face-to-face interaction during fmri: a new tool for social cognitive neuroscience, NeuroImage 50(4) 1639-1647.

5. Kleiner M (2010) Visual stimulus timing precision in psychtoolbox-3: tests, pitfalls and solutions, Perception 39(ECVP Abstract Supplement) 189.

6. Li X, Liang Z, Kleiner M, Lu Z-L (2010) Rtbox: a device for highly accurate response time measurements, Behavior Research Methods 42(1) 212-225.

7. See <>