The world in mind

Heinrich Bülthoff's work shaped modern psychophysics, the scientific discipline that studies the relationship between impulses and their perception. In this interview, the director emeritus of the Max Planck Institute for Biological Cybernetics takes us on a walk through many years of his research career, a virtual Tübingen and to the institute's motion simulators.

How can your research be described in a few words?

In principle, we pursue the question: How does the world get into our minds - and how does it get out again? This essentially involves our senses, with which we grasp the world. But it's also about the world itself, in which we are moving - and so reliably that we don't constantly collide with things and, for example, navigate through foreign cities.

Your scientific career began here in Tübingen ...

I studied biology in Tübingen and accompanied the systems and control theory courses as a lecture assistant for Professor Varju, who held the only chair for biological cybernetics in Germany at the time. This brought me into contact with this field of research, which investigates control processes in biological organisms. Later, for my diploma thesis with Professor Götz at the Max Planck Institute for Biological Cybernetics, I studied an illusion of motion, that is, an optical illusion that the fruit fly Drosophila melanogaster perceives as motion. Actually by chance, because I looked into the apparatus myself, I noticed that this illusion also works in humans. This was a first indication that very elementary information processing probably works similarly in humans and flies.

The work was published in the journal Nature - a brilliant start to a research career. Nevertheless, you decided to give your work a new direction.

I have actually always been interested in information processing in humans. That's why I went to the U.S., first to the Massachusetts Institute of Technology (MIT) in Cambridge, then to Brown University in Providence. There I was able to focus on this topic and at the same time add a new dimension to my work. At MIT, we had high-performance computers at our disposal. We were able to simulate illusions of motion or change visual information in a targeted way.

Can you give an example of the findings from that time?

We discovered, for example, that we could also recognize objects from a perspective - unknown to us until then - if we had previously stored enough views from other angles. This was a big point of contention at the time. The conventional view was that our brain assembles objects from elementary building blocks - much like an architectural program. However, humans do not use slow mental rotation of basic building blocks for rapid recognition, but rather rely on rapid comparison with many stored views. Our results ultimately revolutionized work on machine vision systems as well. Visual computer systems now compute objects using many image data and neural networks rather than geometric computations from a few 3D data sets, in analogy to what happens in the brain.

At Brown University, we were able to make another important discovery: To create a picture of the world in their minds, people attach different importance to different sources of information. It is important to mention that our sense of sight alone provides us with different information such as shadows, textures and movements, in addition to the various data from our other senses. In the experiment, for example, a three-dimensional image of a cup can be shown. At the same time, the subject feels the cup with his fingers. If the image is clear, he relies entirely on his eyes. If the image is blurry, the subjects begin to 'grasp' the cup - its shape, size, surface texture, and so on.

So we were able to show that humans are even able to compute multimodal sensory information in a statistically optimal way. That is, sensory perceptions are weighted according to their reliability. The work on the perception of shape and depth was in principle the forerunner of the later very popular perception research with Bayesian approach.

Back in Tübingen from the U.S., as Director of the Max Planck Institute for Biological Cybernetics, you were the first to recreate Tübingen's inner city on a computer. What was the point of that?

We wanted to know how people orient themselves in their environment, how they find their way around a foreign city. Tübingen is a great place to study navigation, because the town is so winding, with lots of small alleys. I even discovered a connection between two streets in virtual Tübingen that I hadn't known about before. 

Can't you study navigation in a real city?

Virtual realities are important tools for us. We can work under very controlled conditions and still create an experimental situation that is close to reality. In addition, in a virtual city we have the possibility to move so-called landmarks, for example, to move the marketplace or to swap prominent buildings. In this way, we can find out which features are predominantly used for orientation. Do the test persons orient themselves by stored views of street corners or a towering church steeple? In the end, however, it turned out that navigation is also highly individual.

With computer graphics, virtual realities and motion simulators, they became one of the pioneers of modern psychophysics. How have these very technical methods revolutionized the discipline? 

As much as perception fascinated me from the beginning, at some point I became frustrated with the more traditional approach to psychophysics. Most experiments were based on very reduced stimuli, such as the fringe patterns used to study motion vision in both flies and humans over many years. However, life does not take place in the laboratory, and the demands on perception are far more complex in everyday life, because a wide variety of impulses come at us, which the brain must compute into a composite piece of information. Using modern methods of computer graphics and virtual reality, we have succeeded in developing very realistic experiments. This has allowed us to take psychophysics to another level: We can thus paint a more comprehensive picture of the cognitive processes in the brain, of the perceptual and learning processes that take place when people interact with complex environments.

You have an impressive array of equipment at your disposal for your work ...

I was in the fortunate position that the Max Planck Society even gave me my own research building for this purpose, the Cyberneum. Among other things, we have developed two worldwide unique motion simulators here over the last fifteen years, with which we can investigate precisely these questions. With our CyberMotion simulator, a highly modified industrial robot with a cabin, we can investigate the perception of self-motion very precisely. The CableRobot simulator consists of a cabin that is moved via a complex, novel cable construction. Both have a wide range of motion and allow flexible and highly realistic movement sequences - from car racing to helicopter flights to barely perceptible movements, which are important for exploring the sense of balance.

When you look back on your career: What is particularly important to you?

In recent years, we have been able to take advantage of our unique infrastructure to do more applied work as well. We've done a lot of collaborative work with engineers, developers and industry. With our simulators, we can do more than just basic research. We can also support development with our know-how. For example, we were one of the first to start investigating what it takes to move safely in autonomous vehicles. In addition, our findings flow into the further development of efficient motion simulators. Through this application-oriented work, I have the feeling that I can give back to society a little bit of what we have received in terms of opportunities and funding for our research.

But it is also important to me that I have always been lucky enough to work with a lot of exciting people over the years. It's not all my research alone. I have had many very good PhD students and postdocs and always interesting guest scientists who have worked with us. And quite a few professors have also come out of my department. That makes me very happy.