Last update: Mon Nov 24 15:39:17 CET 2008

NIPS Symposium on Computational Photography

Slides and Links

Organizers

Bill Freeman and Bernhard Schölkopf

Date

11 December 2008, 1:30pm-4:30pm

Location

Hyatt Regency Vancouver, BC, Canada

Computation will change photography. The sensor no longer has to record the final image, but only data that can lead to the final image. Computation can solve longstanding photographic problems (eg, deblurring) and well as open the door for radical new designs and capabilities for image capture, processing, and viewing (eg, lightfield cameras). Many of these possibilities offer great machine learning problems, and much of the progress in "computational photography" will rely on solutions to these challenging machine learning problems.

We've gathered 5 leading researchers in this new field to describe their work at the intersection of photography and machine learning.

List of Speakers

Time Lecturer Title Abstract
1:30 - 2:00 Yair Weiss Hebrew University Random projections, computational photography, and computational neuroscience One way of thinking about computational photography is as an attempt to design the optimal camera to capture visual information. This is closely related to Linsker's InfoMax approach for modeling biological sensory systems. Bell and Sejnowski have shown how the InfoMax approach can be used to predict the Gabor-like receptive fields in V1.
I will describe some of our work on UNDERCOMPLETE InfoMax --- when the number of sensors is less than the dimensionality of the input. In this setting, for a surprisingly large number of signal models, it can be shown that random projections are InfoMax optimal. In particular, for whitened natural image patches, random projections are the most informative while searching for the LEAST informative filters gives Gabor filters.

Joint work with Hyung Sung Chang and Bill Freeman
2:00 - 2:30 Steve Seitz University of Washington Navigating the World's Photographs There's a big difference between looking at a photograph of a place and being there. But what if you had access to every photo ever captured of that place and could conjure up any view at will? With billions of photographs currently available online, the Internet is beginning to resemble such a database, capturing most of the world's significant sites from a huge number of vantage points and viewing conditions. For example, a Google image search for "notre dame" or "grand canyon" each returns more than a million photos, showing the sites from myriad viewpoints, different times of day and night, and changes in season, weather and decade.
This talk explores ways of transforming this massive, unorganized photo collection into visualizations of the world's sites, cities, and landscapes. After a brief recap of our work on Photo Tourism and Photosynth, I will focus on current efforts and newest results that seek to discover human preference and behavior patterns in real world scenes.
2:30 - 3:00 Ramesh Raskar M.I.T. Computational Photography: From Epsilon to Coded Photography In this talk, I will focus on Coded Photography. 'Less is more' in Coded Photography. By blocking light over time or space, we can preserve more details about the scene in the recorded single photograph.
  1. Coded Exposure http://raskar.info/deblur: By blocking light in time, by fluttering the shutter open and closed in a carefully chosen binary sequence, we can preserve high spatial frequencies of fast moving objects to support high quality motion deblurring.
  2. Coded Aperture Optical Heterodyning http://raskar.info/Mask/: By blocking light near the sensor with a sinusoidal grating mask, we can record 4D light field on a 2D sensor. And by blocking light with a mask at the aperture, we can extend the depth of field and achieve full resolution digital refocussing.
  3. Coded Illumination http://raskar.info/NprCamera: By observing blocked light at silhouettes, a multi-flash camera can locate depth discontinuities in challenging scenes without depth recovery.
  4. Coded Sensors http://www.umiacs.umd.edu/%7Eaagrawal/gradcam/gradcam.html: By sensing intensities with lateral inhibition, a "Gradient Camera" can record large as well as subtle changes in intensity to recover a high-dynamic range image.
  5. Coded Spectrum http://www.cs.northwestern.edu/%7Eamohan/agile/: By blocking parts of a "rainbow", we can create cameras with digitally programmable wavelength profile.
I will show several applications and describe emerging techniques to recover scene parameters from coded photographs.

Recent joint work with Jack Tumblin, Amit Agrawal, Ashok Veeraraghavan and Ankit Mohan
3:30 - 4:00 Wojciech Matusik Adobe Computational imaging using camera arrays With the success of digital photography and improvements in digital display technologies during the last years, we have witnessed a transformation in the way photographs and videos are captured, processed, and viewed. The emerging field of computational photography proceeds even further by fundamentally rethinking the design of the hardware, the associated algorithms, and representations for images and video.
In this context, camera arrays emerge as a new type of imaging device with greatly extended capabilities compared to traditional cameras. Camera arrays allow for images that are higher quality, richer, and easily post-processed. However, it is crucial to develop novel algorithms that exploit the wealth of captured information in order to both efficiently solve traditionally difficult problems in computer vision and to allow for new applications that were not considered using current cameras.
In my talk I will present a number of algorithms that showcase unique capabilities of camera arrays. First, I address the problem of automatic, real-time, passive, and robust image segmentation -- a long-standing problem in computer vision. I will show how to modify the standard matting equation to work directly with variance measurements captured with a camera array. This leads to development of the first real-time system for video matting. In the second part of my talk, I will present a method to track a 2D object through significant occlusion using a camera array. The proposed method does not require explicit modeling or reconstruction of the scene and enables tracking in complex, dynamic scenes. Finally, I will describe a complete 3-D TV system that allows for real-time acquisition, transmission, and display of dynamic scenes. In this context, I will describe a signal processing framework to address some of the major issues in cinematography for 3-D displays.
4:00 - 4:30 Bill Freeman M.I.T. Bayesian analysis of cameras Computational approaches to photography have led to many new camera designs: plenoptic, coded aperture, multi-lens, etc. The goal of these cameras may be to reconstruct not just an image but the entire lightfield—the light rays at every angle at every position. How can we compare how these very different cameras perform at that task? How do the parameters of each camera affect its performance? The breadth of imaging designs requires new tools to understand the tradeoffs between different cameras.
This talk introduces a unified framework for analyzing computational imaging approaches. Each sensor element is modeled as an inner product over the 4D light field. The imaging task is then posed as Bayesian inference: given the observed noisy light field projections and a prior on light field signals, estimate the original light field. Under common imaging conditions, we compare the performance of various camera designs using 2D light field simulations. This framework allows us to better understand the tradeoffs of each camera type and to analyze their strengths and limitations. Joint work with Anat Levin and Fredo Durand.
The manuscript: (ECCV 2008) http://people.csail.mit.edu/billf/papers/lightfields-Levin-Freeman-Durand-ECCV08.pdf

To participate, you need to register either for the NIPS conference or for the NIPS workshops. If there is sufficient interest, we might do a poster session during the break. If you are interested in presenting a poster, please send an email to Sabrina.Nielebock@tuebingen.mpg.de.