The BiDi Screen is an example of a new type of thin I/O device that possesses the ability both to capture images and display them. Scene depth can be derived from BiDi Screen imagery, allowing for 3D gestural and 2D multi-touch interfaces. This bidirectional screen extends the latest trend in LCD devices, which has seen the incorporation of photo-transistors into every display pixel. Using a novel optical masking technique developed at the Media Lab, the BiDi Screen can capture light field-like quantities, unlocking a wide array of applications from 3D gesture and touch interaction with CE devices, to seamless video communication.

We introduce a novel interactive method to assess cataracts in the human eye by crafting an optical solution that measures the perceptual impact of forward scattering on the foveal region. Current solutions rely on highly trained clinicians to check the back scattering in the crystallin lens and test their predictions on visual acuity tests. Close-range parallax barriers create collimated beams of light to scan through sub-apertures scattering light as it strikes a cataract. User feedback generates maps for opacity, attenuation, contrast, and local point-spread functions. The goal is to allow a general audience to operate a portable, high-contrast, light-field display to gain a meaningful understanding of their own visual conditions. The compiled maps are used to reconstruct the cataract-affected view of an individual, offering a unique approach for capturing information for screening, diagnostic, and clinical analysis.

Research in computer vision is riding a new tide called compressive sensing. Carefully designed capture methods exploit the sparsity of the underlying signal in a transformed domain to reduce the number of measurements and use an appropriate reconstruction method. Traditional progressive methods capture successively more detail using sequence of simple projection basis whereas random projections do not use any sequence except l0 minimization for reconstruction which is computationally in-efficient. Here, we question this new tide and claim for most situations simple methods work better and the best projective method would be in between the two extremes.

For 3D displays to be successful, they must be bright enough to compete with 2D displays and not diminish display resolution. To date, stacked-LCD displays have employed parallax barriers, which use pinhole or stripe patterns to provide view-dependent imagery. We show a prototype that adapts the imagery on both layers to multi-view 3D content, increasing brightness while maintaining display resolution. This promises a future of devices with sharp 2D screens and 3D displays with full horizontal and vertical parallax.

With networked cameras in everyone's pockets, we are exploring the practical and creative possibilities of public imaging. LensChat allows cameras to communicate with each other using trusted optical communications, allowing users to share photos with a friend by taking pictures of each other, or borrow the perspective and abilities of many cameras.

Can a person look at a portable display, click on a few buttons, and recover his refractive condition? Our optometry solution combines inexpensive optical elements and interactive software components to create a new optometry device suitable for developing countries. The technology allows for early, extremely low-cost, mobile, fast, and automated diagnosis of the most common refractive eye disorders: myopia (nearsightedness), hypermetropia (farsightedness), astigmatism, and presbyopia (age-related visual impairment). The patient overlaps lines in up to eight meridians and the Android app computes the prescription. The average accuracy is comparable to the prior art—and in some cases, even better. We propose the use of our technology as a self-evaluation tool for use in homes, schools, and at health centers in developing countries, and in places where an optometrist is not available or is too expensive.

We demonstrate a new technique that allows a camera to rapidly acquire reflectance properties of objects 'in the wild' from a single viewpoint, over relatively long distances and without encircling equipment. This project has a wide variety of applications in computer graphics including image relighting, material identification, and image editing.

We present a new method for scanning 3D objects in a single shot, shadow-based method. We decouple 3D occluders from 4D illumination using shield fields: the 4D attenuation function which acts on any light field incident on an occluder. We then analyze occluder reconstruction from cast shadows, leading to a single-shot light field camera for visual hull reconstruction.

Recently, multi-view display hardware has made compelling progress in graphics. Soundaround is a multi-viewer interactive audio system, designed to be integrated into unencumbered multi-view display systems, presenting localized audio/video channels with no need for glasses or headphones. Our technical work describes a framework for the design of multi-viewer interactive audio systems that is general and supports optimization of the system for multiple observation planes and room responses.

This work focuses on bringing powerful concepts from wave optics to the creation of new algorithms and applications for computer vision and graphics. Specifically, ray-based, 4D lightfield representation, based on simple 3D geometric principles, has led to a range of new applications that include digital refocusing, depth estimation, synthetic aperture, and glare reduction within a camera or using an array of cameras. The lightfield representation, however, is inadequate to describe interactions with diffractive or phase-sensitive optical elements. Therefore we use Fourier optics principles to represent wavefronts with additional phase information. We introduce a key modification to the ray-based model to support modeling of wave phenomena. The two key ideas are "negative radiance" and a "virtual light projector." This involves exploiting higher dimensional representation of light transport.

Computer vision is a class of technologies that lets computers use cameras to automatically stitch together panoramas, reconstruct 3-D geometry from multiple photographs, and even tell you when the water's boiling. For decades, this technology has been advancing mostly within the confines of academic institutions and research labs. Vision on Tap is our attempt to bring computer vision to the masses.