Camera Culture
Making the invisible visible–inside our bodies, around us, and beyond–for health, work, and connection.
The Camera Culture group is building new tools to better capture and share visual information. What will a camera look like in ten years? How should we change the camera to improve mobile photography? How will a billion networked and portable cameras change the social culture? We exploit unusual optics, novel illumination, and emerging sensors to build new capture devices and develop associate algorithms.

Research Projects

  • 6D Display

    Ramesh Raskar, Martin Fuchs, Hans-Peter Seidel, and Hendrik P. A. Lensch

    Is it possible to create passive displays that respond to changes in viewpoint and incident light conditions? Holograms and 4D displays respond to changes in viewpoint. 6D displays respond to changes in viewpoint as well as surrounding light. We encode the 6D reflectance field into an ordinary 2D film. These displays are completely passive and do not require any power. Applications include novel instruction manuals and mood lights.

  • A Switchable Light-Field Camera

    Matthew Hirsch, Sriram Sivaramakrishnan, Suren Jayasuriya, Albert Wang, Aloysha Molnar, Ramesh Raskar, and Gordon Wetzstein

    We propose a flexible light-field camera architecture that represents a convergence of optics, sensor electronics, and applied mathematics. Through the co-design of a sensor that comprises tailored, angle-sensitive pixels and advanced reconstruction algorithms, we show that, contrary to light-field cameras today, our system can use the same measurements captured in a single sensor image to recover either a high-resolution 2D image, a low-resolution 4D light field using fast, linear processing, or a high-resolution light field using sparsity-constrained optimization.

  • AnEye: Extending the Reach of Anterior Segment Ophthalmic Imaging

    Ramesh Raskar and Shantanu Sinha

    Eye exams via a slit lamp are critical in early diagnosis of diseases such as cataracts, corneal injury, and pterygia, in order to avert vision loss. The slit lamp is one of the most versatile tools in an ophthalmologist's clinic, but is big, expensive, and is designed with specialized ophthalmic clinics in mind. AnEye is a suite of portable, computationally driven solutions that leverage modern optics and commercially available consumer electronics to extend the reach of examinations of the anterior segment of the eye well beyond large hospitals and clinics, into resource-constrained settings such as rural mass-screening camps, mobile ophthalmology clinics, and even primary care.

  • Beyond the Self-Driving Car

    Ramesh Raskar, Barmak Heshmat Dehkordi and Gurmukh Bhasin

    This concept gallery shows the chain of startups and ideas that will follow after the emergence of self-driving cars.

  • Blind and Reference-Free Fluorescence Lifetime Estimation via Consumer Time-of-Flight Sensors

    Ayush Bhandari, Christopher Barsi and Ramesh Raskar

    Fluorescence lifetime imaging is a significant bio-imaging tool that finds important applications in life-sciences. Widely known applications include cancer detection and DNA sequencing. To that end, fluorescence microscopy which is at the heart of bio-imaging is an electronically and optically sophisticated device which is prohibitively expensive. Our work is demonstrates the fluorescence microscopy like functionality can be achieved by a simple, consumer sensor such as the Microsoft Kinect which costs about $100. This is done by trading-off the precision in optics and electronics for sophistication in computational methods. Not only this allows for massive cost reduction but leads to several advances in the area. For example, our method is calibration-free in that we do not assume sample's relative placement with respect to the sensor. Furthermore, our work opens new pathways of interaction between bio-imaging, optics and computer vision communities.

  • Bokode: Imperceptible Visual Tags for Camera-Based Interaction from a Distance

    Ramesh Raskar, Ankit Mohan, Grace Woo, Shinsaku Hiura and Quinn Smithwick
    With over a billion people carrying camera-phones worldwide, we have a new opportunity to upgrade the classic bar code to encourage a flexible interface between the machine world and the human world. Current bar codes must be read within a short range, and the codes occupy valuable space on products. We present a new, low-cost, passive optical design so that bar codes can be shrunk to fewer than 3mm and can be read by unmodified ordinary cameras several meters away.
  • CATRA: Mapping of Cataract Opacities Through an Interactive Approach

    Ramesh Raskar, Vitor Pamplona, Erick Passos, Jan Zizka, Jason Boggess, David Schafran, Manuel M. Oliveira, Everett Lawson, and Estebam Clua

    We introduce a novel interactive method to assess cataracts in the human eye by crafting an optical solution that measures the perceptual impact of forward scattering on the foveal region. Current solutions rely on highly trained clinicians to check the back scattering in the crystallin lens and test their predictions on visual acuity tests. Close-range parallax barriers create collimated beams of light to scan through sub-apertures, scattering light as it strikes a cataract. User feedback generates maps for opacity, attenuation, contrast, and local point-spread functions. The goal is to allow a general audience to operate a portable, high-contrast, light-field display to gain a meaningful understanding of their own visual conditions. The compiled maps are used to reconstruct the cataract-affected view of an individual, offering a unique approach for capturing information for screening, diagnostic, and clinical analysis.

  • Coded Computational Photography

    Jaewon Kim, Ahmed Kirmani, Ankit Mohan and Ramesh Raskar
    Computational photography is an emerging multi-disciplinary field at the intersection of optics, signal processing, computer graphics and vision, electronics, art, and online sharing in social networks. The first phase of computational photography was about building a super-camera that has enhanced performance in terms of the traditional parameters, such as dynamic range, field of view, or depth of field. We call this Epsilon Photography. The next phase of computational photography is building tools that go beyond the capabilities of this super-camera. We call this Coded Photography. We can code exposure, aperture, motion, wavelength, and illumination. By blocking light over time or space, we can preserve more details about the scene in the recorded single photograph.
  • Coded Focal Stack Photography

    Ramesh Raskar, Gordon Wetzstein, Xing Lin, Nikhil Naik and Tsinghua University

    We present coded focal stack photography as a computational photography paradigm that combines a focal sweep and a coded sensor readout with novel computational algorithms. We demonstrate various applications of coded focal stacks, including photography with programmable non-planar focal surfaces and multiplexed focal stack acquisition. By leveraging sparse coding techniques, coded focal stacks can also be used to recover a full-resolution depth and all-in-focus (AIF) image from a single photograph. Coded focal stack photography is a significant step towards a computational camera architecture that facilitates high-resolution post-capture refocusing, flexible depth of field, and 3D imaging.

  • Compressive Light-Field Camera: Next Generation in 3D Photography

    Kshitij Marwah, Gordon Wetzstein, Yosuke Bando and Ramesh Raskar

    Consumer photography is undergoing a paradigm shift with the development of light field cameras. Commercial products such as those by Lytro and Raytrix have begun to appear in the marketplace with features such as post-capture refocus, 3D capture, and viewpoint changes. These cameras suffer from two major drawbacks: major drop in resolution (converting a 20 MP sensor to a 1 MP image) and large form factor. We have developed a new light-field camera that circumvents traditional resolution losses (a 20 MP sensor turns into a full-sensor resolution refocused image) in a thin form factor that can fit into traditional DSLRs and mobile phones.

  • Eyeglasses-Free Displays

    Ramesh Raskar, Nikhil Naik and Gordon Wetzstein

    Millions of people worldwide need glasses or contact lenses to see or read properly. We introduce a computational display technology that predistorts the presented content for an observer, so that the target image is perceived without the need for eyewear. We demonstrate a low-cost prototype that can correct myopia, hyperopia, astigmatism, and even higher-order aberrations that are difficult to correct with glasses.

  • Health-Tech Innovations with Tata Trusts, Mumbai

    Ramesh Raskar and Anshuman Das

    We believe that tough global health problems require an innovation pipeline. We must bring together the people and providers facing health challenges to form what we call an innovation continuum: inventors building new low-cost technologies; developers capable of rapidly iterating on these inventions for use in the real world; clinicians and end users to validate our creations; and entrepreneurs, philanthropists, and development agencies to scale our solutions. We are asking big questions such as: What billion-dollar ideas could impact a billion lives in health, education, transportation through digital interfaces, digital opportunities, and applications for physical systems? Using machine learning, computer vision, Big Data, sensors, mobile technology, diagnostics, and crowdsourcing, we are conducting research at the Media Lab, and also collaborating with innovators in three centers in India and in other centers worldwide. Innovations like this launched the effort to create the Emerging Worlds initiative.

  • Hyderabad Eye Health Collaboration with LVP

    Ramesh Raskar and Anshuman Das

    We believe that tough global health problems require an innovation pipeline. We must bring together the people and providers facing health challenges to form what we call an innovation continuum: inventors building new low-cost technologies; developers capable of rapidly iterating on these inventions for use in the real world; clinicians and end users to validate our creations; and entrepreneurs, philanthropists, and development agencies to scale our solutions. We are asking big questions such as: What billion-dollar ideas could impact a billion lives in health, education, transportation through digital interfaces, digital opportunities, and applications for physical systems? Using machine learning, computer vision, Big Data, sensors, mobile technology, diagnostics, and crowdsourcing, we are conducting research at the Media Lab, and also collaborating with innovators in three centers in India and in other centers worldwide. Innovations like this launched the effort to create the Emerging Worlds initiative.

  • Identi-Wheez: A device for in-home diagnosis of asthma

    Ramesh Raskar, Nikhil Naik and Guy Satat

    Asthma is the most common chronic illness among children. The skills required to diagnose it make it an even greater concern. Our solution is a child-friendly wearable device that allows in-home diagnosis of asthma. The device acquires simultaneous measurements from multiple stethoscopes. The recordings are then sent to a specialist who uses assistive diagnosis algorithms that enable auscultation (listening to lung sounds with a stethoscope). Sound refocusing algorithms enable the specialist to listen to any location in the lungs. The specialist also has access to a sound "heat map" that shows the location of sound sources in the lungs.

  • Imaging Behind Diffusive Layers

    Special Interest group(s): 
    Ramesh Raskar, Barmak Heshmat Dehkordi and Dan Raviv

    The use of fluorescent probes and the recovery of their lifetimes allow for significant advances in many imaging systems, in particular medical imaging systems. Here, we propose and experimentally demonstrate reconstructing the locations and lifetimes of fluorescent markers hidden behind a turbid layer. This opens the door to various applications for non-invasive diagnosis, analysis, flowmetry, and inspection. The method is based on a time-resolved measurement which captures information about both fluorescence lifetime and spatial position of the probes. To reconstruct the scene, the method relies on a sparse optimization framework to invert time-resolved measurements. This wide-angle technique does not rely on coherence, and does not require the probes to be directly in line of sight of the camera, making it potentially suitable for long-range imaging.

  • Imaging through Scattering Media Using Femtophotography

    Ramesh Raskar, Christopher Barsi and Nikhil Naik

    We use time-resolved information in an iterative optimization algorithm to recover reflectance of a three-dimensional scene hidden behind a diffuser. We demonstrate reconstruction of large images without relying on knowledge of diffuser properties.

  • Imaging with all photons

    Ramesh Raskar, Barmak Heshmat Dehkordi, Dan Raviv and Guy Satat

    We demonstrate a new method to image through scattering materials like tissue and fog. The demonstration includes imaging an object hidden behind 1.5cm of tissue; it's like imaging through the palm of a hand. Our optical method is based on measuring and using all photons in the signal (as opposed to traditional methods, which use only part of the signal). Specifically, we use a time-resolved method that allows us to distinguish between photons that travel different paths in the tissue. Combining this unique measurement process with novel algorithms allows us to recover the hidden objects. This technique can be used in bio-medical imaging, as well as imaging through fog and clouds.

  • Inverse Problems in Time-of-Flight Imaging

    Ayush Bhandari and Ramesh Raskar

    We are exploring mathematical modeling of time-of-flight imaging problems and solutions.

  • Layered 3D: Glasses-Free 3D Printing

    Gordon Wetzstein, Douglas Lanman, Matthew Hirsch, Wolfgang Heidrich, and Ramesh Raskar

    We are developing tomographic techniques for image synthesis on displays composed of compact volumes of light-attenuating material. Such volumetric attenuators recreate a 4D light field or high-contrast 2D image when illuminated by a uniform backlight. Since arbitrary views may be inconsistent with any single attenuator, iterative tomographic reconstruction minimizes the difference between the emitted and target light fields, subject to physical constraints on attenuation. For 3D displays, spatial resolution, depth of field, and brightness are increased, compared to parallax barriers. We conclude by demonstrating the benefits and limitations of attenuation-based light field displays using an inexpensive fabrication method: separating multiple printed transparencies with acrylic sheets.

  • LensChat: Sharing Photos with Strangers

    Ramesh Raskar, Rob Gens and Wei-Chao Chen
    With networked cameras in everyone's pockets, we are exploring the practical and creative possibilities of public imaging. LensChat allows cameras to communicate with each other using trusted optical communications, allowing users to share photos with a friend by taking pictures of each other, or borrow the perspective and abilities of many cameras.
  • Looking Around Corners

    Andreas Velten, Di Wu, Christopher Barsi, Ayush Bhandari, Achuta Kadambi, Nikhil Naik, Micha Feigin, Daniel Raviv, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G. Bawendi, and Ramesh Raskar
    Using a femtosecond laser and a camera with a time resolution of about one trillion frames per second, we recover objects hidden out of sight. We measure speed-of-light timing information of light scattered by the hidden objects via diffuse surfaces in the scene. The object data are mixed up and are difficult to decode using traditional cameras. We combine this "time-resolved" information with novel reconstruction algorithms to untangle image information and demonstrate the ability to look around corners.
  • Nashik Smart Citizen Collaboration with TCS

    Ramesh Raskar and Anshuman Das

    We believe that tough global health problems require an innovation pipeline. We must bring together the people and providers facing health challenges to form what we call an innovation continuum: inventors building new low-cost technologies; developers capable of rapidly iterating on these inventions for use in the real world; clinicians and end users to validate our creations; and entrepreneurs, philanthropists, and development agencies to scale our solutions. We are asking big questions such as: What billion-dollar ideas could impact a billion lives in health, education, transportation through digital interfaces, digital opportunities, and applications for physical systems? Using machine learning, computer vision, Big Data, sensors, mobile technology, diagnostics, and crowdsourcing, we are conducting research at the Media Lab, and also collaborating with innovators in three centers in India and in other centers worldwide. Innovations like this launched the effort to create the Emerging Worlds initiative.

  • NETRA: Smartphone Add-On for Eye Tests

    Vitor Pamplona, Manuel Oliveira, Erick Passos, Ankit Mohan, David Schafran, Jason Boggess and Ramesh Raskar

    Can a person look at a portable display, click on a few buttons, and recover his or her refractive condition? Our optometry solution combines inexpensive optical elements and interactive software components to create a new optometry device suitable for developing countries. The technology allows for early, extremely low-cost, mobile, fast, and automated diagnosis of the most common refractive eye disorders: myopia (nearsightedness), hypermetropia (farsightedness), astigmatism, and presbyopia (age-related visual impairment). The patient overlaps lines in up to eight meridians, and the Android app computes the prescription. The average accuracy is comparable to the traditional method -- and in some cases, even better. We propose the use of our technology as a self-evaluation tool for use in homes, schools, and at health centers in developing countries, and in places where an optometrist is not available or is too expensive.

  • New Methods in Time-of-Flight Imaging

    Ramesh Raskar, Christopher Barsi, Ayush Bhandari, Anshuman Das, Micha Feigin-Almon and Achuta Kadambi

    Time-of-flight (ToF) cameras are commercialized consumer cameras that provide a depth map of a scene, with many applications in computer vision and quality assurance. Currently, we are exploring novel ways of integrating the camera illumination and detection circuits with computational methods to handle challenging environments, including multiple scattering and fluorescence emission.

  • Optical Brush: Enabling Deformable Imaging Interfaces

    Ramesh Raskar and Barmak Heshmat Dehkordi

    Our deformable camera exploits new, flexible form factors for imaging in turbid media. In this study we enable a brush-like form factor with a time-of-flight camera. This has enabled us to reconstruct images through a set of 1100 optical fibers that are randomly distributed and permuted in a medium.

  • PhotoCloud: Personal to Shared Moments with Angled Graphs of Pictures

    Ramesh Raskar, Aydin Arpa, Otkrist Gupta and Gabriel Taubin

    We present a near real-time system for interactively exploring a collectively captured moment without explicit 3D reconstruction. Our system favors immediacy and local coherency to global consistency. It is common to represent photos as vertices of a weighted graph. The weighted angled graphs of photos used in this work can be regarded as the result of discretizing the Riemannian geometry of the high dimensional manifold of all possible photos. Ultimately, our system enables everyday people to take advantage of each others' perspectives in order to create on-the-spot spatiotemporal visual experiences similar to the popular bullet-time sequence. We believe that this type of application will greatly enhance shared human experiences, spanning from events as personal as parents watching their children's football game to highly publicized red-carpet galas.

  • Polarization Fields: Glasses-Free 3DTV

    Douglas Lanman, Gordon Wetzstein, Matthew Hirsch, Wolfgang Heidrich, and Ramesh Raskar

    We introduce polarization field displays as an optically efficient design for dynamic light field display using multi-layered LCDs. Such displays consist of a stacked set of liquid crystal panels with a single pair of crossed linear polarizers. Each layer is modeled as a spatially controllable polarization rotator, as opposed to a conventional spatial light modulator that directly attenuates light. We demonstrate that such displays can be controlled, at interactive refresh rates, by adopting the SART algorithm to tomographically solve for the optimal spatially varying polarization state rotations applied by each layer. We validate our design by constructing a prototype using modified off-the-shelf panels. We demonstrate interactive display using a GPU-based SART implementation supporting both polarization-based and attenuation-based architectures.

  • Portable Retinal Imaging

    Everett Lawson, Jason Boggess, Alex Olwal, Gordon Wetzstein, and Siddharth Khullar

    The major challenge in preventing blindness is identifying patients and bringing them to specialty care. Diseases that affect the retina, the image sensor in the human eye, are particularly challenging to address, because they require highly trained eye specialists (ophthalmologists) who use expensive equipment to visualize the inner parts of the eye. Diabetic retinopathy, HIV/AIDS-related retinitis, and age-related macular degeneration are three conditions that can be screened and diagnosed to prevent blindness caused by damage to retina. We exploit a combination of two novel ideas to simplify the constraints of traditional devices, with simplified optics and cleaver illumination in order to capture and visualize images of the retina in a standalone device easily operated by the user. Prototypes are conveniently embedded in either a mobile hand-held retinal camera, or wearable eyeglasses.

  • Reading through closed books: THz time-gated spectral imaging for content extraction through layered structures

    Justin Romberg, Ramesh Raskar, Barmak Heshmat Dehkordi, Mingjie Zhang and Salman Naqvi

    We exploit the sub-picosecond time resolution along with spectral resolution provided by terahertz time-domain spectroscopy to extract occluding content from layers whose thicknesses are wavelength comparable. The method uses the statistics of the THz E-field at subwavelength gaps to lock into each layer position and then uses a time-gated spectral kurtosis to tune to highest spectral contrast of the content on that specific layer. To demonstrate, occluding textual content was successfully extracted from a sample similar to a closed book down to nine pages without human supervision. The method provides over an order of magnitude enhancement in the signal contrast and can impact inspection of structural defects in wooden objects, plastic components, composites, drugs, and especially cultural artifacts with subwavelength or wavelength comparable layers.

  • Reflectance Acquisition Using Ultrafast Imaging

    Ramesh Raskar and Nikhil Naik

    We demonstrate a new technique that allows a camera to rapidly acquire reflectance properties of objects "in the wild" from a single viewpoint, over relatively long distances and without encircling equipment. This project has a wide variety of applications in computer graphics, including image relighting, material identification, and image editing.

  • Second Skin: Motion Capture with Actuated Feedback for Motor Learning

    Ramesh Raskar, Kenichiro Fukushi, Nikhil Naik, Christopher Schonauer and Jan Zizka

    We have created a 3D motion-tracking system with automatic, real-time vibrotactile feedback and an assembly of photo-sensors, infrared projector pairs, vibration motors, and a wearable suit. This system allows us to enhance and quicken the motor learning process in a variety of fields such as healthcare (physiotherapy), entertainment (dance), and sports (martial arts).

  • Shield Field Imaging

    Jaewon Kim
    We present a new method for scanning 3D objects through a single-shot, shadow-based method. We decouple 3D occluders from 4D illumination using shield fields: the 4D attenuation function which acts on any light field incident on an occluder. We then analyze occluder reconstruction from cast shadows, leading to a single-shot light-field camera for visual hull reconstruction.
  • Single Lens Off-Chip Cellphone Microscopy

    Ramesh Raskar, Aydin Arpa and Nikhil Naik

    Within the last few years, cellphone subscriptions have spread widely and now cover even the remotest parts of the planet. Adequate access to healthcare, however, is not widely available, especially in developing countries. We propose a new approach to converting cellphones into low-cost scientific devices for microscopy. Cellphone microscopes have the potential to revolutionize health-related screening and analysis for a variety of applications, including blood and water tests. Our optical system is more flexible than previously proposed mobile microscopes, and allows for wide field-of-view panoramic imaging, the acquisition of parallax, and coded background illumination, which optically enhances the contrast of transparent and refractive specimens.

  • Single-Photon Sensitive Ultrafast Imaging

    Special Interest group(s): 
    Ramesh Raskar and Barmak Heshmat Dehkordi

    The ability to record images with extreme temporal resolution enables a diverse range of applications, such as time-of-flight depth imaging and characterization of ultrafast processes. Here we present a demonstration of the potential of single-photon detector arrays for visualization and rapid characterization of events evolving on picosecond time scales. The single-photon sensitivity, temporal resolution, and full-field imaging capability enables the observation of light-in-flight in air, as well as the measurement of laser-induced plasma formation and dynamics in its natural environment. The extreme sensitivity and short acquisition times pave the way for real-time imaging of ultrafast processes or visualization and tracking of objects hidden from view.

  • Skin Perfusion Photography

    Special Interest group(s): 
    Guy Satat and Ramesh Raskar

    Skin and tissue perfusion measurements are important parameters for diagnosis of wounds and burns, and for monitoring plastic and reconstructive surgeries. In this project, we use a standard camera and a laser source in order to image blood-flow speed in skin tissue. We show results of blood-flow maps of hands, arms, and fingers. We combine the complex scattering of laser light from blood with computational techniques found in computer science.

  • Slow Display

    Daniel Saakes, Kevin Chiu, Tyler Hutchison, Biyeun Buczyk, Naoya Koizumi and Masahiko Inami

    How can we show our 16-megapixel photos from our latest trip on a digital display? How can we create screens that are visible in direct sunlight as well as complete darkness? How can we create large displays that consume less than 2W of power? How can we create design tools for digital decal application and intuitive-computer aided modeling? We introduce a display that is high-resolution but updates at a low frame rate: a slow display. We use lasers and monostable light-reactive materials to provide programmable space-time resolution. This refreshable, high-resolution display exploits the time decay of monostable materials, making it attractive in terms of cost and power requirements. Our effort to repurpose these materials involves solving underlying problems in color reproduction, day-night visibility, and optimal time sequences for updating content.

  • Smartphone spectrometer for food sensing

    Anshuman Das and Ramesh Raskar

    A smartphone based spectrometer design that is standalone and supported on a wireless platform. The device is low-cost and the power consumption is minimal making it portable to perform a range of studies in the field. Essential components of the device like the light source, spectrometer, filters, microcontroller and wireless circuits have been assembled in a housing that fits into a pocket and the entire device weighs 48 g. The device has a dedicated app on the smartphone to communicate, receive, plot and analyze spectral data. Validations of the device were carried out by demonstrating non-destructive ripeness testing in fruits. Ultra-Violet fluorescence from Chlorophyll present in the skin was measured across various apple varieties during the ripening process and correlated with destructive firmness tests. This demonstration is a step towards possible consumer, bio-sensing and diagnostic applications that can be carried out in a rapid manner.

  • SpeckleSense

    Alex Olwal, Andrew Bardagjy, Jan Zizka and Ramesh Raskar

    Motion sensing is of fundamental importance for user interfaces and input devices. In applications where optical sensing is preferred, traditional camera-based approaches can be prohibitive due to limited resolution, low frame rates, and the required computational power for image processing. We introduce a novel set of motion-sensing configurations based on laser speckle sensing that are particularly suitable for human-computer interaction. The underlying principles allow these configurations to be fast, precise, extremely compact, and low cost.

  • SpecTrans: Classification of Transparent Materials and Interactions

    Munehiko Sato, Alex Olwal, Boxin Shi, Shigeo Yoshida, Atsushi Hiyama, Michitaka Hirose and Tomohiro Tanikawa, Ramesh Raskar

    Surface and object recognition is of significant importance in ubiquitous and wearable computing. While various techniques exist to infer context from material properties and appearance, they are typically neither designed for real-time applications nor for optically complex surfaces that may be specular, textureless, and even transparent. These materials are, however, becoming increasingly relevant in HCI for transparent displays, interactive surfaces, and ubiquitous computing. We present SpecTrans, a new sensing technology for surface classification of exotic materials, such as glass, transparent plastic, and metal. The proposed technique extracts optical features by employing laser and multi-directional, multi-spectral LED illumination that leverages the material's optical properties. The sensor hardware is small in size, and the proposed classification method requires significantly lower computational cost than conventional image-based methods, which use texture features or reflectance analysis, thereby providing real-time performance for ubiquitous computing.

  • StreetScore

    Nikhil Naik, Jade Philipoom, Ramesh Raskar, Cesar Hidalgo

    StreetScore is a machine learning algorithm that predicts the perceived safety of a streetscape. StreetScore was trained using 2,920 images of streetscapes from New York and Boston and their rankings for perceived safety obtained from a crowdsourced survey. To predict an image's score, StreetScore decomposes this image into features and assigns the image a score based on the associations between features and scores learned from the training dataset. We use StreetScore to create a collection of map visualizations of perceived safety of street views from cities in the United States. StreetScore allows us to scale up the evaluation of streetscapes by several orders of magnitude when compared to a crowdsourced survey. StreetScore can empower research groups working on connecting urban perception with social and economic outcomes by providing high-resolution data on urban perception.

  • Tensor Displays: High-Quality Glasses-Free 3D TV

    Gordon Wetzstein, Douglas Lanman, Matthew Hirsch and Ramesh Raskar

    Tensor displays are a family of glasses-free 3D displays comprising all architectures employing (a stack of) time-multiplexed LCDs illuminated by uniform or directional backlighting. We introduce a unified optimization framework that encompasses all tensor display architectures and allows for optimal glasses-free 3D display. We demonstrate the benefits of tensor displays by constructing a reconfigurable prototype using modified LCD panels and a custom integral imaging backlight. Our efficient, GPU-based NTF implementation enables interactive applications. In our experiments we show that tensor displays reveal practical architectures with greater depths of field, wider fields of view, and thinner form factors, compared to prior automultiscopic displays.

  • The Next 30 Years of VR

    Ramesh Raskar and Barmak Heshmat Dehkordi

    In this visual brainstorming, we present the next 30 years of VR in a set of concept designs.

  • Theory Unifying Ray and Wavefront Lightfield Propagation

    George Barbastathis, Ramesh Raskar, Belen Masia, Nikhil Naik, Se Baek Oh and Tom Cuypers
    This work focuses on bringing powerful concepts from wave optics to the creation of new algorithms and applications for computer vision and graphics. Specifically, ray-based, 4D lightfield representation, based on simple 3D geometric principles, has led to a range of new applications that include digital refocusing, depth estimation, synthetic aperture, and glare reduction within a camera or using an array of cameras. The lightfield representation, however, is inadequate to describe interactions with diffractive or phase-sensitive optical elements. Therefore we use Fourier optics principles to represent wavefronts with additional phase information. We introduce a key modification to the ray-based model to support modeling of wave phenomena. The two key ideas are "negative radiance" and a "virtual light projector." This involves exploiting higher dimensional representation of light transport.
  • Time-of-Flight Microwave Camera

    Ramesh Raskar, Micha Feigin-Almon, Nikhil Naik, Andrew Temme and Gregory Charvat

    Our architecture takes a hybrid approach to microwaves and treats them like waves of light. Most other work places antennas in a 2D arrangement to directly sample the RF reflections that return. Instead of placing antennas in a 2D arrangment, we use a single, passive, parabolic reflector (dish) as a lens. Think of every point on that dish as an antenna with a fixed phase-offset. This means that the lens acts as a fixed set of 2D antennas which are very dense and spaced across a large aperture. We then sample the focal-plane of that lens. This architecture makes it possible for us to capture higher resolution images at a lower cost.

  • Towards In-Vivo Biopsy

    Guy Satat, Barmak Heshmat, Dan Raviv and Ramesh Raskar

    A new method to detect and distinguish between different types of fluorescent materials. The suggested technique has provided a dramatically larger depth range compared to previous methods; thus it enables medical diagnosis of body tissues without removing the tissue from the body, which is the current medical standard. It uses fluorescent probes, which are commonly used in medical diagnosis. One of these parameters is the fluorescence lifetime, that is the average time the fluorescence emission lasts. The new method can distinguish between different fluorescence lifetimes, which allows diagnosis of deep tissues. Locating fluorescence probes in the body using this method can, for example, indicate the location of a tumor in deep tissue, and classify it as malignant or benign according to the fluorescence lifetime, thus eliminating the need for X-ray or biopsy.

  • Trillion Frames Per Second Camera

    Andreas Velten, Di Wu, Adrián Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar

    We have developed a camera system that captures movies at an effective rate of approximately one trillion frames per second. In one frame of our movie, light moves only about 0.6 mm. We can observe pulses of light as they propagate through a scene. We use this information to understand how light propagation affects image formation and to learn things about a scene that are invisible to a regular camera.

  • Ultrasound Tomography

    Ramesh Raskar, Micha Feigin-Almon, Nikhil Naik and Brian Anthony

    Traditional medical ultrasound assumes that we are imaging ideal liquids. We are interested in imaging muscle and bone as well as measuring elastic properties of tissues, all of which are places where this assumption fails quite miserably. Interested in cancer detections, Duchenne muscular dystrophy, and prosthetic fitting, we use tomographic techniques as well as ideas from seismic imaging to deal with these issues.

  • Unbounded High Dynamic Range Photography Using a Modulo Camera

    Hang Zhao, Boxin Shi, Christy Fernandez-Cull, Sai-Kit Yeung and Ramesh Raskar

    We present a novel framework to extend the dynamic range of images called Unbounded High Dynamic Range (UHDR) photography with a modulo camera. A modulo camera could theoretically take unbounded radiance levels by keeping only the least significant bits. We show that with limited bit depth, very high radiance levels can be recovered from a single modulus image with our newly proposed unwrapping algorithm for natural images. We can also obtain an HDR image with details equally well preserved for all radiance levels by merging the least number of modulus images. Synthetic experiments and experiments with a real modulo camera show the effectiveness of the proposed approach.

  • VisionBlocks

    Chunglin Wen and Ramesh Raskar

    VisionBlocks is an on-demand, in-browser, customizable, computer-vision application-building platform for the masses. Even without any prior programming experience, users can create and share computer vision applications. End-users drag and drop computer vision processing blocks to create their apps. The input feed could be either from a user's webcam or a video from the Internet. VisionBlocks is a community effort where researchers obtain fast feedback, developers monetize their vision applications, and consumers can use state-of-the-art computer vision techniques. We envision a Vision-as-a-Service (VaaS) over-the-web model, with easy-to-use interfaces for application creation for everyone.