Time-folded optics

While the time resolutions or speed of camera sensors have been improving through advancement in electronics, the imaging optics of such cameras have not changed for a long time. Today the imaging optics of ultrafast cameras, time-of-flight cameras, and depth cameras are simply inherited from the conventional photography optics of low-speed cameras. Such optics first conceived by Ibn al-Haytham almost a thousand years ago have evolved in terms of performance and capability but not in terms of principle. This unaltered incorporation of optics has overshadowed the true potential of new emerging sensors; especially ultrafast and depth-sensitive sensors. Since time and space (or time and depth) are conveniently interchangeable at ultrafast shutter speeds (time range less than 1 ns), the question arises whether the design of imaging optics for a high speed camera or depth sensitive camera should follow the same rules as conventional low-speed cameras. Would it be possible to use time instead of space when arranging optics? Can an ultrafast sensor benefit from a time-coded set of optics, each with dramatically different optical functionality? These are the fundamental questions that need to be answered as we advance in the fabrication of ultrafast and ToF sensors. This was the motivation of this study to discover, explore, and understand the new principles of designing optics in time.

Cavities and how they convert time to space

Infinity mirrors appear to do the impossible, they appear to recede back forever into darkness. In case you have never experienced an infinity mirror, you have likely experienced a similar effect looking into a mirror with a second mirror behind you in an elevator or a barber shop. Your reflection repeats over and over, retreating back in space. The infinity mirror gives the illusion of an enlarged space, a small room now looks like an infinite hallway. This simple construct of two mirrors facing each other is known as a Fabry-Perot (FP) cavity and is the most basic cavity that one can conceive. If the mirrors are chosen to be semi-reflective then at each roundtrip that light makes between the facets some portion of the light escapes out. FP cavity evolves the wavefront of the light, that is, it does not change the angle of the light rays that are circulating inside it after each round trip. Therefore, every round trip that light makes inside the cavity it is as if the light has actually travelled a longer distance in space and this is how cavity converts time to space for entering photons.

Key idea

In its simplest form, we found a way to design optics in time dimension and while doing so we found that many unconventional capabilities can be realized by leveraging this dimension. This is done by introducing a cavity into conventional photography optics and using a fast or depth-sensitive sensor. The cavity is partially reflective, and partially transmissive (just like the infinity mirror we discussed in the introduction). Some of the light will pass right through the cavity to the sensor, some light will bounce in the cavity once, twice, etc times before reaching the sensor. You can imagine that the cavity is like an infinity mirror, repeating your reflection multiple times at different times. The key insight is that we can modify or evolve the captured wavefront during each roundtrip in the cavity and then record it at the "right time." But due to arrangement of optics this "right time" is equivalent to "right distance," "right color," "right zoom," "right focal length," or any other desired optical functionality that was previously achieved by having physical distances or arrangements in front of the sensor. The study explores and demonstrates few of these possibilities.

Compressing lenses by folding focal length in time

Think about the infinity mirror example, in that example we were able to increase the perceptive size of a small room. Now if we think about a camera lens, the lenses goal is focus the light entering the camera on a small sensor. In order to focus the light, the lens needs to be at a set distance from the sensor based on thin lens equation (1/f=1/o+1/i where f is focal length o is object to lens distance and i is image to lens distance). Imagine if we placed our infinity mirror in this space between the lens and the sensor. This would increase the perceptive distance between the lens and camera allowing the lense to be brought much closer to the sensor. This is exactly what we did, except instead of an infinity mirror, we placed a cavity consisting of two partially reflective mirrors between the focusing lens and the sensor. With each roundtrip in the cavity, the image become more and more focused. We can retrieve the focused image by only looking at the time frame associated with the distance the lens requires. In experiments we were able to decrease the lens-sensor distance or lens axial size by an order of magnitude (see publication link).

Multi-zoom capture enabled by time-folding

If you wanted to capture both a wide angle and zoomed in video of a scene you traditionally had two choices. You could either use two cameras, or crop your wide angle image after the fact at the cost of resolution. However, in time dimension a multi-zoom capture is possible by placing the focusing lens inside of the cavity. During each round trip inside of the cavity, the light will pass through the lens twice more, effectively changing the focal length (zoom and magnification factor). We call this optics a multi-zoom optics because depending on the time of the acquisition the magnification can drastically change. 

Multispectral imaging enabled by time-folding

In the previous two examples we had a cavity to alter the focusing aspects of the imaging system. It turns out we can also use cavities to alter other aspects of the system such as enabling multispectral imaging. Recall the the cavity repeats the signal on the sensor at a regular time interval. Imagine you are back inside the infinity mirror with a super fast clock in your hand but this time instead of having only two parallel half-mirrors that form the cavity you have three half mirrors. Now imagine that the half-mirrors behind you is a silver mirror and reflects all the colors but the first half-mirrors ahead of you reflects only red and is transparent to other colors, right after that red mirror there is another half-mirror that only reflects blue. Now what will happen when you have a colorful scene inside the cavity. When you look at your clock you will see that the red points of the scene will repeat with faster repetition rate than blue and this way you can correlate the time of repetition with color thus making a multispectral recording of the scene. 

There are many other applications and architectures that can branch from the concept of time folding. For example, one can sample the light-field evolving in the optical system using time-folding as seen in the image below, which shows how time-folding can realize ultrafast focal stack imaging. Each line is a cross section of the light passing through the optics. Frames are showing different x-t cross sections at different y values.

This technology is patented under: US Patent B. Heshmat, M. Tancik Time-folded imaging; Methods and Apparatus for Imaging Using Optical Cavity, 15682145, (2017).

If you have a new applications and would like to license this technology please contact Daniel Dardani at MIT technology licensing office: ddardani@mit.edu