Hologram Anything: 3D Illusion Experiments

When I was re-archiving my old projects, I came across an autostereogram demo I made a while ago. An autostereogram, commonly known as "magic eye" pictures, is an 2D image that, when you look at it with a thousand-yard-stare, turns into 3D. I was never able to see them, until a couple years ago, when I suddenly mastered the trick. Excited, I made this demo that turns an interactive 3D WebGL scene into autostereogram.

As I stare blankly at the demo minute after minute, my love of 3D illusions gradually started to rekindle. Autostereograms are so weirdly awesome: after you're "locked in" to the right focus, you can move your gaze and look at the scene from different angles, as if a portal to another world has opened in the middle of your screen. What's even cooler is that it does't require any special equipment to view! However the limitations are also obvious: texturing is limited (you need to have a repeating pattern), depth resolution isn't that high, and most sadly, not all people can master the wall-eyed stare.

What about anaglyphs? The red-cyan glasses cost a dime a pair, and the images are so easy to generate! I remember back in the days it was pretty cool, but I think nowadays VR headsets have taken over. Nevertheless it is perhaps the second-cheapest 3D illusion you can get. After playing with it for a while (using the 3D API of my new programming language), I ended up building an IDE that has syntax depth in place of syntax highlighting -- comments look further away, keywords look closer. And, you can use the IDE to write code that generates an anaglyph scene, which will show as the background of the text editor. Since the scene will be behind the zero parallax plane while all the text will be in front, the latter will look as if they're floating on top.

After I showed them to our group, amused by my experiments, my new collegue Yufeng pulled out a device that once belonged to my former collegue Char: the Looking Glass Display. It is a so called "lenticular" or "light-field" or "holographic" 3D display. They work the same way as those lenticular prints: at each angle you're seeing a different image due to the refraction of light through an intricate layer of lenses. If these images happen to be renderings of a same 3D object from different angles, you get a 3D illusion of that object floating behind the display.

And most examples and programs I see use the display that way -- to show some objects floating in 3D. However, I think it has more potential. What if the object morph into another object when you look at it from a different angle? What if the object subtly change? What if you get to play some sort of animation or game where your viewing angle is the control? You see, after all, we live in a 3D world where, it is easy to forget that, everything is 3D! So it is perhaps more exciting to have something that we won't normally see in the real world.

The Looking Glass display works like an HDMI monitor. You can simply drag windows onto it, and, if your image is correctly interlaced, it'll appear 3D. It was quite a hassle to get it to do what I want however. Looking Glass's libraries assume that you're working with some 3D scene, and "nicely" wrap everything for you: you just provide the virtual camera, some 3D objects, and boom! hologram! But what I really want is simply a function that turns a bunch of images into a lenticular/interlaced image to be manually put on the display.

It took me a while to figure out the function. The interlacing shader needs to be extremely exact, so even though I generally understood the math, and have plugged in the device-specific callibration, even a minor difference in computation would make the result look garbled instead of 3D. I narrowed the problem down to the formulae with which the callibration values are processed before being passed into the main shader computation. But what is it? At this time of need, I turned to my friend (ChatGPT), who promptly started hallucinating. Finally, I dug out a piece of source code in one of Looking Glass's repos. The magic formula dictates that the so named "pitch" is not in fact "pitch", but in fact `pitch = pitch * width / DPI * cos(atan(1/slope))`. Whelp! No wonder neither me nor my friend could think of it. And everything just worked after that.

The first demo I made is an object that morphs into another when you look at it from left to right. When you view the display from the left, it looks like a coffee mug, and as you move your head to the right, it gradually turns into a donut. It is a visualization of the well-known joke, that a topologist cannot tell the difference between a coffee mug and a donut, as they both have a genus of one. The scene itself is implemented as a raymarched SDF, a technique popularized by Quilez and other geniuses on Shadertoy. It is quite a convenient technique for the task, as you get smoothed CSG (union of components) directly from SDF representation, and all I need to do is to scale the components over the axis. Later, I also made a demo morphing between "real" meshes, which was a quite arduous process in comparison (I had to design the meshes so that each vertex/UV on the mug correspond to a vertex/UV on the torus).

The idea of the second demo came from Char (who heard that I was making use of her old device): A face that always look at you no matter where you are. Initially I was planning to procedurally generate a cartoon face, but then had a better idea: I took a video of myself by moving my phone in front of me from left to right in constant speed, while looking at the camera all the time and keeping my head fixed. Then I simply took the frames of the video to use for the lenticular images. It worked! And it was very uncanny (as intended), though at some angles the effect is not perfect due to shaky hand. Yufeng named the piece "Creepy Asian Dude Looking at You", and Zach said he wanted one for his desk.

We showed the demos at member's week and it's been a hit, though some were disappointed that the displays are not manufactured by our group.

Afterwards I was thinking: "What if we have say, 50 cameras pointing at your face from different angles, then we can have a real-time hologram of your face on the Looking Glass!". But then I realized (besides that I don't have that kind of money) that I can utilize the latest machine learning models to infer depth from an RGB image, and create the lenticular images by rendering the depth-mesh from different angles.

For that I used the DepthAnything model, or more specifically the Apple CoreML port of it, and it worked well (and fast)! After I added some basic face-tracking, I now have a real-time hologram of myself in a nice little frame. Then I thought: this would be great for video calls! You would be able to see a hologram of the other person, and you could argue it adds more "presence" to an otherwise remote meeting. As I couldn't be bothered with figuring out how to make a plug-in for Zoom or FaceTime (I doubt it is possible anyways), I decided to simply run the depth neural net on a portion of my desktop, like a screen recording.

What I didn't expect is that I ended up having much more fun than a holographic zoom meeting. When the model calls itself "DepthAnything", it literally means "anything". So I ended up getting interesting depth definition for various GUIs, windows, icons on my screen. When I move a window showing a picture, it looks as if there's a relief sculpture on a wall; When I open up Google image search, it looks as if there're a hundred relief sculptures on a wall; When I open up a 3D modelling software, my 3D models pop into life. Now the real fun part is when you watch videos on YouTube. You essentially turn every video into 3D video!

Soon I was surrounded by an excited horde of friends and colleagues arguing over the next video to watch in 3D. Later we even had a movie night where the movie is played through the depth pipeline (we had to drop the Looking Glass part since the display is way too small). We seem to have reached a consensus that Pixar's Cars benefits the most from becoming 3D.

The depth model (or the CoreML port of it) seemed to be unfixably leaking memory. Other CoreML models don't have this problem, and no matter how I twiddle the janky auto-reference-count v.s. manual-release mechanism of Objective C, it just leaks anyways. I don't know if it is a bug in CoreML, or a problem with the model itself, but I think it's likely that the combination of this model + CoreML just don't go well with each other. You can avoid allocations all you like, but as soon as the inferencing starts, the unstoppable leak errupts like a fractured pipe, and it even managed to break Instruments (the leak-debugging app on mac). I have previously worked with many CoreML models, and double checked that none of them exhibit this problem when swapped in. Desperate, I implmented a desperate solution. Two identical subprocesses, each having a copy of the neural net, are spawn, and the input image is sent to the first subprocess via a FIFO pipe, after a few seconds (of accumulated leak), the first subprocess is forcibly killed, and tasks are sent to the second subprocess as the first is being ressurected. The main process continues alternating between killing the first subprocess and second subprocess, making sure each of them die before they could consume to much memory, while there's always a living process to process the next image in real time. This worked.