Hiroshi Ishii, Sean Follmer, Daniel Leithinger, Samuel Luescher, Alex Olwal, Akimitsu Hogge and Jinha Lee
Recent research in 3D user interfaces has pushed in two directions: immersive graphics and actuated tangible shape displays. We seek their hybrid by thinking about physical material density as a parameter in 3D rendering. We want to explore how digital models, handles, and controls can be rendered either as virtual 3D graphics or dynamic physical shapes, and move fluidly and quickly between these states, allowing physical affordances to be rendered only when needed. We were inspired by the different states of water: solid, gas, and liquid. We view digital computation and models as liquid, which can be vaporized into mid-air graphics, or solidified into dynamic physical shape. We also investigate transitions between solid and gas: sublimation and vaporization. To explore this, we have implemented a system which combines an actuated shape display and a spatial augmented reality display. This system can render physical shapes and volumetric graphics, co-located in the same space. We explore interaction techniques and motivating demonstration applications to explore 3D interaction between these boundaries. We also present results of a user study showing that freehand interaction with a physical shape display with co-located graphics outperforms direct interaction with only 3D graphics through a wand.