Tangible Media
Seamlessly coupling the worlds of bits and atoms by giving dynamic physical form to digital information and computation.
We live between two worlds: our physical environment and cyberspace. The Tangible Media group's focus is on the design of seamless interfaces between humans, digital information, and the physical environment. People have developed sophisticated skills for sensing and manipulating our physical environments. However, most of these skills are not employed by traditional GUIs (Graphical User Interfaces). The Tangible Media group is designing a variety of "tangible interfaces" based on these skills by giving physical form to digital information, seamlessly coupling the dual worlds of bits and atoms. The goal is to change the "painted bits" of GUIs to "tangible bits," taking advantage of the richness of multimodal human senses and skills developed through our lifetime of interaction with the physical world.

Research Projects

  • Andante

    Xiao Xiao and Hiroshi Ishii

    Andante is a representation of music as animated characters walking along the piano keyboard that appear to play the physical keys with each step. Based on a view of music pedagogy that emphasizes expressive, full-body communication early in the learning process, Andante promotes an understanding of the music rooted in the body, taking advantage of walking as one of the most fundamental human rhythms.

  • inFORM

    Hiroshi Ishii, Alex Olwal, Daniel Leithinger and Sean Follmer

    Shape displays can be used to render both 3D physical content and user interface elements. We propose to use shape displays in three different ways to mediate interaction: facilitate, providing dynamic physical affordances through shape change; restrict, guiding users through dynamic physical constraints; and manipulate, actuating passive physical objects on the interface surface. We demonstrate this on a new, high-resolution shape display.

  • jamSheets: Interacting with Thin Stiffness-Changing Material

    Jifei Ou, Lining Yao, Daniel Tauber, Juergen Steimle, Ryuma Niiyama, Hiroshi Ishii

    This project introduces layer jamming as an enabling technology for designing deformable, stiffness-tunable, thin sheet interfaces. Interfaces that exhibit tunable stiffness properties can yield dynamic haptic feedback and shape deformation capabilities. In comparison to the particle jamming, layer jamming allows for constructing thin and lightweight form factors of an interface. We propose five layer structure designs and an approach which composites multiple materials to control the deformability of the interfaces. We also present methods to embed different types of sensing and pneumatic actuation layers on the layer-jamming unit. Through three application prototypes we demonstrate the benefits of using layer jamming in interface design. Finally, we provide a survey of materials that have proven successful for layer jamming.

  • MirrorFugue

    Xiao Xiao and Hiroshi Ishii

    MirrorFugue is an installation for a player piano that evokes the impression that the "reflection" of a disembodied pianist is playing the physically moving keys. Live music emanates from a grand piano, whose keys move under the supple touch of a pianist's hands reflected on the lacquered surface of the instrument. The pianist's face is displayed on the music stand, with subtle expressions projecting the emotions of the music. MirrorFugue recreates the feeling of a live performance, but no one is actually there. The pianist is an illusion of light and mirrors, a ghost both present and absent. Viewing MirrorFugue evokes the sense of walking into a memory, where the pianist plays without awareness of the viewer's presence; or, it is as if viewers were ghosts in another's dream, able to sit down in place of the performing pianist and play along.

  • Physical Telepresence

    Daniel Leithinger, Sean Follmer, Alex Olwal and Hiroshi Ishii

    We propose a new approach to physical telepresence, based on shared workspaces with the ability to capture and remotely render the shapes of people and objects. In this paper, we describe the concept of shape transmission, and propose interaction techniques to manipulate remote physical objects and physical renderings of shared digital content. We investigate, how the representation of user’s body parts can be altered to amplify their capabilities for teleoperation. A preliminary evaluation found users were able to manipulate simple objects remotely, and found many different techniques for manipulation that highlight the expressive nature of our system.

  • Pneumatic Shape-Changing Interfaces

    Hiroshi Ishii, Jifei Ou, Lining Yao, Ryuma Niiyama and Sean Follmer

    An enabling technology to build shape-changing interfaces through pneumatically driven soft composite materials. The composite materials integrate the capabilities of both input sensing and active shape output. We explore four applications: a multi-shape mobile device, table-top shape-changing tangibles, dynamically programmable texture for gaming, and shape-shifting lighting apparatus.

  • Radical Atoms

    Hiroshi Ishii
    Radical Atoms is our vision of interactions with future material. Radical Atoms takes a leap beyond Tangible Bits by assuming a hypothetical generation of materials that can change form and appearance dynamically, becoming as reconfigurable as pixels on a screen. Radical Atoms is a computationally transformable and reconfigurable material that is bidirectionally coupled with an underlying digital model (bits) so that dynamic changes of physical form can be reflected in digital states in real time, and vice versa.
  • Tangible Bits

    Hiroshi Ishii, Sean Follmer, Jinha Lee, Daniel Leithinger and Xiao Xiao
    People have developed sophisticated skills for sensing and manipulating our physical environments, but traditional GUIs (Graphical User Interfaces) do not employ most of them. Tangible Bits builds upon these skills by giving physical form to digital information, seamlessly coupling the worlds of bits and atoms. We are designing "tangible user interfaces" that employ physical objects, surfaces, and spaces as tangible embodiments of digital information. These include foreground interactions with graspable objects and augmented surfaces, exploiting the human senses of touch and kinesthesia. We also explore background information displays that use "ambient media"—light, sound, airflow, and water movement—to communicate digitally mediated senses of activity and presence at the periphery of human awareness. We aim to change the "painted bits" of GUIs to "tangible bits," taking advantage of the richness of multimodal human senses and skills developed through our lifetimes of interaction with the physical world.
  • THAW

    Sang-won Leigh, Philipp Schoessler, Hiroshi Ishii, Pattie Maes

    We present a novel interaction system that allows collocated screen devices to work together. The system tracks the position of a smartphone placed on a host computer screen. As a result, the smartphone can interact directly with data displayed on the host computer, which opens up a novel interaction space. We believe that the space on and above the computer screen will open up huge possibilities for new types of interactions. What makes this technology especially interesting is today’s ubiquity of smartphones and the fact that we can achieve the tracking solely through installing additional software on potentially any phone or computer.

  • TRANSFORM

    Hiroshi Ishii, Sean Follmer, Daniel Leithinger, Philipp Schoessler, Amit Zoran and LEXUS International

    TRANSFORM fuses technology and design to celebrate its transformation from still furniture to a dynamic machine driven by a stream of data and energy. TRANSFORM aims to inspire viewers with unexpected transformations and the aesthetics of the complex machine in motion. First exhibited at LEXUS DESIGN AMAZING MILAN (April 2014), the work comprises three dynamic shape displays that move over one thousand pins up and down in real time to transform the tabletop into a dynamic tangible display. The kinetic energy of the viewers, captured by a sensor, drives the wave motion represented by the dynamic pins. The motion design is inspired by dynamic interactions among wind, water, and sand in nature, Escher’s representations of perpetual motion, and the attributes of sand castles built at the seashore. TRANSFORM tells of the conflict between nature and machine, and its reconciliation, through the ever-changing tabletop landscape.