By Stephanie Strom

When Professor Hiroshi Ishii stepped up to the podium to deliver the keynote speech at the 2019 ACM CHI Conference on Human Factors in Computing Systems in Glasgow, Scotland, the room was energized with the reach and impact of his work.

Ishii was the winner of this year’s ACM SIGCHI Lifetime Research Award, which is bestowed on individuals who have contributed in fundamental ways to the study of human-computer interaction. As a founder of tangible user interfaces (TUI), Ishii’s work has spawned rich research aimed at, in the words of Nicholas Negroponte, “transforming human-computer interaction from abstract mousings and keystrokes into hands-on engagement.”  

Over a nearly 25-year career at the Media Lab, Ishii has stewarded new technologies, radical ways of thinking, and unique design approaches through the rigorous process of academic inquiry and, in many cases, out into the real world. His prominence in the field is due not only to his own remarkable work, but also the indelible influence of his thinking and teaching—on his students, and on others in the field around the world.

Many of Ishii’s students in the Tangible Media group have gone on to lead labs at other academic institutions that build on the study of TUI and Ishii’s subsequent work on Radical Atoms, which together have contributed to new research on so-called Shape-Changing User Interfaces.

Building the world between digital and physical

Sean Follmer, who earned his PhD at the Media Lab in 2015 via the Tangible Media group, first encountered Ishii in Paul Dourish’s book, Where the Action Is, which featured Ishii as the driving force behind research into embodied interaction with thinking machines.

“I had been living between two worlds, on one side physical product design and mechanical engineering and on the other, interaction design and computer science,” recalled Follmer, who now directs the Stanford SHAPE Lab, which explores how humans might more physically and tangibly interact with digital information. “And all of a sudden, there was this clear connection between the two that Hiroshi poetically illustrated in his work—in his words, at the shore between the sea of digital computation and the terra firma of the physical world.”

By the time Follmer arrived at the Media Lab, Ishii had already articulated his next groundbreaking concept, Radical Atoms, which aimed to ensure that physical interfaces and materials could reconfigure as easily as pixels drawn on a screen. Together with two other Tangible Media students, Daniel Leithinger and Lining Yao, Follmer brought the Radical Atoms vision to life in reconfigurable and shape-changing interfaces in projects like inFORM, and in the approaches that are now bringing soft robotics to human-computer interactions. Likewise, Leithinger and Yao, now researchers at the University of Colorado and Carnegie Mellon University, respectively, are carrying on the Radical Atoms vision in their own work; Leithinger on shape-changing displays, and Yao on morphing matter.

Follmer and other alumni of the Tangible Media group cite Ishii’s dedication to not only excellence in engineering and computer science, but also to excellence in art and design. “I was interested in making things that could go out and live in the world immediately, that were practical, but he would always push me away from that,” said Brygg Ullmer, who was among Ishii’s first students at the Media Lab, and now chairs the Human-Centered Computer Division at Clemson University. “How do you buff and refine and improve the inner core of whatever it was we were working on? He demanded that of us, he lived and breathed it.”

Art, design, and delight

Both Follmer and Ullmer note Ishii’s strong aesthetic sensibility, recalling his project musicBottles, an early work in which Ishii used bottles and corks to deliver the weather, news, and music to his mother. A bottle that opened to chirping noise meant a nice day, while other bottles contained the music of a jazz trio. “Hearing Hiroshi's story of building the music bottles for his mother transcends research, and becomes immersive poetry,” Follmer said. “Hiroshi's story, full of emotion, materials (artful blown glass), sound, light, and a little magic is 10,000 times more powerful than reading a research paper, even now, 20 years after he created it.”

In fact, many of his students have populated art departments around the world. Joanna Berzowska, the associate dean of research faculty of fine arts at Concordia University in Montreal, Canada, was already at the Media Lab working with another adviser when Ishii arrived in 1995.

She recalls very little talk of tangible or physical interfaces at the time. Rather, most professors at the Lab were working on graphic interfaces or backend algorithm innovation.

“Hiroshi was the first one who arrived and started talking about the physical world as a playground for digital interaction,” Berzowska said. “I found it so interesting, the idea that we could use our hands and eyes to manipulate not just screens or a mouse, but that every component of our physical world could become augmented.”

As an artist, she had found MIT to be “very technological,” in general. Ishii, she said, sought to incorporate the arts in everything he did. “Hiroshi certainly pushed the boundaries of technology but always in conversation with artists and musicians, people from other disciplines. He understands the importance that cultural industries, including art, play in human experience.”

Excellence in all things

Ishii’s current and former students describe him as a perfectionist, pushing them to not only create a project, but to be able to explain it well.

Berzowska said that insistence on quality and depth meant the work coming out of Ishii’s lab was always very high-caliber and led to published papers in highly respected publications. “He expected students to do really cool stuff but also to think about how to create a systematic approach to this new field he had invented,” she said. “That’s part of why he’s been so influential—he didn’t just push for cool hacks, he also pushed us to create a framework for how to think about this whole area of tangible interaction.”

This commitment to excellence also attracted the support of companies ranging from Bose, Ford, and Toshiba to NTT Data, Ferrero, and LEGO. LEGO, for instance, collaborated with the Tangible Media group in creating the Record and Play set of toys, which was launched in 2003. The toys were inspired by curlybot, a toy that records and plays back physical motion, using the repetition of gestures to create expressive patterns.

“The notion of turning everyday objects into interfaces or carriers of information was very inspiring to us,” Erik Hansen, tech innovation director at LEGO, wrote in an email.

Through dozens of students, hundreds of projects, and an infinite well of inspiration, Ishii has blazed a trail through entirely new territory in human-computer interaction, combining science, technology, design, and art in unforeseen and unique ways. From this trail are endless offshoots forged by his mentees and colleagues, explorations into yet stranger and more challenging undiscovered worlds.

And as for the road ahead? In his speech accepting the CHI award, Ishii said he sees no specific path ahead of him. Rather, he said, “I charge forward, and a road emerges behind me.”