Soft Toys with Computer Hearts:
Building Personal Storytelling Environments

Marina Umaschi
MIT Media Laboratory
20 Ames st.,E15-320R
Cambridge, MA 02139 USA
+1 617 253 7211


SAGE is an authoring tool that allows children to design their own wise storytellers to interact with. It explicitly aims to enable them to explore their inner world, as well as to learn about storytelling and technology. In order to foster emotional engagement and explore the integration of physical and computer interfaces, the sage storyteller w as embodied in a interactive stuffed animal.


Personal storytelling, authoring environments, physical interfaces, metaphorical objects, learning.


As part of the Things That Think Consortium at the MIT Media Lab, in the Gesture & Narrative Language Group we are studying how toys can be designed with psychosocial competencies, based on an understanding of human linguistic, cognitive and social abilities. In the light of this interest, the project SAGE Storytellers is intended to explore the s elf-awareness possibilities of a storytelling environment embodied in a soft stuffed animal.

SAGE (Storytelling Agent Generation Environment) is a conversational personal storytelling system that simulates a sage or wise person that offers a story related to speaker's experiences or concerns. The storyteller keeps his/her story free of explanations and leaves to the user the responsibility of making personal connections and interpretation s.

SAGE offers two levels of engagement with storytelling. The first level, "Interacting", allows children to converse with previously designed storytellers and it is fully implemented. The second level, "Authoring", allows children to program their own storytellers with a repertoire of stories and is currently under development.

The sage storyteller agent was embodied in a stuffed rabbit that takes different personalities according to the hat it wears. For example, by putting a Yin-yang hat, the toy behaves as a Taoist (see Fig. 1) and by replacing it with a "yarmulke", the toy becomes the disciple of a Rabbi that tells Hasidic stories. By attaching small sensors to di fferent clothing and by using the SAGE authoring level, kids can create their own characters to interact with.

Figure 1: The interactive rabbit is the disciple or assistant of the Taoist storyteller displayed on the screen


Children establish intimate relationships with their stuffed animals. The favorite toy, as described by Winnicot [1] "must seem... to give warmth, or to move, or to have texture, or to do something that seems to show it has vitality or reality of its own." The natural tendency to communicate in a deep level with soft objects is leveraged by adding computational abilities so the toy can be a good listener and give adequate linguistic and non-linguistic feedback [2]. Richness in communicational cues makes it easier for the user to open up and evoke emotional awareness.

The metaphorical properties of objects are essential elements to explore the world of narrative and personal storytelling. If much of our common cultural knowledge is structured as metaphoric models [3], we need to understand the objects around us and the metaphors they convey. In this direction, Dr. LegoHead [4] is a computational construction kit to build creatures out of high level Lego parts such as eyes and glasses. The creature behaves differently depending on how these parts are attached.

The physical constraints of the Lego bricks, well suited to build engineering and "boys-like" constructions, can be very limiting in other domains. In 1987, Druin [5] started to explore these questions by building a giant furry animal called Noobie. Her purpose was to create computer environments that do not deprive our senses a nd that emphasize emotional contact.


Building a computer toy that simulates a sage storyteller and appears to be a good listener involves the integration of two different domains: the physical object and the software.

Actuators and circuitry resistor values of hats placed on the toy's head and report the values to the Handy Board (*) have been given to the stuffed rabbit in order to: 1)move the ears from a straight up to a folded position in order to signal attention, 2) rotate shoulders in order to convey head and gaze direction, 3) light t he pupils of the eyes to show interest, 4) deliver audio from a speaker hidden behind the mouth, 5) read the resistor values of hats placed on the toy's head and report the values to the Handy Board [6]. This microcontroller interfaces, via a serial connection, the physical toy with the SAGE Storytellers software that resides i n an external Macintosh computer.

These features were motivated by the need to embed in the toy the mechanisms to give non-verbal as well as verbal feedback. For example, to show attention, when the rabbit becomes a listener, its ears go up. As part of the authoring experience, children might be able to program the physical behavioral responses of the toy as well as the verbal one s.

The software architecture is implemented on top of a programming language specially designed to allow children to create 1) the conversational flow and turn-taking rules, 2)the script followed by the storyteller and 3) the data-base of stories [7].

The system parses user discourse by using a part-of-speech tagger. It checks for keywords that describe the point being made by the user's story and in each turn it assigns higher scores to the most repeated nouns and verbs and expands them with related concepts found through WordNet [8], a semantic lexical reference system. Thi s process is done in order to match user's personal story with a traditional story in the data-base.


SAGE has been tested with three interfaces: 1)Silent Screen: a cartoon of the sage with its talking balloon and a balloon where user's input is displayed. Children can input information only through the keyboard. 2)Integrated Screen-Toy: a stuffed rabbit interacts with the user. It behaves as the sage's disciple and is in charge of conversing with the user. After collecting information, it calls the sage storyteller. The screen displays the sage's cartoon while canned recorded stories are played back. Input is still through the keyboard. 3)Multimodal Toy: there is no computer screen. The toy maintains the conversation with the kids. Since this version is not completed, a "wizard" simulates the lacking speech recognition and acknowledges the child's gaze, gestures and body language through the servo-mechanisms installed in the rabbit.

Current work is being done on analyzing results of the user study conducted with twelve 11 to 13 years old. The main questions being explored are: what differences in children's storytelling style are elicited by the different interfaces?, and what kind of interface make them feel emotionally engaged and help them open up? The study is aimed at ga ining deeper understanding of human communication in order to design computational soft toys that allow interesting interaction. The general goal is to identify better design principles to integrate physical and computer objects.

There is still a long way to go in both the research and the implementation aspects of the project. Effort is being made to build a toy that supports symmetry in the interaction and that "understands" not only what we say with our speech but also with our body. Children's natural tendency to see their favorite toys as "real" or alive needs to be a ugmented and not inhibited by technology.

Figure 2: Ten years old interacting with the disciple of the storyteller rabbi.


I thank my advisor Prof. Justine Cassell for her guidance, Prof. Mitchel Resnick, Fred Martin, Rick Borovoy and Hannes Vilhj‡lmsson for their support and collaboration, and the undergraduate research assistants, Ien, Inki, Anthony, Peter and Andy, for their coding. I am also grateful to Interval Research Corp. and specially Lee Felsenstein for "g iving life" to the stuffed rabbit.


1.Winnicott,D.W. Playing and Reality. London: Tavistock Publications, 1971.

2.Cassell, J. "The role of Gestures in stories as multiple participant frameworks". AAAI Spring Symposium: Interactive Story Systems, (Stanford, March 1995), 23-26.

3.Lakoff G. & M. Johnson Metaphors we live by. Chicago Press, 1980.

4.Borovoy, R. Genuine Object Oriented Programming. MIT Master's Thesis, 1996.

5.Druin, A. Building an alternative to the traditional computer terminal, MIT Master's Thesis, 1987.

6.Martin F.(1995) The Handy Board

7.Umaschi, M. "SAGE Storytellers: Learning about Identity, Language and Technology" in Proceedings of ICLS 96 (Chicago IL, July1996), AACE, 526-531.

8.Miller, Miller, G., Beckwith R., Fellbaum C. ,Gross D. & Miller K. (1993) Introduction to WordNet. (ftp: documentation @ claritz.


(*) The design and construction of the actuators and circuitry was done by Lee Felsenstein at Interval Research Corp. Interfacing with the Handy Board is being implemented at the MIT Media Laboratory.