Object-Based Media
Changing storytelling, communication, and everyday life through sensing, understanding, and new interface technologies.
We explore the future of electronic visual communication and expression, and how the distribution of computational intelligence throughout video and audio communication systems can make a richer connection between the people at the ends of the systems, whether a broadcast system or a peer-to-peer environment. We also develop hardware and software technologies to support the requirements of such a scenario, with particular focus on new input and output technologies, advanced interfaces for consumer electronics, and self-organization among smart devices.

Research Projects

  • 3D Telepresence Chair

    Daniel Novy

    An autostereoscopic (no glasses) 3D display engine is combined with a "Pepper's Ghost" setup to create an office chair that appears to contain a remote meeting participant. The system geometry is also suitable for other applications, such as tabletop or automotive heads-up displays.

  • 4K/8K Comics

    V. Michael Bove and Daniel Novy

    4K/8K Comics applies the affordances of ultra-high-resolution screens to traditional print media such as comic books, graphic novels, and other sequential art forms. The comic panel becomes the entry point to the corresponding moment in the film adaptation, while scenes from the film indicate the source frames of the graphic novel. The relationships among comics, films, social media, parodies, and other support materials can be navigated using native touch screens, gestures, or novel wireless control devices. Big data techniques are used to sift, store, and explore vast catalogs of long-running titles, enabling sharing and remixing among friends, fans, and collectors.

  • 8K Time Machine

    Yukiko Oshio (NHK), V. Michael Bove, and Hisayuki Ohmata (NHK)

    Archived TV programs evoke earlier times. This application combines a video and music archive with an immersive screen and a simple user interface suitable for everyone, from children to the elderly, to create a "Time Machine" effect. The only key for exploring is the user's age. People can enjoy over 1,300 TV programs from the last seven decades without having to do tedious text searches. This catalogue intuitively guides the user with an image array (64 different videos on one screen at the same time) that simplifies navigation and makes it immediate, rather than referencing it to previous screens.

  • Aerial Light-Field Display

    V. Michael Bove, Daniel Novy and Henry Holtzman (Samsung NExD Lab)

    Suitable for anywhere a "Pepper's Ghost" display could be deployed, this display adds 3D with motion parallax, as well as optically relaying the image into free space such that gestural and haptic interfaces can be used to interact with it. The current version is able to display a person at approximately full-size.

  • BigBarChart

    V. Michael Bove and Laura Perovich

    BigBarChart is an immersive, 3D bar chart that provides a new physical way for people to interact with data. It takes data beyond visualizations to map out a new area--data experiences--that are multisensory, embodied, and aesthetic interactions. BigBarChart is made up of a number of bars that extend up to 10 feet to create an immersive experience. Bars change height and color in response to interactions that are direct (a person entering the room), tangible (pushing down on a bar to get meta information), or digital (controlling bars and performing statistical analyses through a tablet). BigBarChart helps both scientists and the general public understand information from a new perspective. Early prototypes are available.

  • Bottles&Boxes: Packaging with Sensors

    Ermal Dreshaj and Daniel Novy

    We have added inexpensive, low-power, wireless sensors to product packages to detect user interactions with products. Thus, a bottle can register when and how often its contents are dispensed (and generate side effects, like causing a music player to play music when the bottle is picked up, or generating an automatic refill order when near-emptiness is detected). A box can understand usage patterns of its contents. Consumers can vote for their favorites among several alternatives simply by handling them more often.

  • Calliope

    V. Michael Bove Jr., Edwina Portocarrero and Ye Wang

    Calliope is the follow-up to the NeverEnding Drawing Machine. A portable, paper-based platform for interactive story making, it allows physical editing of shared digital media at a distance. The system is composed of a network of creation stations that seamlessly blend analog and digital media. Calliope documents and displays the creative process with no need to interact directly with a computer. By using human-readable tags and allowing any object to be used as material for creation, it offers opportunities for cross-cultural and cross-generational collaboration among peers with expertise in different media.

  • Consumer Holo-Video

    V. Michael Bove Jr., Bianca Datta, Sundeep Jolly, Nickolaos Savidis and Daniel Smalley (BYU)
    The goal of this project, building upon work begun by Stephen Benton and the Spatial Imaging group, is to enable consumer devices such as tablets, phones, or glasses to display holographic video images in real time, suitable for entertainment, engineering, telepresence, or medical imaging. Our research addresses real-time scene capture and transmission, computational strategies, display technologies, interaction models, and applications.
  • Dressed in Data

    V. Michael Bove and Laura Perovich

    This project steps beyond data visualizations to create data experiences. It aims to engage not only the analytic mind, but also the artistic and emotional self. In this project, chemicals found in people's bodies and homes are turned into a series of fashions. Quantities, properties, and sources of chemicals are represented through various parameters of the fashion, such as fabric color, textures, and sizes. Wearing these outfits allows people to live the data–to experience tangibly the findings from their homes and bodies. This is the first project in a series of works that seek to create aesthetic data experiences that prompt researchers and laypeople to engage with information in new ways.

  • DUSK

    Special Interest group(s): 
    V. Michael Bove, Bianca Datta and Ermal Dreshaj

    DUSK was created as part of the Media Lab's Advancing Wellbeing initiative (supported by the Robert Wood Johnson Foundation) to create private, restful spaces for people in the workplace. DUSK promotes a vision of a new type of "nap pod," where workers are encouraged to use the structure on a daily basis for regular breaks and meditation. The user is provided with the much-needed privacy to take a phone call, focus, or rest inside the pod for short periods during the day. The inside can be silent, or filled by binaural beats audio; pitch black, or illuminated by a sunlamp; whatever works for users to get the rest and relaxation needed to continue to be healthy and productive. DUSK is created with a parametric press-fit design, making it scalable and suitable for fabrication customizable on a per-user basis.

  • Emotive Materials

    V. Michael Bove and Bianca Datta

    The design process is no longer limited to one group of individuals, as number, level, and cost make tools ever more accessible. As we move towards tools that allow us to create our own materials, having a set of rules with which to evaluate, interpret, and design them will become increasingly important. One way of approaching this problem is by unpacking the ways in which materials create meaning. This project explores the more emotive aspects of materials, such as haptic responses to, cognitive evaluation of, and emotive perception of materials to understand how materials communicate meaning.The development of an effective methodology aims to lower the barriers of fabrication of engaging objects. By incorporating qualities that were not previously quantifiable, we aim to encourage a more interactive design process that allows for the production of experiences tailored to individual preference, and a framework for conversations around material issues.

  • EmotiveModeler: An Emotive Form Design CAD Tool

    V. Michael Bove and Philippa Mothersill

    Whether or not we're experts in the design language of objects, we have an unconscious understanding of the emotional character of their forms. EmotiveModeler integrates knowledge about our emotive perception of shapes into a CAD tool that uses descriptive adjectives as an input to aid both expert and novice designers in creating objects that can communicate emotive character.

  • Everything Tells a Story

    V. Michael Bove Jr., David Cranor and Edwina Portocarrero
    Following upon work begun in the Graspables project, we are exploring what happens when a wide range of everyday consumer products can sense, interpret into human terms (using pattern recognition methods), and retain memories, such that users can construct a narrative with the aid of the recollections of the "diaries" of their sporting equipment, luggage, furniture, toys, and other items with which they interact.
  • Free-Space Haptic Feedback for 3D Displays

    V. Michael Bove and Ali Shtarbanov

    What if you could not only see but also feel virtual objects as you interacted with them? This would enable richer and more realistic user experiences. We have designed a low-cost air-vortex generator to provide midair haptic feedback when a user touches virtual objects displayed on holographic, aerial, and other 3D displays. The system consists of a 3D-printed chamber and nozzle, five low-frequency transducers, and a custom-designed driver board. The air-vortex generator can provide localized haptic feedback to a range of over 100cm. With increased driving power and a more optimized nozzle design, this range could be extended to several meters.

  • Guided-Wave Light Modulator for Holographic Video

    V. Michael Bove Jr., Bianca Datta, Sunny Jolly, Nickolaos Savidis and Daniel Smalley (BYU)
    We are developing inexpensive, efficient, high-bandwidth light modulators based on lithium niobate guided-wave technology. These full-color modulators support hundreds of thousands of pixels per scan line, making them suitable for fixed or wearable holographic displays.
  • Infinity-by-Nine

    V. Michael Bove Jr. and Daniel Novy

    We are expanding the home-video viewing experience by generating imagery to extend the TV screen and give the impression that the scene wraps completely around the viewer. Optical flow, color analysis, and heuristics extrapolate beyond the screen edge, where projectors provide the viewer's perceptual vision with low-detail dynamic patterns that are perceptually consistent with the video imagery and increase the sense of immersive presence and participation. We perform this processing in real time using standard microprocessors and GPUs.

  • ListenTree: Audio-Haptic Display in the Natural Environment

    V. Michael Bove, Joseph A. Paradiso, Gershon Dublon and Edwina Portocarrero

    ListenTree is an audio-haptic display embedded in the natural environment. Visitors to our installation notice a faint sound emerging from a tree. By resting their heads against the tree, they are able to hear sound through bone conduction. To create this effect, an audio exciter transducer is weatherproofed and attached to the tree's roots, transforming it into a living speaker, channeling audio through its branches, and providing vibrotactile feedback. In one deployment, we used ListenTree to display live sound from an outdoor ecological monitoring sensor network, bringing a faraway wetland into the urban landscape. Our intervention is motivated by a need for forms of display that fade into the background, inviting attention rather than requiring it. We consume most digital information through devices that alienate us from our surroundings; ListenTree points to a future where digital information might become enmeshed in material.

  • Live Objects

    V. Michael Bove, Arata Miyamoto and Valerio Panzica La Manna

    A Live Object is a small device that can stream media content wirelessly to nearby mobile devices without an Internet connection. Live Objects are associated with real objects in the environment, such as an art piece in a museum, a statue in a public space, or a product in a store. Users exploring a space can discover nearby Live Objects and view content associated with them, as well as leave comments for future visitors. The mobile device retains a record of the media viewed (and links to additional content), while the objects can retain a record of who viewed them. Future extensions will look into making the system more social, exploring game applications such as media “scavenger hunts” built on top of the platform, and incorporating other types of media such as live and historical data from sensors associated with the objects.

  • Narratarium

    V. Michael Bove Jr., Fransheska Colon, Catherine Havasi, Katherine (Kasia) Hayden, Daniel Novy, Jie Qi and Robert H. Speer

    Narratarium augments printed and oral stories and creative play by projecting immersive images and sounds. We are using natural language processing to listen to and understand stories being told, and analysis tools to recognize activity among sensor-equipped objects such as toys, then thematically augmenting the environment using video and sound. New work addresses the creation and representation of audiovisual content for immersive story experiences and the association of such content with viewer context.

  • Networked Playscapes: Dig Deep

    V. Michael Bove and Edwina Portocarrero

    Networked Playscapes re-imagine outdoor play by merging the flexibility and fantastical of the digital world with the tangible, sensorial properties of physical play to create hybrid interactions for the urban environment. Dig Deep takes the classic sandbox found in children's playgrounds and merges it with the common fantasy of "digging your way to the other side of the world" to create a networked interaction in tune with child cosmogony.

  • Pillow-Talk

    V. Michael Bove Jr., Edwina Portocarrero and David Cranor

    Pillow-Talk is the first of a series of objects designed to aid creative endeavors through the unobtrusive acquisition of unconscious, self-generated content to permit reflexive self-knowledge. Composed of a seamless recording device embedded in a pillow, and a playback and visualization system in a jar, Pillow-Talk crystallizes that which we normally forget. This allows users to capture their dreams in a less mediated way, aiding recollection by priming the experience and providing no distraction for recall and capture through embodied interaction.

  • Printed Wearable Holographic Display

    V. Michael Bove, Bianca Datta, Sunny Jolly, Nickolaos Savidis and Daniel Smalley (BYU)

    Holographic displays offer many advantages, including comfort and maximum realism. In this project we adapt our guided-wave light-modulator technology to see-through lenses to create a wearable 3D display suitable for augmented or virtual reality applications. As part of this work we also are developing a femtosecond-laser-based process that can fabricate the entire device by "printing."

  • Programmable Synthetic Hallucinations

    V. Michael Bove and Daniel Novy

    We are creating consumer-grade appliances and authoring methodologies that will allow hallucinatory phenomena to be programmed and utilized for information display and narrative storytelling.

  • ShAir: A Platform for Mobile Content Sharing

    Yosuke Bando, Daniel Dubois, Konosuke Watanabe, Arata Miyamoto, Henry Holtzman, and V. Michael Bove

    ShAir is a platform for instantly and easily creating local content-shareable spaces without requiring an Internet connection or location information. ShAir-enabled devices can opportunistically communicate with other mobile devices and optional pervasive storage devices such as WiFi SD cards whenever they enter radio range of one another. Digital content can hop through devices in the background without user intervention. Applications that can be built on top of the platform include ad-hoc photo/video/music sharing and distribution, opportunistic social networking and games, digital business card exchange during meetings and conferences, and local news article-sharing on trains and buses.

  • Slam Force Net

    V. Michael Bove Jr., Santiago Alfaro and Daniel Novy

    A basketball net incorporates segments of conductive fiber whose resistance changes with degree of stretch. By measuring this resistance over time, hardware associated with this net can calculate force and speed of a basketball traveling through the net. Applications include training, toys that indicate the force and speed on a display, dunk competitions, and augmented-reality effects on television broadcasts. This net is far less expensive and more robust than other approaches to measuring data about the ball (e.g., photosensors or ultrasonic sensors) and the only physical change required for the hoop or backboard is electrical connections to the net. Another application of the material is a flat net that can measure velocity of a ball hit or pitched into it (as in baseball or tennis); it can measure position as well (e.g., for determining whether a practice baseball pitch would have been a strike).

  • Smell Narratives

    Carol Rozendo and V. Michael Bove

    We are adding an olfactory dimension to storytelling in order to create more immersive and evocative experiences. Smell Narratives allows the authoring of a "smell track," involving individual or proportionally mixed fragrance components.

  • SurroundVision

    V. Michael Bove Jr. and Santiago Alfaro
    Adding augmented reality to the living-room TV, we are exploring the technical and creative implications of using a mobile phone or tablet (and possibly also dedicated devices like toys) as a controllable "second screen" for enhancing television viewing. Thus, a viewer could use the phone to look beyond the edges of the television to see the audience for a studio-based program, to pan around a sporting event, to take snapshots for a scavenger hunt, or to simulate binoculars to zoom in on a part of the scene. Recent developments include the creation of a mobile device app for Apple products and user studies involving several genres of broadcast television programming.
  • Thermal Fishing Bob: In-Place Environmental Data Visualization

    V. Michael Bove, Laura Perovich, Don Blair and Sara Wiley (Northeastern University)

    Two of the most important traits of environmental hazards today are their invisibility and the fact that they are experienced by communities, not just individuals. Yet we don't have a good way to make hazards like chemical pollution visible and intuitive. The thermal fishing bob seeks to visceralize rather than simply visualize data by creating a data experience that makes water pollution data present. The bob measures water temperature and displays that data by changing color in real time. Data is also logged to be physically displayed elsewhere and can be further recorded using long-exposure photos. Making environmental data experiential and interactive will help both communities and researchers better understand pollution and its implications.