Speech + Mobility
Enhancing mobile life through improved user interactions.
We use speech technologies and portable devices to enhance human communication and make digitized audio more useful as a data type. Our focus is on developing novel applications, user interfaces, and services to exploit computer speech processing for interacting with and through computers far removed from keyboards and monitors.

Research Projects

  • Activ8

    Misha Sra and Chris Schmandt

    Activ8 is a system of three short games: See-Saw, a balancing game for Glass; Jump Beat, a music beat matching game for Glass; and Learning to Fly, a Kinect game where users keep a virtual bird in the air by flapping their arms. Recent epidemiological evidence points at sitting as being the most common contributor to an inactive lifestyle. We aim to offer a starting point towards designing and building an understanding about how "physical casual games" can contribute to helping address the perils of sitting.

  • Back to the Desktop

    Andrea Colaco, Hye Soo Yang, Chris Schmandt

    In this project, we construct a virtual desktop centered around the smartphone display with the surface around the display opportunistically used for input. We use a 3-pixel optical time-of-flight sensor, Mime, to capture hand motion. The sensor on the phone allows the table surface next to the phone to be mapped to conventional desktop windows, and the phone's display is a small viewport onto this desktop. Moving the hand is like moving the mouse, and as the user shifts into another part of the desktop, the phone viewport display moves with it. We demonstrate that instead of writing new applications to use smart surfaces, existing applications can be readily controlled with the hands.

  • Flickr This

    Chris Schmandt and Dori Lin
    Inspired by the fact that people are communicating more and more through technology, Flickr This explores ways for people to have emotion-rich conversations through all kinds of media provided by people and technology. By grounding them in shared media, the technology allows remote people to have conversations that are more like face-to-face experiences. Flickr This lets viewable content provide structure for a conversation; conversation can move between synchronous and asynchronous, and evolve into a richer collaborative conversation/media.
  • Glass Ear

    Dhruv Jain and Chris Schmandt

    Persons with hearing loss use visual signals such as gestures and lip movement to interpret speech. While hearing aids and cochlear implants can improve sound recognition, they generally do not help the wearer localize sound necessary to leverage these visual cues. We design and evaluate visualizations for spatially locating sound on a head-mounted display (HMD). After a large scale formative study and gathering preferences for specific design options, we implemented a real-time proof-of-concept HMD prototype on Google Glass and solicited feedback from deaf and hard of hearing individuals. Current developments are aiming towards creating a wearable prototype for field deployments.

  • Going My Way

    Chris Schmandt and Jaewoo Chung
    When friends give directions, they often don't describe the whole route, but instead provide landmarks along the way that they think will be familiar. Friends can assume we have certain knowledge because they know our likes and dislikes. Going My Way attempts to mimic a friend by learning about where you travel, identifying the areas that are close to the desired destination from your frequent path, and picking a set of landmarks to allow you to choose a familiar one. When you select one of the provided landmarks, Going My Way will provide directions from it to the destination.
  • Guiding Light

    Chris Schmandt, Jaewoo Chung, Ig-Jae Kim and Kuang Xu
    Guiding Light is a navigation-based application that provides directions by projecting them onto physical spaces both indoors and outdoors. It enables a user to get relevant spatial information by using a mini projector in a cell phone. The core metaphor involved in this design is that of a flashlight which reveals objects in and information about the space it illuminates. For indoor navigation, Guiding Light uses a combination of e-compass, accelerometer, proximity sensors, and tags to place information appropriately. In contrast to existing heads-up displays that push information into the user's field of view, Guiding Light works on a pull principle, relying entirely on users' requests and control of information.
  • InterTwinkles

    Chris Schmandt and Charlie DeTar

    Bringing deliberative process and consensus decision-making to the 21st century! A practical set of tools for assisting in meeting structure, deliberative process, brainstorming, and negotiation. Helping groups to democratically engage with each other, across geographies and time zones.

  • Live Trace

    Hye Soo Yang, Andrea Colaco and Chris Schmandt

    In this interactive experience we are interested in enabling quick input actions to Google Glass. The application allows users to trace an object or region of interest in their live view. We use the trace as the foundation for allowing the user to indicate interest in a visual region. Once selected, the user can choose to apply filters to the region, annotate the selection through speech input, or capture text through optical character recognition. These selection and processing tools could naturally integrate with quick note-taking applications where limited touchpad input precludes such input. The Live Trace app demonstrates the effectiveness of gestural control for head-mounted displays.

  • LocoRadio

    Chris Schmandt and Wu-Hsi Li

    LocoRadio is a mobile, augmented-reality, audio browsing system that immerses you within a soundscape as you move. To enhance the browsing experience in high-density spatialized audio environments, we introduce a UI feature, "auditory spatial scaling," which enables users to continuously adjust the spatial density of perceived sounds. The audio will come from a custom, geo-tagged audio database. The current demo uses iconic music to represent restaurants. As users move in the city, they encounter a series of pieces of music and the perception enhances their awareness of the numbers, styles, and locations of nearby restaurants.

  • Mime

    Andrea Colaco, Chris Schmandt

    Mime is a compact, low-power 3D sensor for short-range gestural control of small display devices. The sensor's performance is based on a novel signal processing pipeline that combines low-power time-of-flight (TOF) sensing for 3D hand-motion tracking with RGB image-based computer vision algorithms for finer gestural control. Mime is an addition to a growing number of input devices developed around the engineering design philosophy of sacrificing generality for battery-friendly and accurate performance to retain the portability advantages of our smart devices. We demonstrate the utility of Mime for head-mounted display control and smartphones with a variety of application scenarios, including 3D spatial input using close range gestures, gaming, on-the-move interaction, and operation in cluttered environments and in broad daylight conditions.

  • MugShots

    Cindy Hsin-Liu Kao and Chris Schmandt

    MugShots enables visual communication though everyday objects. We embed a small display into a coffee mug, an object with frequent daily use. Targeted for the workplace, the mug transitions between different communication modes in public and private spaces. In the private office space, the mug is an object for intimate communication between remote friends; users receive emoticon stickers via the display. When brought to a public area, the mug switches to a pre-selected image of the user's choice, serving as a social catalyst to trigger conversations in public spaces.

  • Musicpainter

    Chris Schmandt, Barry Vercoe and Wu-Hsi Li
    Musicpainter is a networked, graphical composing environment that encourages sharing and collaboration within the composing process. It provides a social environment where users can gather and learn from each other. The approach is based on sharing and managing music creation in small and large scales. At the small scale, users are encouraged to begin composing by conceiving small musical ideas, such as melodic or rhythmic fragments, all of which are collected and made available to all users as a shared composing resource. The collection provides a dynamic source of composing material that is inspiring and reusable. At the large scale, users can access full compositions that are shared as open projects. Users can listen to and change any piece. The system generates an attribution list on the edited piece, allowing users to trace how it evolves in the environment.
  • OnTheGo

    Misha Sra, Chris Schmandt

    As mobile device screens continue to get smaller (smartwatches, head-mounted devices like Google Glass), touch-based interactions with them become harder. With OnTheGo, our goal is to compliment touch and voice based input on these devices by adding interactions through in-air gestures around the devices. Gestural interactions are not only intuitive for certain situations where touch may be cumbersome like running, skiing, or cooking, but are also convenient for things like quick application and task management, certain types of navigation and interaction, and simple inputs to applications.

  • OnTheRun

    Chris Schmandt and Matthew Joseph Donahoe

    OnTheRun is a location-based exercise game designed for the iPhone. The player assumes the role of a fugitive trying to gather clues to clear his name. The game is played outdoors while running, creating missions that are tailored to the player's neighborhood and running ability. The game is primarily an audio experience, and gameplay involves following turn-by-turn directions, outrunning virtual enemies, and reaching destinations.

  • Pintail

    Chris Schmandt and Sujoy Kumar Chowdhury

    Pintail is a travel companion app for guided storytelling. It will start by capturing your travel plan so that it can nudge you with personalized story-creation triggers at the right context. Pintail would act as a work-in-progress scrapbook from the moment a trip is planned. It will provide users the structure and tools for storytelling while taking into account the short attention span of today's audience. Pintail would use priming as a technique by showing the user what others feel or have drawn about the places he or she is visiting. Some of the content of Pintail prompts would be automatically collected from travel review sites. Users can then use the Pintail story-creation tools to reflect and create their own stories. Pintail would also attempt to balance between the story creation activity and the actual trip experience.

  • Puzzlaef

    Chris Schmandt, Sinchan Banerjee, and Drew Harry

    How can one understand and visualize the lifestyle of a person on the other side of the world? Puzzlaef attempts to tackle this question through a mobile picture puzzle game, which users collaboratively solve with pictures from their lifestyles.

  • Radio-ish Media Player

    Chris Schmandt, Barry Vercoe and Wu-Hsi Li
    How many decisions does it take before you hear a desired piece of music on your iPod? First, you are asked to pick a genre, then an artist, then an album, and finally a song. The more songs you own, the tougher the choices are. To resolve these issues, we have turned the modern music player into an old analog radio tuner, the Radio-ish Media Player. No LCDs, no favorite channels, just a knob that will help you surf through channel after channel accompanied by synthesized noise. Radio-ish is our attempt to revive the lost art of channel surfing in the old analog radio tuner. Let music find you: your ears will tell you if the music is right. This project is not only a retrospective design, but also our reflection on lost simplicity in the process of digitalization. A mobile phone version is also available for demo.
  • ROAR

    Chris Schmandt and Drew Harry

    The experience of being in a crowd is visceral. We feel a sense of connection and belonging through shared experiences like watching a sporting event, speech, or performance. In online environments, though, we are often part of a crowd without feeling it. ROAR is designed to allow very large groups of distributed spectators to have meaningful conversations with strangers or friends while creating a sense of presence of thousands of other spectators. ROAR is also interested in creating opportunities for collective action among spectators and providing flexible ways to share content among very large groups. These systems combine to let you feel the roar of the crowd even if you're alone in your bedroom.

  • Spellbound

    Misha Sra and Chris Schmandt

    Spellbound is a mobile game that gets you outdoors. It explores the space between physical sports and the fantastical worlds of video games as a place to create new game dynamics around real-time mobile, social, and physically active digital play.

  • Spotz

    Chris Schmandt and Misha Sra

    Exploring your city is a great way to make friends, discover new places, find new interests, and invent yourself. Spotz is an Android app where everyone collectively defines the places they visit and the places in turn define them. Spotz allows you to discover yourself by discovering places. You tag a spot, create some buzz for it and, if everyone agrees the spot is 'fun' this bolsters your 'fun' quotient. If everyone agrees the spot is 'geeky' it pushes up your ‘geeky’ score. Thus emerges your personal tag cloud. Follow tags to chance upon new places. Find people with similar 'tag clouds' as your own and experience new places together. Create buzz for your favorite spots and track other buzz to find who has the #bestchocolatecake in town!

  • techNailogy

    Cindy Hsin-Liu Kao, Artem Dementyev, Chris Schmandt

    techNailogy is a nail-mounted gestural input surface. Using capacitive sensing on printed electrodes, the interface can distinguish on-nail finger swipe gestures with high accuracy. techNailogy works in real-time: we miniaturized the system to fit on the fingernail, while wirelessly transmitting the sensor data to a mobile phone or PC. techNailogy allows one-handed and always-available input, while being unobtrusive and discreet. Inspired by commercial nail stickers, the device blends into the user’s body, is customizable, fashionable, and even removable. We show example applications of using the device as a remote controller when hands are busy and using the system to increase the input space of mobile phones.

  • Tin Can

    Chris Schmandt, Matthew Donahoe and Drew Harry

    Distributed meetings present a set of interesting challenges to staying engaged and involved. Because one person speaks at a time, it is easy (particularly for remote participants) to disengage from the meeting undetected. However, non-speaking roles in a meeting can be just as important as speaking ones, and if we could give non-speaking participants ways to contribute, we could help support better-run meetings of all kinds. Tin Can collects background tasks like taking notes, managing the agenda, sharing relevant content, and tracking to-dos in a distributed interface that uses meeting participants' phones and laptops as input devices, and represents current meeting activities on an iPad in the center of the table in each meeting location. By publicly representing these background processes, we provide meeting attendees with new ways to contribute and be recognized for their non-verbal participation.

  • Tin Can Classroom

    Chris Schmandt, Drew Harry and Eric Gordon (Emerson College)

    Classroom discussions may not seem like an environment that needs a new kind of supporting technology. But we've found that augmenting classroom discussions with an iPad-based environment to help promote discussion, keep track of current and future discussion topics, and create a shared record of class keeps students engaged and involved with discussion topics, and helps restart the discussion when conversation lags. Contrary to what you might expect, having another discussion venue doesn't seem to add to student distraction; rather it tends to focus distracted students on this backchannel discussion. For the instructor, our system offers powerful insights into the engagement and interests of students who tend to speak less in class, which in turn can empower less-active students to contribute in a venue in which they feel more comfortable.