Affective Computing

Affect as Index is a tool that takes group physiological data as input, aggregates it across different demographic dimensions, and attaches them to media content. Users can review videotaped or prerecorded events by clicking on points of interest in a physiological graph. This software addresses two challenges: the difficulty of expressing and sharing emotions with others, and the laborious task of monitoring interpersonal interactions within natural settings. For the former, groups interested in discussing shared and dissimilar emotions evoked during experiences can use this tool to place context around their dialogue. For the latter, "meaningful moments" observed within natural interactions can be marked and superimposed on the physiological data collected. In this way, affect and observations of affect can be used to index group-level significant moments that occur within volumes of video data.

Recent findings in affective neuroscience and psychology indicate that human affect and emotional experience play a significant and useful role in human learning and decision-making. Most machine-learning and decision-making models, however, are based on old, purely cognitive models, and are slow, brittle, and awkward to adapt. We aim to redress many of these classic problems by developing new models that integrate affect with cognition. Ultimately, such improvements will allow machines to make smarter and more human-like decisions for better human-machine interaction.

Companies would like more new products to be successful in the marketplace, but current evaluation methods such as focus groups do not accurately predict customer decisions. We are developing new technology-assisted methods to try to improve the customer-evaluation process and better predict customer decisions. The new methods involve multi-modal affective measures (such as facial expression and skin conductance) together with behavioral measures, anticipatory-motivational measures, and self-report cognitive measures. These measures are combined into a novel computational model, the form of which is motivated by findings in affective neuroscience and human behavior. The model is being trained and tested with customer product evaluations and marketplace outcomes from real product launches.

Persons on the autism spectrum often report hypersensitivity to sound.
Efforts have been made to manage this condition, but there is wide room for improvement. One approach - exposure therapy – has promise, and a recent study showed that it helped several individuals diagnosed with autism overcome their sound sensitivities. In this project, we borrow principles from exposure therapy, and use fun, engaging, games to help individuals gradually get used to sounds that they might ordinarily find frightening or painful.

We are exploring innovative use of cell-phone Bluetooth technologies for consumer research and customer measurement. We have developed a small, portable, Bluetooth base station that can monitor consumer activity in a retail space and also enable new interactive services. This Bluetooth hub also serves as a network gateway for other wireless sensors in the local area.

People who have difficulty communicating verbally (such as many people with autism) sometimes send nonverbal messages that do not match what is happening inside them. For example, a child might look calm and receptive to learning, while having a heart rate of over 120 bpm and being on the verge of a meltdown or shutdown. This mismatch can lead to serious problems, including misunderstandings such as "he became aggressive for no reason." We are creating new technologies to address this fundamental communication problem and enable the first long-term, ultra-dense longitudinal data analysis of emotion-related physiological signals. We hope to equip individuals with personalized tools to understand the influences of their physiological state on their own behavior (e.g., "which state helps me best maintain my attention and focus for learning?"). Data from daily life will also advance basic scientific understanding of the role of autonomic nervous system regulation in autism.

Social-emotional communication difficulties lie at the core of autism spectrum disorders, making interpersonal interactions overwhelming, frustrating, and stressful. We are developing the world's first wearable affective technologies to help the growing number of individuals diagnosed with autism—approximately 1 in 150 children in the United States—learn about nonverbal communication in a natural, social context. We are also developing technologies that build on the nonverbal communication that individuals are already using to express themselves, to help families, educators, and other persons who deal with autism spectrum disorders to better understand these alternative means of nonverbal communication.

To help people improve their reading of faces during natural conversations, we developed a video tool to evaluate this skill. We collected over 100 videos of conversations between pairs of both autistic and neurotypical people, each wearing a Self-Cam. The videos were manually segmented into chunks of 7-20 seconds according to expressive content, labeled, and sorted by difficulty—all tasks we plan to automate using technologies under development. Next, we built a rating interface including videos of self, peers, familiar adults, strangers, and unknown actors, allowing for performance comparisons across conditions of familiarity and expression. We obtained reliable identification (by coders) of categories of smiling, happy, interested, thinking, and unsure in the segmented videos. The tool was finally used to assess recognition of these five categories for eight neurotypical and five autistic people. Results show some autistics approaching the abilities of neurotypicals while several score just above random.

We propose a set of customizable, easy-to-understand, and low-cost physiological toolkits in order to enable people to visualize and utilize autonomic arousal information. In particular, we aim for the toolkits to be usable in one of the most challenging usability conditions: helping individuals diagnosed with autism. This toolkit includes: wearable, wireless, heart-rate and skin-conductance sensors; pendant-like and hand-held physiological indicators hidden or embedded into certain toys or tools; and a customized software interface that allows caregivers and parents to establish a general understanding of an individual's arousal profile from daily life and to set up physiological alarms for events of interest. We are evaluating the ability of this externalization toolkit to help individuals on the autism spectrum to better communicate their internal states to trusted teachers and family members.

People express and communicate their mental states—such as emotions, thoughts, and desires—through facial expressions, vocal nuances, gestures, and other non-verbal channels. We have developed a computational model that enables real-time analysis, tagging, and inference of cognitive-affective mental states from facial video. This framework combines bottom-up, vision-based processing of the face (e.g., a head nod or smile) with top-down predictions of mental-state models (e.g., interest and confusion) to interpret the meaning underlying head and facial signals over time. Our system tags facial expressions, head gestures, and affective-cognitive states at multiple spatial and temporal granularities in real time and offline, in both natural human-human and human-computer interaction contexts. The system is being made available on multiple platforms, including portable devices. Applications range from measuring people's experiences to a training tool for autism spectrum disorders.

"Frame It" is an interactive tangible-digital puzzle game intended as a play-centered teaching and therapeutic tool. Current work is focused on the development of a social-signals puzzle game for children with Autism that will help them recognize social-emotional cues from information surrounding the eyes. In addition, we are investigating if this play-centered therapy results in the children become less averse to direct eye contact with others.

Emotions are often conveyed through gesture. Instruments that respond to gestures offer musicians new, exciting modes of musical expression. This project gives musicians wireless, gestural-based control over guitar effects parameters. For example, with this system, a guitarist can manipulate any style of pitch bending (from subtle vibrato, to whole step bends, to two-octave dive bombs) just by moving the headstock of the guitar.

In this research, a proactive emotional health system, geared toward supporting emotional self-awareness and empathy, was built as a part of a long-term research plan for understanding the role digital technology can play in helping people to reflect on their beliefs, attitudes, and values. The system, G.I.R.L.S. (Girls Involved in Real-Life Sharing), allows users to reflect actively upon the emotions related to their situations through the construction of pictorial narratives. The system employs common-sense reasoning to infer affective content from the users' stories and support emotional reflection. Users of this new system were able to gain new knowledge and understanding about themselves and others through the exploration of authentic and personal experiences. Currently, the project is being turned into an online system for use by school counselors.

We are developing wearable sensors that measure cardiovascular parameters such as heart rate and heart rate variability (HRV) in real time. HRV provides a sensitive index of autonomic nervous system activity. These sensors will be capable of communication with mobile devices such as the iPhone and iPod Touch.

We are developing a tiny wearable wireless sensor platform that allows comfortable, long-term sensing of physiological information coupled with low-cost connectivity to consumer devices including mobile phones and the XO laptop. This platform has many applications, including health monitoring for outpatients or the elderly, communication of affective information for people who are non-speaking or otherwise interested in sharing this information, education for individuals who want to learn about their own internal physiological changes during daily life, and customer experience data gathering in mobile situations.

We have been developing comfortable, safe, attractive physiological sensors that infants can wear around the clock to wirelessly communicate their internal physiological state changes. The sensors capture sympathetic nervous system arousal, temperature, physical activity, and other physiological indications that can be processed to signal changes in sleep, arousal, discomfort or distress, all of which are important for helping parents better understand the internal state of their child and what things stress or soothe their baby. The technology can also be used to collect physiological and circadian patterns of data in infants at risk for developmental disabilities.

This project develops new theory and algorithms to enable computers to make rapid and accurate inferences from multiple modes of data, such as determining a person's affective state from multiple sensors—video, mouse behavior, chair pressure patterns, typed selections, or physiology. Recent efforts focus on understanding the level of a person's attention, useful for things such as determining when to interrupt. Our approach is Bayesian: formulating probabilistic models on the basis of domain knowledge and training data, and then performing inference according to the rules of probability theory. This type of sensor fusion work is especially challenging due to problems of sensor channel drop-out, different kinds of noise in different channels, dependence between channels, scarce and sometimes inaccurate labels, and patterns to detect that are inherently time-varying. We have constructed a variety of new algorithms for solving these problems and demonstrated their performance gains over other state-of-the-art methods.

People on the autism spectrum face a number of challenges, including motor movement issues that can cause limbs to cease activity. Circumstantial evidence suggests that autonomic nervous system influences related to stress and overload may arise from and contribute to these problems. We propose allowing individuals to monitor several physiological parameters to see if there are patterns that recognize or predict the onset of their individual motor problems. We plan to develop new, wearable technology to treat these problems via the use of tiny, vibrotactile devices carefully placed at the joints. We hypothesize that some methods of touch-feedback and vibration at the joints may enable individuals to recover motor functioning during episodes of intermittent loss. We are also exploring the development of personally controlled devices that facilitate finer motor movement for augmenting communication as needed for assisting in typing or pointing.

How can technology help us understand ourselves better? Measuring the physiological arousal of children with sensory challenges such as ASD and ADHD, tools were developed to help children understand and control what makes them overexcited. Using iCalm hardware, children in therapy sessions measured their arousal while eating, throwing tantrums, playing in ball pits, and making challenging choices. Beyond progressive findings in the field of occupational therapy, this research is a basis for bio-information technology: tools to help children, their parents, and their teachers better understand what is going on in their bodies in a comfortable, affordable, and adaptable way. With future work, technology will be developed to help children understand and control their own internal state. In addition, this project will go beyond children’s therapy—helping adults in various settings including business and home life.

We have developed a low-cost device that can wirelessly detect a beating heart over a short distance (1m) and does not require any sensor placed on the person's body. This device can be used for wireless medical/health applications as well as security and safety applications, such as automobile/truck drivers as well as ATM machines. We have also created a small battery-powered version of this sensor that can be worn on a person's clothing but does not require touching the person's skin.

We have developed a novel market game, Prediction Game and Experience Sharing (PreGES, pronounced PreGuess), that harnesses people's collective prediction and experience sharing to forecast success or failure of new items (e.g., products, services, UI designs). Companies can register their new items on this market (as a testbed) to ask for collective opinions. In each PreGES trial session, participants makes their own best predictions on other people's overall opinions about the new items to get incentives (e.g., real opportunities to experience the items) and have fun in gambling-like games. As a participant’s guess (or portfolio) approaches the collective guess of all participants, he or she has a greater chance of winning an incentive. Participants improve the accuracy of their next prediction by sharing experiences. As participants have more trial sessions, their collective prediction converges into one common opinion (forecasting the success or failure of new items).

A robotic computer that moves its monitor "head" and "neck," but that has no explicit face, is being designed to interact with users in a natural way for applications such as learning, rapport-building, interactive teaching, and posture improvement. In all these applications, the robot will need to move in subtle ways that express its state and promote appropriate movements in the user, but that don't distract or annoy. Toward this goal, we are giving the system the ability to recognize states of the user and also to have subtle expressions.

A small number of studies support the notion of a functional relationship between movement stereotypy and arousal in individuals with ASD, such that changes in autonomic activity either precede or are a consequence of engaging in stereotypical motor movements. Unfortunately, it is difficult to generalize these findings as previous studies fail to report reliability statistics that demonstrate accurate identification of movement stereotypy start and end times, and use autonomic monitors that are obtrusive and thus only suitable for short-term measurement in laboratory settings. The current investigation further explores the relationship between movement stereotypy and autonomic activity in persons with autism by combining state-of-the-art ambulatory heart rate monitors to objectively assess arousal across settings; and wireless, wearable motion sensors and pattern recognition software that can automatically and reliably detect stereotypical motor movements in individuals with autism in real time.

SmileSeeker is a novel, machine-vision system that captures and provides quantified information about nonverbal communication where social interactions naturally happen. For example, in banking services, tellers observe facial expressions, head gestures, and eye gaze of customers, but this tool lets them both observe their own expressions and analyze how these interact with those of the customer to influence their mutual experience. The tool allows either real-time or offline feedback to help people reflect on what these interactions mean and determine how to elicit better experiences, such as true customer delight. The first deployment of this project focuses on eliciting and capturing smiles, and doing so in a way that is respectful of both customer and employee feelings. This project will also explore ways to share this information and link it to outcomes such as banking fee reductions or donations to charity.