Research Group Projects and Descriptions

The Affective Computing research group aims to bridge the gap between computational systems and human emotions. Our research addresses machine recognition and modeling of human emotional expression, machine learning of human preferences as communicated by user affect, intelligent computer handling of human emotions, computer communication of affective information between people, affective expression in machines and computational toys, emotion modeling for intelligent machine behavior; tools to help develop human social-emotional skills, and new sensors and devices to help gather, communicate, and express emotional information.

As members of different groups within the world, we accumulate knowledge that influences how we see, interpret, and, therefore, understand the world. This knowledge can lead to miscommunication and misunderstandings among groups. Many have endeavored to reduce misunderstandings by bringing different groups into contact with one another; however, these meetings do not guarantee that groups will have authentic opportunities to learn from, let alone understand, one another. People must be able to go through the process of analyzing and developing an empathetic eye toward themselves and others—a sort of reflection that we are not often trained to do. This project explores how physiological data can index media content that can guide discussions about diversity and personal experience in order to develop intergroup understanding.

Recent findings in affective neuroscience and psychology indicate that human affect and emotional experience play a significant and useful role in human learning and decision-making. Most machine-learning and decision-making models, however, are based on old, purely cognitive models, and are slow, brittle, and awkward to adapt. We aim to redress many of these classic problems by developing new models that integrate affect with cognition. Ultimately, such improvements will allow machines to make smarter and more human-like decisions for better human-machine interaction.

Alumni Contributor(s): Alea Teeters

Companies would like more new products to be successful in the marketplace; however, current evaluation methods such as focus groups do not accurately predict customer decisions. We are developing new technology-assisted methods to try to improve the customer evaluation process and better predict customer decisions. The new methods involve multi-modal affective measures (such as facial expression and skin conductance) together with behavioral measures, anticipatory-motivational measures, and self-report cognitive measures. These measures are combined into a novel computational model, the form of which is motivated by findings in affective neuroscience and human behavior. The model is being trained and tested with customer product evaluations and marketplace outcomes from real product launches.

People who have difficulty communicating verbally (such as many people with autism) sometimes send nonverbal messages that do not match what is happening inside them. For example, a child might look calm and receptive to learning, while having a heart rate of over 120 bpm and being on the verge of a meltdown or shutdown. This mismatch can lead to serious problems, including misunderstandings such as "he became aggressive for no reason." We are creating new technologies to address this fundamental communication problem and enable the first long-term, ultra-dense longitudinal data analysis of emotion-related physiological signals. We hope to equip individuals with personalized tools to understand the regulatory influences of emotion on their own state (e.g., "what state helps me best maintain my attention and focus for learning?"), and also enable scientists to accurately measure and understand the role of emotion regulation in autism.

Social-emotional communication difficulties lie at the core of autism spectrum disorders, making interpersonal interactions overwhelming, frustrating, and stressful. We are developing the world’s first wearable affective technologies to help the growing number of individuals with autism–approximately 1 in 150 children in the United States–learn about nonverbal communication in a natural, social context. We are also developing technologies that build on the nonverbal communication that individuals on the autism spectrum are already using to express themselves, to help families, educators, and other persons who deal with autism spectrum disorders to better understand these alternative means of nonverbal communication. Our work leverages advances in affect sensing and perception to (1) develop technologies that are sensitive to people's affective-cognitive states; (2) advance autism research; and (3) create new technologies that enhance the social-emotional communications of people diagnosed with autism, as well as those who are not.

Alumni Contributor(s): Alea Teeters

To help people improve their reading of faces during natural conversations, we developed a video tool to evaluate this skill. First, we collected over 100 videos of conversations between pairs of both autistic and neurotypical people, each of whom wore a Self-Cam. Next, the videos were manually segmented into chunks of 7-20 seconds according to expressive content, labeled, and sorted by difficulty—all tasks we plan to automate using technologies under development. Next, we built a rating interface including videos of self, peers, familiar adults, strangers, and unknown actors, allowing for performance comparisons across conditions of familiarity and expression. We obtained reliable identification (by coders) of categories of smiling, happy, interested, thinking, and unsure in the segmented videos. The tool was finally used to assess recognition of these five categories for eight neurotypical and five autistic people. Results show some autistics approaching the abilities of the neurotypicals while several score just above random.

Alumni Contributor(s): Alea Teeters

While modern communication technologies mean that we can connect to more people, these connections lack the affective subtleties inherent in situated interactions. EyeJacking is an application for the sharing of experiences in which one or more persons “eyejack” a person’s visual field to share what he or she sees. Using a wearable camera/micorphone system, remote interaction partners can share an experience first-hand and play an active role in shaping the experience. We explore the application of EyeJacking as a tool for situated learning for individuals on the autism spectrum, where parents, caregivers, or peers could “eyejack” and tag the world remotely. We also explore the application of EyeJacking to leverage the power of the masses to bootstrap people-sense abilities in robots.

Sometimes learners have to focus while experiencing strong emotions (e.g., family problems). They may also face challenges in perservering when encountering repeated failures in problem solving. The ability to know what one is feeling (e.g., worried, frustrated) and rise above it and handle the situation productively involves meta-affective skills. With such skills, a learner feeling "I can't do this; I want to quit," might instead think, "I am frustrated, but this is OK—it happens to experts. I should look for a different way to solve this." This research develops theory and technology to help learners develop meta-affective skills. Two recent achievements are development of (1) a technology with machine "common-sense" emotion—reasoning for enabling teenage girls to reflect on emotions in stories that they've constructed and improve their affect awareness; and (2) a technology to help students become stronger learners even when they feel like quitting.

Alumni Contributor(s): Winslow Burleson and Seth Raphael

In this research, a proactive emotional health system, geared toward supporting emotional self-awareness and empathy, was built as a part of a long-term research plan for understanding the role digital technology can play in helping people to reflect on their beliefs, attitudes, and values. The system, G.I.R.L.S. (Girls Involved in Real-Life Sharing), allows users to reflect actively upon the emotions related to their situations through the construction of pictorial narratives. The system employs common-sense reasoning to infer affective content from the users' stories and support emotional reflection. Users of this new system were able to gain new knowledge and understanding about themselves and others through the exploration of authentic and personal experiences. Currently, the project is being turned into an online system for use by school counselors.

We are developing a wireless sensor platform that allows easy integration of wearable biosensors with various consumer products, such as an iPod or cell phone. This platform has many applications, including health monitoring for outpatients or eldercare, fitness products, and various types of interactive content (e.g., MP3 music, video) that respond to the wearer's health or mood. Initial applications include: (1) personalized relapse-prevention messages for abstinent drug addicts, triggered by physiological craving signals; (2) mood-triggered music selections; and (3) a personal monitor for understanding the influence of autonomic arousal in autism.

This project develops new theory and algorithms to enable computers to make rapid and accurate inferences from multiple modes of data, such as determining a person's affective state from multiple sensors—video, mouse behavior, chair pressure patterns, typed selections, physiology, and more. Recent efforts focus on understanding the level of a person's attention, which is useful for things such as determining when to interrupt. Our approach is Bayesian: formulating probabilistic models on the basis of domain knowledge and training data, and then performing inference according to the rules of probability theory. This type of sensor fusion work is especially challenging because of the problems of sensor channel drop-out, different kinds of noise in different channels, dependence between channels, scarce and sometimes inaccurate labels, and patterns to detect that are inherently time-varying. We have constructed a variety of new algorithms for solving these problems and demonstrated their performance gains over other state-of-the-art methods.

Alumni Contributor(s): Ashish Kapoor

People on the autism spectrum face a number of challenges, including motor movement issues that can cause limbs to cease activity. Circumstantial evidence suggests that autonomic nervous system influences related to stress and overload may arise from and contribute to these problems. We propose to allow individuals to monitor several physiological parameters to see if there are patterns that recognize or predict the onset of their individual motor problems. We plan to develop new, wearable technology to treat these problems via the use of tiny, vibrotactile devices carefully placed at the joints. We hypothesize that some methods of touch-feedback and vibration at the joints may enable individuals to recover motor functioning during episodes of intermittent loss. We are also exploring the development of personally controlled devices that facilitate finer motor movement for augmenting communication as needed for assisting in typing or pointing.

A robotic computer that moves its monitor "head" and "neck," but that has no explicit face, is being designed to interact with users in a natural way for applications such as learning, rapport-building, interactive teaching, and posture improvement. In all these applications, the robot will need to move in subtle ways that express its state and promote appropriate movements in the user, but that don't distract or annoy. Toward this goal, we are giving the system the ability to recognize states of the user and also to have subtle expressions.

Alumni Contributor(s): Ashish Kapoor and Karen Liu

The Self-Cam is a wearable camera apparatus that consists of a chest-mounted camera aimed at the wearer’s face. Self-Cam was designed to be used in conjunction with a belt-mounted computer and real-time mental-state inference software that can be used with visual, auditory, or tactile output as personal feedback for the wearer. As the camera faces inward, many privacy issues are avoided–only those who choose to wear the Self-cam appear in the recorded video. Head movement can be seen and analyzed alongside facial expressions because the system rests on the chest and the light, simple nature of the structure allows it to be worn without any physical discomfort. By wearing the Self-Cam, you can explore who you appear to be from the outside. The Self-Cam acts as an objective point of view that might help you to understand yourself in a different light.

Shybot is a personal mobile robot designed to both embody and elicit reflection on shyness behaviors. Shybot is being designed to detect human presence and familiarity from face detection and proximity sensing in order to categorize people as friends or strangers for interaction. Shybot also can reflect elements of the anxious state of its human companion through LEDs and a spinning propeller. We designed this simple social interaction to open up a new direction for intervention for children living with autism. We hope that from minimal social interaction, a child with autism or social anxiety disorders could reflect on and more deeply attain understanding about personal shyness behaviors, as a first step toward helping to make progress in developing greater capacity for complex social interactions.

Persons with autism often report extreme hypersensitivity to sound. Researchers believe this hypersensitivity may be related to the acoustic quality of the sound (e.g., its frequency, intensity, and duration), and the context within which it occurs. Our primary aim is to offer persons with autism more control over their acoustic environment, regardless of the context. We are developing new technology for autistic individuals that will allow real-time control over the intensity and frequency characteristics of everyday sounds. This technology will also offer persons with autism new ways to record and document the sounds they find particularly aversive. Psychophysiological sensors will also be incorporated to assess the role of arousal in auditory hypersensitivity.