Affective Computing
Advancing wellbeing using new ways to communicate, understand, and respond to emotion.
The Affective Computing group aims to bridge the gap between human emotions and computational technology. Current research addresses machine recognition and modeling of human emotional expression, including the invention of new software and hardware tools to help people gather, communicate, and express emotional information, together with tools to help people better manage and understand the ways emotion impacts health, social interaction, learning, memory, and behavior. Our projects are diverse: from inventing ways to help people who face communication and emotion regulation challenges; to enabling customers to give rich emotional feedback; to quantifying patterns of autonomic activity (core emotional physiology) during seizures, stress-related disorders, and sleep.

Research Projects

  • "Kind and Grateful": Promoting Kindness and Gratitude with Pervasive Technology

    Asma Ghandeharioun, Asaph Azaria, and Rosalind W. Picard

    We have designed a novel system to promote kindness and gratitude. We leverage pervasive technologies to naturally embed gratitude inspiration in everyday life. Mobile sensor data is utilized to infer optimal moments for stimulating contextually relevant thankfulness and appreciation. We analyze the interplay between mood, contextual cues, and gratitude expressions.

  • Affective Response to Haptic signals

    Grace Leslie, Rosalind Picard, Simon Lui, Suranga Nanayakkara

    This study attempts to examine humans' affective responses to superimposed sinusoidal signals. These signals can be perceived either through sound, in the case of electronically synthesized musical notes, or through vibro-tactile stimulation, in the case of vibrations produced by vibrotactile actuators. This study is concerned with the perception of superimposed vibrations, whereby two or more sinusoisal signals are perceived simultaneously, producing a perceptual impression that is substantially different than of each signal alone, owing to the interactions between perceived sinusoidal vibrations that give rise to a unified percept of a sinusoidal chord. The theory of interval affect was derived from systematic analyses of Indian, Chinese, Greek, and Arabic music theory and tradition, and proposes a universal organization of affective response to intervals organized using a multidimensional system. We hypothesize that this interval affect system is multi-modal and will transfer to the vibrotactile domain.

  • An EEG and Motion-Capture Based Expressive Music Interface for Affective Neurofeedback

    Grace Leslie, Rosalind Picard, and Simon Lui

    This project examines how the expression granted by new musical interfaces can be harnessed to create positive changes in health and wellbeing. We are conducting experiments to measure EEG dynamics and physical movements performed by participants who are using software designed to invite physical and musical expression of the basic emotions. The present demonstration of this system incorporates an expressive gesture sonification system using a Leap Motion device, paired with an ambient music engine controlled by EEG-based affective indices. Our intention is to better understand affective engagement, by creating both a new musical interface to invite it, and a method to measure and monitor it. We are exploring the use of this device and protocol in therapeutic settings in which mood recognition and regulation are a primary goal.

  • Automated Tongue Analysis

    Special Interest group(s): 
    Javier Hernandez Rivera, Weixuan 'Vincent' Chen, Akane Sano, and Rosalind W. Picard

    A common practice in Traditional Chinese Medicine (TCM) is visual examination of the patient's tongue. This study will examine ways to make this process more objective and to test its efficacy for understanding stress- and health-related changes in people over time. We start by developing an app that makes it comfortable and easy for people to collect tongue data in daily life together with other stress- and health-related information. We will obtain assessment from expert practitioners of TCM, and also use pattern analysis and machine learning to attempt to create state-of-the-art algorithms able to help provide better insights for health and prevention of sickness.

  • Automatic Stress Recognition in Real-Life Settings

    Special Interest group(s): 
    Rosalind W. Picard, Robert Randall Morris and Javier Hernandez Rivera

    Technologies to automatically recognize stress are extremely important to prevent chronic psychological stress and pathophysiological risks associated with it. The introduction of comfortable and wearable biosensors has created new opportunities to measure stress in real-life environments, but there is often great variability in how people experience stress and how they express it physiologically. In this project, we modify the loss function of Support Vector Machines to encode a person's tendency to feel more or less stressed, and give more importance to the training samples of the most similar subjects. These changes are validated in a case study where skin conductance was monitored in nine call center employees during one week of their regular work. Employees working in this type of setting usually handle high volumes of calls every day, and they frequently interact with angry and frustrated customers that lead to high stress levels.

  • Autonomic Nervous System Activity in Epilepsy

    Rosalind W. Picard and Ming-Zher Poh

    We are performing long-term measurements of autonomic nervous system (ANS) activity on patients with epilepsy. In certain cases, autonomic symptoms are known to precede seizures. Usually in our data, the autonomic changes start when the seizure shows in the EEG, and can be measured with a wristband (much easier to wear every day than wearing an EEG). We found that the larger the signal we measure on the wrist, the longer the duration of cortical brain-wave suppression following the seizure. The duration of the latter is a strong candidate for a biomarker for SUDEP (Sudden Unexpected Death in Epilepsy), and we are working with scientists and doctors to better understand this. In addition, bilateral changes in ANS activity may provide valuable information regarding seizure focus localization and semiology.

  • BrightBeat: An On-Screen Intervention for Regulating Breathing

    Asma Ghandeharioun, Rosalind Picard

    Deep and slow breathing techniques are components of various relaxation methods and have been used in treatment of many psychiatric and somatic disorders, as well as to improve mental function and attentiveness in healthy individuals. Many adults spend more than eight hours a day in front of screens, suggesting a need for an on-screen system to guide the user toward healthier breathing habits without requiring interruption. In this project, we explore the design and implementation of unobtrusive systems to promote healthier breathing habits.

  • Building the Just-Right-Challenge in Games and Toys

    Rosalind W. Picard and Elliott Hedman

    With the LEGO Group and Hasbro, we looked at the emotional experience of playing with games and LEGO bricks. We measured participants' skin conductance as they learned to play with these new toys. By marking the stressful moments, we were able to see what moments in learning should be redesigned. Our findings suggest that framing is key: how can we help children recognize their achievements? We also saw how children are excited to take on new responsibilities but are then quickly discouraged when they aren't given the resources to succeed. Our hope for this work is that by using skin conductance sensors, we can help companies better understand the unique perspective of children and build experiences fit for them.

  • EDA Explorer

    Sara Taylor, Natasha Jaques, Victoria Xia, and Rosalind W. Picard

    Electrodermal Activity (EDA) is a physiological indicator of stress and strong emotion. While an increasing number of wearable devices can collect EDA, analyzing the data to obtain reliable estimates of stress and emotion remains a difficult problem. We have built a graphical tool that allows anyone to upload their EDA data and analyze it. Using a highly accurate machine learning algorithm, we can automatically detect noise within the data. We can also detect skin conductance responses, which are spikes in the signal indicating a "fight or flight" response. Users can visualize these results and download files containing features calculated on the data to be used in their own analysis. Those interested in machine learning can also view and label their data to train a machine learning classifier. We are currently adding active learning, so the site can intelligently select the fewest possible samples for the user to label.

  • Fathom: Probabilistic Graphical Models to Help Mental Health Counselors

    Special Interest group(s): 
    Karthik Dinakar, Jackie Chen, Henry A. Lieberman, and Rosalind W. Picard

    We explore advanced machine learning and reflective user interfaces to scale the national Crisis Text Line. We are using state-of-the-art probabilistic graphical topic models and visualizations to help a mental health counselor extract patterns of mental health issues experienced by participants, and bring large-scale data science to understanding the distribution of mental health issues in the United States.

  • FEEL: A Cloud System for Frequent Event and Biophysiological Signal Labeling

    Yadid Ayzenberg and Rosalind W. Picard

    The wide availability of low-cost, wearable, biophysiological sensors enables us to measure how the environment and our experiences impact our physiology. This creates a new challenge: in order to interpret the collected longitudinal data, we require the matching contextual information as well. Collecting weeks, months, and years of continuous biophysiological data makes it unfeasible to rely solely on our memory for providing the contextual information. Many view maintaining journals as burdensome, which may result in low compliance levels and unusable data. We present an architecture and implementation of a system for the acquisition, processing, and visualization of biophysiological signals and contextual information.

  • IDA: Inexpensive Networked Digital Stethoscope

    Yadid Ayzenberg

    Complex and expensive medical devices are mainly used in medical facilities by health professionals. IDA is an attempt to disrupt this paradigm and introduce a new type of device: easy to use, low cost, and open source. It is a digital stethoscope that can be connected to the Internet for streaming physiological data to remote clinicians. Designed to be fabricated anywhere in the world with minimal equipment, it can be operated by individuals without medical training.

  • Large-Scale Pulse Analysis

    Special Interest group(s): 
    Weixuan 'Vincent' Chen, Javier Hernandez Rivera, Akane Sano and Rosalind W. Picard

    This study aims to bring objective measurement to the multiple "pulse" and "pulse-like" measures made by practitioners of Traditional Chinese Medicine (TCM). The measures are traditionally made by manually palpitating the patient's inner wrist in multiple places, and relating the sensed responses to various medical conditions. Our project brings several new kinds of objective measurement to this practice, compares their efficacy, and examines the connection of the measured data to various other measures of health and stress. Our approach includes the possibility of building a smartwatch application that can analyze stress and health information from the point of view of TCM.

  • Lensing: Cardiolinguistics for Atypical Angina

    Special Interest group(s): 
    Catherine Kreatsoulas (Harvard), Rosalind W. Picard, Karthik Dinakar, David Blei (Columbia) and Matthew Nock (Harvard)

    Conversations between two individuals--whether between doctor and patient, mental health therapist and client, or between two people romantically involved with each other--are complex. Each participant contributes to the conversation using her or his own "lens." This project involves advanced probabilistic graphical models to statistically extract and model these dual lenses across large datasets of real-world conversations, with applications that can improve crisis and psychotherapy counseling and patient-cardiologist consultations. We're working with top psychologists, cardiologists, and crisis counseling centers in the United States.

  • Mapping the Stress of Medical Visits

    Rosalind W. Picard and Elliott Hedman

    Receiving a shot or discussing health problems can be stressful, but does not always have to be. We measure participants' skin conductance as they use medical devices or visit hospitals and note times when stress occurs. We then prototype possible solutions and record how the emotional experience changes. We hope work like this will help bring the medical community closer to their customers.

  • Measuring Arousal During Therapy for Children with Autism and ADHD

    Rosalind W. Picard and Elliott Hedman

    Physiological arousal is an important part of occupational therapy for children with autism and ADHD, but therapists do not have a way to objectively measure how therapy affects arousal. We hypothesize that when children participate in guided activities within an occupational therapy setting, informative changes in electrodermal activity (EDA) can be detected using iCalm. iCalm is a small, wireless sensor that measures EDA and motion, worn on the wrist or above the ankle. Statistical analysis describing how equipment affects EDA was inconclusive, suggesting that many factors play a role in how a child's EDA changes. Case studies provided examples of how occupational therapy affected children's EDA. This is the first study of the effects of occupational therapy's in situ activities using continuous physiologic measures. The results suggest that careful case study analyses of the relation between therapeutic activities and physiological arousal may inform clinical practice.

  • Mobile Health Interventions for Drug Addiction and PTSD

    Rich Fletcher and Rosalind W. Picard

    We are developing a mobile phone-based platform to assist people with chronic diseases, panic-anxiety disorders, or addictions. Making use of wearable, wireless biosensors, the mobile phone uses pattern analysis and machine learning algorithms to detect specific physiological states and perform automatic interventions in the form of text/images plus sound files and social networking elements. We are currently working with the Veterans Administration drug rehabilitation program involving veterans with PTSD.

  • Modulating Peripheral and Cortical Arousal Using a Musical Motor Response Task

    Grace Leslie, Rosalind Picard, Simon Lui, Annabel Chen

    We are conducting EEG studies to identify the musical features and musical interaction patterns that universally impact measures of arousal. We hypothesize that we can induce states of high and low arousal using electrodermal activity (EDA) biofeedback, and that these states will produce correlated differences in concurrently recorded skin conductance and EEG data, establishing a connection between peripherally recorded physiological arousal and cortical arousal as revealed in EEG. We also hypothesize that manipulation of musical features of a computer-generated musical stimulus track will produce changes in peripheral and cortical arousal. These musical stimuli and programmed interactions may be incorporated into music technology therapy, designed to reduce arousal or increase learning capability by increasing attention. We aim to provide a framework for the neural basis of emotion-cognition integration of learning that may shed light on education and possible applications to improve learning by emotion regulation.

  • Objective Asessment of Depression and Its Improvement

    Special Interest group(s): 
    Rosalind W. Picard, Szymon Fedor, Brigham and Women's Hospital and Massachusetts General Hospital

    Current methods to assess depression and then ultimately select appropriate treatment have many limitations. They are usually based on having a clinician rate scales, which were developed in the 1960s. Their main drawbacks are lack of objectivity, being symptom-based and not preventative, and requiring accurate communication. This work explores new technology to assess depression, including its increase or decrease, in an automatic, more objective, pre-symptomatic, and cost-effective way using wearable sensors and smart phones for 24/7 monitoring of different personal parameters such as physiological data, voice characteristics, sleep, and social interaction. We aim to enable early diagnosis of depression, prevention of depression, assessment of depression for people who cannot communicate, better assignment of a treatment, early detection of treatment remission and response, and anticipation of post-treatment relapse or recovery.

  • Panoply

    Special Interest group(s): 
    Rosalind W. Picard and Robert Morris

    Panoply is a crowdsourcing application for mental health and emotional wellbeing. The platform offers a novel approach to computer-based psychotherapy, targeting accessibility without stigma, engagement, and therapeutic efficacy. A three-week randomized-controlled trial with 166 participants showed Panoply conferred greater or equal benefits for nearly every therapeutic outcome measure compared to an active control task (online expressive writing). Panoply significantly outperformed the control task also on all measures of engagement, and is now being commercialized at itskoko.com.

  • Predicting Students' Wellbeing from Physiology, Phone, Mobility, and Behavioral Data

    Natasha Jaques, Sara Taylor, Akane Sano, Ehi Nosakhare and Rosalind Picard

    The goal of this project is to apply machine learning methods to model the wellbeing of MIT undergraduate students. Extensive data is obtained from the SNAPSHOT study, which monitors participating students on a 24/7 basis, collecting data on their location, sleep schedule, phone and SMS communications, academics, social networks, and even physiological markers like skin conductance, skin temperature, and acceleration. We extract features from this data and apply a variety of machine learning algorithms including Gaussian Mixture Models and Multi-task Multi-Kernel Learning; we are currently working to apply Bayesian Hierarchical Multi-task Learning and Deep Learning as well. Interesting findings include: when participants visit novel locations they tend to be happier; when they use their phones or stay indoors for long periods they tend to be unhappy; and when several dimensions of wellbeing (including stress, happiness, health, and energy) are learned together, classification accuracy improves.

  • Real-Time Assessment of Suicidal Thoughts and Behaviors

    Special Interest group(s): 
    Rosalind W. Picard, Szymon Fedor, Harvard and Massachusetts General Hospital

    Depression correlated with anxiety is one of the key factors leading to suicidal behavior, and is among the leading causes of death worldwide. Despite the scope and seriousness of suicidal thoughts and behaviors, we know surprisingly little about what suicidal thoughts look like in nature (e.g., How frequent, intense, and persistent are they among those who have them? What cognitive, affective/physiological, behavioral, and social factors trigger their occurrence?). The reason for this lack of information is that historically researchers have used retrospective self-report to measure suicidal thoughts, and have lacked the tools to measure them as they naturally occur. In this work we explore use of wearable devices and smartphones to identify behavioral, affective, and physiological predictors of suicidal thoughts and behaviors.

  • SNAPSHOT Expose

    Miriam Zisook, Sara Taylor, Akane Sano and Rosalind Picard

    We are applying learnings from the SNAPSHOT study to the problem of changing behavior, exploring the design of user-centered tools which can harness the experience of collecting and reflecting on personal data to promote healthy behaviors--including stress management and sleep regularity. We draw on commonly used theories of behavior change as the inspiration for distinct conceptual designs for a behavior-change application based on the SNAPSHOT study. This approach will enable us to compare the types of visualization strategies that are most meaningful and useful for acting on each theory.

  • SNAPSHOT Study

    Special Interest group(s): 
    Akane Sano, Amy Yu, Sara Taylor, Cesar Hidalgo and Rosalind Picard

    The SNAPSHOT study seeks to measure Sleep, Networks, Affect, Performance, Stress, and Health using Objective Techniques. It is an NIH-funded collaborative research project between the Affective Computing and Macro Connections groups, and Harvard Medical School's Brigham & Women's hospital. Since fall 2013, we've run this study to collect one month of data every semester from 50 MIT undergraduate students who are socially connected. We have collected data from about 170 participants, totaling over 5,000 days of data. We measure physiological, behavioral, environmental, and social data using mobile phones, wearable sensors, surveys, and lab studies. We investigate how daily behaviors and social connectivity influence sleep behaviors and health, and outcomes such as mood, stress, and academic performance. Using this multimodal data, we are developing models to predict onsets of sadness and stress. This study will provide insights into behavioral choices for wellbeing and performance.

  • SPRING: A Smart Platform for Research, Intervention, & Neurodevelopmental Growth

    Kristy Johnson and Rosalind Picard

    SPRING is a custom-built hardware and software platform for children with neuro-differences. The system automates data acquisition, optimizes learning progressions, and encourages social, cognitive, and motor development in a positive, personalized, child-led play environment. The quantitative data and developmental trajectories captured by this platform enable systematic, mutli-modal, long-term studies of different therapeutic and educational approaches to autism and other developmental disorders, as well as a better understanding of motivation, engagement, and learning for the general population.

  • StoryScape

    Rosalind W. Picard and Micah Eckhardt

    Stories, language, and art are at the heart StoryScape. While StoryScape began as a tool to meet the challenging language learning needs of children diagnosed with autism, it has become much more. StoryScape was created to be the first truly open and customizable platform for creating animated, interactive storybooks that can interact with the physical world. Download the android app: https://play.google.com/store/apps/details?id=edu.mit.media.storyscape and make your own amazing stories at https://storyscape.io/.

  • The Challenge

    Special Interest group(s): 
    Natasha Jaques, Niaja Farve, Pattie Maes and Rosalind W. Picard

    Mental wellbeing is intimately tied to both social support and physical activity. The Challenge is a tool aimed at promoting social connections and decreasing sedentary activity in a workplace environment. Our system asks participants to sign up for short physical challenges and pairs them with a partner to perform the activity. Social obligation and social consensus are leveraged to promote participation. Two experiments were conducted in which participants’ overall activity levels were monitored with a fitness tracker. In the first study, we show that the system can improve users' physical activity, decrease sedentary time, and promote social connection. As part of the second study, we provide a detailed social network analysis of the participants, demonstrating that users’ physical activity and participation depends strongly on their social community.

  • The enTRAIN Study

    Rosalind W. Picard, Kristina Johnson and Northeastern University

    Individuals with autism are known to have difficulties connecting with other people, reciprocating social interactions, and being emotionally regulated by others. Yet, until recently, very little attention has been given to the way people interact together, in a system, rather than by themselves. We propose a new way to collect data on how caregivers and their children, with and without autism, affect and are affected by each other (i.e., how they "sync up" with one another), both in their behavior and in their physiology. We also introduce a customizable digital-physical smart toy platform that will allow us to test hypotheses and collect data about patterns of caregiver-child synchrony in a naturalistic and engaging environment. MIT and Northeastern are forging a new collaboration between smart toy technology and autism research that will help uncover how the social brain develops.

  • Tributary

    Yadid Ayzenberg, Rosalind Picard

    The proliferation of smartphones and wearable sensors is creating very large data sets that may contain useful information. However, the magnitude of generated data creates new challenges as well. Processing and analyzing these large data sets in an efficient manner requires computational tools. Many of the traditional analytics tools are not optimized for dealing with large datasets. Tributary is a parallel engine for searching and analyzing sensor data. The system utilizes large clusters of commodity machines to enable in-memory processing of sensor time-series signals, making it possible to search through billions of samples in seconds. Users can access a rich library of statistics and digital signal processing functions or write their own in a variety of languages.

  • Unlocking Sleep

    Rosalind W. Picard, Thariq Shihipar and Sara Taylor

    Despite a vast body of knowledge about the importance of sleep, our daily schedules are often planned around work and social events, not healthy sleep. While we're prompted throughout the day by devices and people to plan and think about our schedules in terms of things to do, sleep is rarely considered until we're tired and it's late. This project proposes a way that our everyday use of technology can help improve sleep habits. Smartphone unlock screens are an unobtrusive way of prompting user reflection throughout the day by posing "microquestions" as users unlock their phone. The questions are easily answered with a single-swipe. Since we unlock our phones 50 to 200 times per day, microquestions can collect information with minimal intrusiveness to the user’s daily life. Can these swipe-questions help users mentally plan their day around sleep, and trigger healthier sleep behaviors?

  • Valinor: Mathematical Models to Understand and Predict Self-Harm

    Special Interest group(s): 
    Rosalind W. Picard, Karthik Dinakar, Eric Horvitz (Microsoft Research) and Matthew Nock (Harvard)

    We are developing statistical tools for understanding, modeling, and predicting self-harm by using advanced probabilistic graphical models and fail-soft machine learning in collaboration with Harvard University and Microsoft Research.

  • Wavelet-Based Motion Artifact Removal for Electrodermal Activity

    Weixuan 'Vincent' Chen, Natasha Jaques, Sara Taylor, Akane Sano, Szymon Fedor and Rosalind W. Picard

    Electrodermal activity (EDA) recording is a powerful, widely used tool for monitoring psychological or physiological arousal. However, analysis of EDA is hampered by its sensitivity to motion artifacts. We propose a method for removing motion artifacts from EDA, measured as skin conductance (SC), using a stationary wavelet transform (SWT). We modeled the wavelet coefficients as a Gaussian mixture distribution corresponding to the underlying skin conductance level (SCL) and skin conductance responses (SCRs). The goodness-of-fit of the model was validated on ambulatory SC data. We evaluated the proposed method in comparison with three previous approaches. Our method achieved a greater reduction of artifacts while retaining motion-artifact-free data.