We are examining autonomic sleep patterns using a wrist-worn biosensor that enables comfortable measurement of skin conductance, skin temperature, and motion. The skin conductance reflects sympathetic arousal. We are looking at sleep patterns in healthy groups, in groups with autism, and in groups with sleep disorders. We are looking especially at sleep quality and at performance on learning and memory tasks.

Technologies to automatically recognize stress, are extremely important to prevent chronic psychological stress and the pathophysiological risks associated to it. The introduction of comfortable and wearable biosensors have created new opportunities to measure stress in real-life environments, but there is often great variability in how people experience stress and how they express it physiologically. In this project, we modify the loss function of Support Vector Machines to encode a person's tendency to feel more or less stressed, and give more importance to the training samples of the most similar subjects. These changes are validated in a case study where skin conductance was monitored in nine call center employees during one week of their regular work. Employees working in this type of settings usually handle high volumes of calls every day, and they frequently interact with angry and frustrated customers that lead to high stress levels.

Efforts to build emotionally responsive forms of artificial intelligence have been hampered by many difficulties, not least of which include the challenges of natural language processing. Although there have been many gains in this domain, it is still difficult to build technologies that offer nuanced forms of emotional support. To address these challenges, researchers might look towards human computation – an approach that harnesses the power of large, distributed online communities to solve artificial intelligence problems that might otherwise be intractable. We present a new technological approach that uses human computation algorithms, in conjunction with on-demand online workforces, to provide expedient emotional support.

People who have difficulty communicating verbally (such as many people with autism) sometimes send nonverbal messages that do not match what is happening inside them. For example, a child might appear calm and receptive to learning—but have a heart rate over 120 bpm and be about to meltdown or shutdown. This mismatch can lead to misunderstandings such as "he became aggressive for no reason." We are creating new technologies to address this fundamental communication problem and enable the first long-term, ultra-dense longitudinal data analysis of emotion-related physiological signals. We hope to equip individuals with personalized tools to understand the influences of their physiological state on their own behavior (e.g., "which state helps me best maintain my attention and focus for learning?"). Data from daily life will also advance basic scientific understanding of the role of autonomic nervous system regulation in autism.

To help people improve their reading of faces during natural conversations, we developed a video tool to evaluate this skill. We collected over 100 videos of conversations between pairs of both autistic and neurotypical people, each wearing a Self-Cam. The videos were manually segmented into chunks of 7-20 seconds according to expressive content, labeled, and sorted by difficulty—all tasks we plan to automate using technologies under development. Next, we built a rating interface including videos of self, peers, familiar adults, strangers, and unknown actors, allowing for performance comparisons across conditions of familiarity and expression. We obtained reliable identification (by coders) of categories of smiling, happy, interested, thinking, and unsure in the segmented videos. The tool was finally used to assess recognition of these five categories for eight neurotypical and five autistic people. Results show some autistics approaching the abilities of neurotypicals while several score just above random.

We propose a set of customizable, easy-to-understand, and low-cost physiological toolkits in order to enable people to visualize and utilize autonomic arousal information. In particular, we aim for the toolkits to be usable in one of the most challenging usability conditions: helping individuals diagnosed with autism. This toolkit includes: wearable, wireless, heart-rate and skin-conductance sensors; pendant-like and hand-held physiological indicators hidden or embedded into certain toys or tools; and a customized software interface that allows caregivers and parents to establish a general understanding of an individual's arousal profile from daily life and to set up physiological alarms for events of interest. We are evaluating the ability of this externalization toolkit to help individuals on the autism spectrum to better communicate their internal states to trusted teachers and family members.

We present the first project analyzing facial expressions over the internet. The interface analyzes the participants' smile intensity as they watch popular commercials. They can compare their responses to an aggregate from the larger population. The system also allows us to crowd-source data for training expression recognition systems.

Frame It is an interactive, blended, tangible-digital puzzle game intended as a play-centered teaching and therapeutic tool. Current work is focused on the development of a social-signals puzzle game for children with autism that will help them recognize social-emotional cues from information surrounding the eyes. In addition, we are investigating if this play-centered therapy results in the children becoming less averse to direct eye contact with others. The study uses eye-tracking technology to measure gaze behavior while participants are exposed to images and videos of social settings and expressions. Results indicate that significant changes in expression recognition and social gaze are possible after repeated uses of the Frame It game platform.

We have been developing comfortable, safe, attractive physiological sensors that infants can wear around the clock to wirelessly communicate their internal physiological state changes. The sensors capture sympathetic nervous system arousal, temperature, physical activity, and other physiological indications that can be processed to signal changes in sleep, arousal, discomfort or distress, all of which are important for helping parents better understand the internal state of their child and what things stress or soothe their baby. The technology can also be used to collect physiological and circadian patterns of data in infants at risk for developmental disabilities.

Physiological arousal is an important part of occupational therapy for children with autism and ADHD, but therapists do not have a way to objectively measure how therapy affects arousal. We hypothesize that when children participate in guided activities within an occupational therapy setting, informative changes in electrodermal activity (EDA) can be detected using iCalm. iCalm is a small, wireless sensor that measures EDA and motion, worn on the wrist or above the ankle. Statistical analysis describing how equipment affects EDA was inconclusive, suggesting that many factors play a role in how a child’s EDA changes. Case studies provided examples of how occupational therapy affected children’s EDA. This is the first study of the effects of occupational therapy’s in situ activities using continuous physiologic measures. The results suggest that careful case study analyses of the relation between therapeutic activities and physiological arousal may inform clinical practice.

MIT Mood Meter is designed to assess and display the overall mood of the MIT community, by placing cameras at four different prime spots on the MIT campus (Student Center, Infinite Corridor, Stata Center, and Media Lab). The cameras are equipped with affect-sensing software that counts number of people and whether they are smiling or not. Although smiles are not the only sign of a good mood, in our project, we have used it as a barometer of happiness. This project is intended to raise awareness of how our own smiles can positively affect the surrounding environment, and to assess how congenial MIT is as a community. The dynamic, real-time information may lead to answers to questions such as: Are people from one department happier than others?, Do midterms lower the mood?, or Does warmer weather lead to happiness?”

This project will define and explore a new research area we call Computational Behavior Science–integrated technologies for multimodal computational sensing and modeling to capture, measure, analyze, and understand human behaviors. Our motivating goal is to revolutionize diagnosis and treatment of behavioral and developmental disorders. Our thesis is that emerging sensing and interpretation capabilities in vision, audition, and wearable computing technologies, when further developed and properly integrated, will transform this vision into reality. More specifically, we hope to: (1) enable widespread autism screening by allowing non-experts to easily collect high-quality behavioral data and perform initial assessment of risk status; (2) improve behavioral therapy through increased availability and improved quality, by making it easier to track the progress of an intervention and follow guidelines for maximizing learning progress; and (3) enable longitudinal analysis of a child's development based on quantitative behavioral data, using new tools for visualization.

We are looking at the emotional experience created when children learn games. Why do we start games with the most boring part, reading directions? How can we create a product that does not create an abundance of work for parents? Key insights generated from field work, interviews, and measurement of electrodermal activity are: kids become bored listening to directions, "it's like going to school"; parents feel rushed reading directions as they sense their children's boredom; children and parents struggle for power in interpreting and enforcing rules; children learn games by mimicking their parents, and; children enjoy the challenge of learning new games.