Intentional Inference of Emotions

How can you tell if your friend is happy, sad, frustrated, or engaged in a conversation? Their facial expressions, body language, and the tone of their voice are all nonverbal behaviors that are clues. But reading people’s emotions isn’t a textbook exercise, and people don’t normally wear their emotions on their sleeve. So to gain a better sense, we also ask our friends subtle but probing questions like “how is your day going so far?” or change our behavior, like being goofy, to see if we get a reaction. It takes a few tries to reach their true underlying feelings. Understanding emotions is a process!

As shown in the video above, today’s emotion detection and recognition technologies try to classify a person’s emotions based on just surface observations. But social agents, like Apple’s Siri, Amazon’s Alexa, or personal robots, operate on a social medium of dialog and have the opportunity to engage users in this emotion understanding process.

In our work, we computationally model emotion understanding as a goal-oriented, or intentional, inference process directed by a social agent. The social agent has a goal of wanting to form a correct inference about their partner’s emotions and tries out different behaviors to close gaps in their certainty.

For storytelling applications, we modeled how storytellers are able to guess whether a listener is attentive or inattentive to their story. We modeled their strategy in how they go about making this decision as a partially observable Markov decision process (a POMDP model). In representing emotion recognition as an active process from the perspective of storytellers, the POMDP model is more accurate in knowing a listener’s true attentive state.

Social agents should not be passive observers when trying to understand the emotions of their human partners. They can engage with them interactively, and through this back-n-forth gain a more accurate understanding about how a person is feeling.