Project

Commalla: Communication for All

Copyright

Authors retain copyright

Kristina T. Johnson

(Note: this project was previously referred to as ECHOS)

Over 1 million people in the U.S. are non- or minimally verbal (nv/mv), including but not limited to people with autism spectrum disorders (ASD), Down syndrome (DS), and other genetic disorders.  Nv/mv individuals often communicate emotions and desires through vocalizations that do not have typical verbal content, as well as through gestures, e.g., pulling a caregiver to a desired toy. Some vocalizations have self-consistent phonetic content (e.g., “ba” to mean “bathroom”) and others vary in tone, pitch, and duration depending on the individual’s emotional or physical state or intended communication.  

We present, to our knowledge, the first project studying communicative intent and affect in naturalistic vocalizations that do not have typical verbal content for non verbal and minimally verbal individuals. Interviewed parents of nv/mv children cited miscommunication with people who do not know their child well as a major source of stress. Our long-term vision is to design a device that can help others better understand and communicate with nonverbal … View full description

(Note: this project was previously referred to as ECHOS)

Over 1 million people in the U.S. are non- or minimally verbal (nv/mv), including but not limited to people with autism spectrum disorders (ASD), Down syndrome (DS), and other genetic disorders.  Nv/mv individuals often communicate emotions and desires through vocalizations that do not have typical verbal content, as well as through gestures, e.g., pulling a caregiver to a desired toy. Some vocalizations have self-consistent phonetic content (e.g., “ba” to mean “bathroom”) and others vary in tone, pitch, and duration depending on the individual’s emotional or physical state or intended communication.  

We present, to our knowledge, the first project studying communicative intent and affect in naturalistic vocalizations that do not have typical verbal content for non verbal and minimally verbal individuals. Interviewed parents of nv/mv children cited miscommunication with people who do not know their child well as a major source of stress. Our long-term vision is to design a device that can help others better understand and communicate with nonverbal and minimally verbal individuals by training machine learning models using primary caregivers’ unique knowledge of the meaning of an individual’s nonverbal communication. Our focus is currently on developing personalized models to classify vocalizations using in the moment live labels from caregivers via the Commalla labeling app. As part of this work, we are developing scalable methods for collecting and live labeling naturalistic data, and processing methods for using the data in machine learning algorithms. We are currently piloting and refining our data collection, machine learning models, and vision with a small number of families through a highly participatory design process.