Publication

Zero-Shot Transfer Learning to Enhance Communication for Minimally Verbal Individuals with Autism using Naturalistic Data

Narain, J.* & Johnson, K.T.*, Picard, R.W., Maes, P. "Zero-Shot Transfer Learning to Enhance Communication for Minimally Verbal Individuals with Autism using Naturalistic Data," NeurIPS Workshop on AI for Social Good, December 2019. (*equal contribution)

Abstract

We applied zero-shot transfer learning to classify vocalizations from a nonverbal individual with autism using captured audio. Data were recorded in natural environments using a small wearable camera and sparsely labeled in real-time with a custom-built open-source app. We then trained LSTM models on VGGish audio embeddings from the generic AudioSet database for three categories of vocalizations: laughter, negative affect, and self-soothing sounds. We applied these models to the unique audio recordings of a young autistic boy with no spoken words. The models identified laughter and negative affect with 70% and 69% accuracy, respectively, but classification of the self-soothing sounds produced accuracies around chance. This work highlights both the need and potential for specialized, naturalistic databases and novel computational methods to enhance translational communication technologies in underserved populations.

Related Content