Rudovic, O., Utsumi, Y., Lee, J., Hernandez, J., Castello Ferrer, E., Schuller, B., Picard, R. "CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism." IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2018).

Abstract

Many children on autism spectrum have atypical behavioral expressions of engagement compared to their neurotypical peers. In this paper, we investigate the performance of deep learning models in the task of automated engagement estimation from face images of children with autism. Specifically, we use the video data of 30 children with different cultural backgrounds (Asia vs. Europe) recorded during a single session of a robot-assisted autism therapy. We perform a thorough evaluation of the proposed deep architectures for the target task, including within- and across-culture evaluations, as well as when using the child-independent and child-dependent settings. We also introduce a novel deep learning model, named CultureNet, which efficiently leverages the multi-cultural data when performing the adaptation of the proposed deep architecture to the target culture and child. We show that due to the highly heterogeneous nature of the image data of children with autism, the child-independent models lead to overall poor estimation of target engagement levels. On the other hand, when a small amount of data of target children is used to enhance the model learning, the estimation performance on the held-out data from those children increases significantly. This is the first time that the effects of individual and cultural differences in children with autism have empirically been studied in the context of deep learning performed directly from face images.