A memory-friendly multi-modal emotion analysis for smart toy

Abstract

The recent advance in deep learning technologies has provided many opportunities to the industries in developing smarter products, such as smart toys, smart cars and smart homes. Unfortunately, a common practical issue to these deep learning methods is the high requirements in computing power even for the prediction part. As the deep learning models' complexities increase, the memory requirements for these models usually exceed the limit of many low end computing devices, such as mobile IoT devices. It is even worse when a multimodel approach is employed. In this paper, we will demonstrate with the development of multimodal emotion analysis on a smart toy, where the user's speech, facial expression and action are used for the understanding of user's emotion. By trimming down DNN complexity or replaced by other learning approaches, we are able to sequeeze four classifiers in 800MB of memory. Finally, results of these methods are ensembled with a fusion approach using a fully connected neural network to obtain a more accuracy and stable result. Our multimodal approach achieved an improvement of about 20% when comparing any unimodal emotion analysis.