End-to-End Listening Agent for Audiovisual Emotional and Naturalistic Interactions
 
End-to-End Listening Agent for Audiovisual Emotional and Naturalistic Interactions ■
 
Kevin EI Haddad, Yara Rizk, Louise Heron, Nadine Hajj, Yong Zhao, Jaebok Kim, Trung Ngo Trong, Minha Lee, Marwan Doumit, Payton Lin, Yelin Kim, Hüseyin Çakmak
 
Abstract ■

In this work, we established the foundations of a framework with the goal to build an end-to-end naturalistic expressive listening agent. The project was split into modules for recognition of the user{\textquoteright}s paralinguistic and nonverbal expressions, prediction of the agent{\textquoteright}s reactions, synthesis of the agent{\textquoteright}s expressions and data recordings of nonverbal conversation expressions. First, a multimodal multitask deep learning-based emotion classification system was built along with a rule-based visual expression detection system. Then several sequence prediction systems for nonverbal expressions were implemented and compared. Also, an audiovisual concatenation-based synthesis system was implemented. Finally, a naturalistic, dyadic emotional conversation database was collected. We report here the work made for each of these modules and our planned future improvements.