End-to-End Listening Agent for Audiovisual Emotional and Naturalistic Interactions

End-to-End Listening Agent for Audiovisual Emotional and Naturalistic Interactions ■

Kevin EI Haddad, Kevin EI Haddad, Kevin EI Haddad, Yara Rizk, Yara Rizk, Yara Rizk, Louise Heron, Louise Heron, Louise Heron, Nadine Hajj, Nadine Hajj, Nadine Hajj, Yong Zhao, Yong Zhao, Yong Zhao, Jaebok Kim, Jaebok Kim, Jaebok Kim, Trung Ngo Trong, Trung Ngo Trong, Trung Ngo Trong, Minha Lee, Minha Lee, Minha Lee, Marwan Doumit, Marwan Doumit, Marwan Doumit, Payton Lin, Payton Lin, Payton Lin, Yelin Kim, Yelin Kim, Yelin Kim, Hüseyin Çakmak, Hüseyin Çakmak, Hüseyin Çakmak

Abstract ■

In this work, we established the foundations of a framework with the goal to build an end-to-end naturalistic expressive listening agent. The project was split into modules for recognition of the user{\textquoteright}s paralinguistic and nonverbal expressions, prediction of the agent{\textquoteright}s reactions, synthesis of the agent{\textquoteright}s expressions and data recordings of nonverbal conversation expressions. First, a multimodal multitask deep learning-based emotion classification system was built along with a rule-based visual expression detection system. Then several sequence prediction systems for nonverbal expressions were implemented and compared. Also, an audiovisual concatenation-based synthesis system was implemented. Finally, a naturalistic, dyadic emotional conversation database was collected. We report here the work made for each of these modules and our planned future improvements.