Robust temporal alignment of spontaneous and dubbed speech and its application for Automatic Dialogue Replacement
 
Robust temporal alignment of spontaneous and dubbed speech and its application for Automatic Dialogue Replacement 
 
Pieter Soens, Werner Verhelst
 
Abstract 

In this paper, we present a robust system for the temporal alignment of 2 renditions of the same speech utterance. The system operates in 2 steps: during analysis, the timing relationships between the speech segments of the utterance that serves as a timing reference and the corresponding speech segments in the replacement utterance are measured by means of a dedicated dynamic time warping algorithm. The obtained warping paths are then processed and used to synthesize a high-quality speech utterance that is time-aligned with the reference. Subjective audio-visual listening tests performed within the context of a difficult Automatic Dialogue Replacement task demonstrated that the proposed system achieves a significant improvement compared to the industry-standard benchmark, both in terms of achieved lip-synchronization accuracy as well as in overall sound quality of the synthesized utterances.