Since its introduction, model-based coding has received much attention, because it forms a basis for the next generation of video communication services: MPEG4 Synthetic Natural Hybrid Coding. In many cases, such as videophony, for which a 3D model of the scene is given, a `Knowledge Based Analysis/Synthesis Codec' is considered. In such a system, the image content is usually described by parametric models. At the encoder, the images are analyzed using computer vision techniques in order to estimate these parameters. At the decoder, the transmitted parameters are used to synthesize the images with computer graphics techniques. A widespread approach, for videophone applications, is to use a 3D wireframe model that approximates the facial anatomy (geometry) and physiology (expressions).
The control mechanism (model) used to simulate facial expressions, allows to solve the inverse problem: estimating the active facial parameters from the 2D images. To date different approaches were used: model fitting, muscle action modeling and physically based modeling. In addition to describing the (non rigid) facial expressions, the rigid 3D motion of the face should be also estimated.
In this research project, a `Facial Analysis Synthesis Scheme' (FASS) will be build up. The problem is divided into several sub-problems:
路 Head detection/tracking and its rigid 3D motion estimation based on color/shape and apparent motion
路 Fine extraction/tracking of the facial features shape and estimation of the non rigid 3D motion parameters (imposed by the control mechanism) for expressions synthesis, based on geodesic region contours
路 Adaptation of the generic 3D object model (structure calibration)
路 Definition of the control mechanism to simulate facial expressions using physically based modeling, for the inverse kinematic parameter estimation
路 Development of an MPEG4 demonstrator for videophony sequences.
This project links computer generated with natural videosequences by stearing a 3D animation of a facial model using analyzed 2D videophone data.
Runtime: 2000 - 2004