Text-based Editing of Talking-head Video (SIGGRAPH 2019)
SIGGRAPH 2019 Paper. Read below for important ethical considerations.
Project page: https://www.ohadf.com/projects/text-based-editing/
My webpage (with this and other related projects): https://www.ohadf.com/
Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation…