Replacement of faces in existing video

• US 7,768,528 B1
• Filed: 11/03/2006
• Issued: 08/03/2010
• Est. Priority Date: 11/09/2005
• Status: Active Grant

• Current Assignee: Image Metrics Ltd. (Image Metrics, Inc.)
• Original Assignee: Image Metrics Ltd. (Image Metrics, Inc.)
• Inventors: Caulkin, Steven, Walker, Kevin, Edwards, Gareth
• Primary Examiner(s): Amini; Javid A
• Application Number:
  US11/592,720
• Time in Patent Office: 1,369 Days
• Field of Search: 345/473, 345/619
• US Class Current: 345/619
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G11B 27/036 Insert-editing

Abstract

Methods and computer program code computer program codes for modifying an expression of an actor in at least one video frame based on performance of the actor in a sequence of frames of video footage. A plurality of control points are identified for characterizing the shape of the actor in each frame of the video footage by a shape vector comprising positions of the control points. The shape vectors of each frame are then warped to a common standard frame thereby generating an aligned shape vector and a transformed image for each frame. The aligned shape vectors and attendant texture vectors are transformed in accordance with specified pose and expression parameters to create new shape and texture vectors which are reinserted into the sequence of frames so as to reanimate the actor.

Claims

1. A modification method for modifying an expression of an individual in at least one video frame, said modification method being based on a model of the individual derived from a plurality of video frames of video footage, while a pose of the individual'"'"'s performance that is associated with each video frame is retained, the individual being characterized by a shape, the method comprising:

a. identifying a plurality of control points with respect to the individual for characterizing the shape of the individual;

b. characterizing at least one feature of the individual in each frame of the video footage by a shape vector, the shape vector comprising x and y image coordinate positions of the plurality of control points;

c. warping each shape vector of each frame to a common standard frame thereby generating an aligned shape vector and a transformed image for each frame;

d. representing values of pixels of the transformed image as a texture vector;

e. specifying a subset of the plurality of control points as belonging to a group of pose control points;

f. creating a first model to predict a control expression shape vector from the group of pose control points;

g. creating a further model to predict a texture vector from the plurality of control points;

h. extracting a set of exemplar frames from among the plurality of video frames of video footage to serve as a basis for expression manipulation;

i. representing each frame of the basis for expression manipulation as a difference between observed shape and texture vectors and the control expression shape and texture vectors predicted in steps (f) and (g) from the plurality of control points;

j. transforming each shape vector for each frame in accordance with specified pose control points and weights applied to the said basis for expression manipulation in such a manner as to create new shape and new texture vectors; and

k. reinserting the new shape and new texture vectors into the sequence of frames so as to create a graphical depiction of the individual bearing a modified expression.

2. A method in accordance with claim 1, wherein pose and expression parameters are cast as difference vectors with respect to the plurality of control points.

3. A memory device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for modifying an expression of an individual displayed in an original image in a plurality of video frames from a sequence of frames of a video footage based on original images of the individual displayed in the sequence of frames, the method comprising:

a. identifying a plurality of control points with respect to the individual for characterizing the shape of the individual;

b. characterizing at least one feature of the actor in each frame of the video footage by a shape vector, the shape vector comprising x and y image coordinate positions of the plurality of control points;

c. warping each shape vector of each frame to a common standard frame thereby generating an aligned shape vector and a transformed image for each frame;

d. representing values of pixels of the transformed image as a texture vector;

e. specifying a subset of the plurality of control points as belonging to a group of pose control points;

f. creating a first model to predict a control expression shape vector from the group of pose control points;

g. creating a further model to predict a texture vector from the plurality of control points;

h. extracting a set of exemplar frames from among the plurality of video frames of video footage to serve as a basis for expression manipulation;

i. representing each frame of the basis for expression manipulation as a difference between observed shape and texture vectors and the control expression shape and texture vectors predicted in steps (f) and (g) from the plurality of control points;

j. transforming each shape vector for each frame in accordance with specified pose control points and weights applied to the said basis for expression manipulation in such a manner as to create new shape and new texture vectors; and

k. reinserting the new shape and new texture vectors into the sequence of frames so as to create a graphical depiction of the individual bearing a modified expression.

4. A computer program product for use on a computer system for modifying an expression of an individual displayed in an original image in a plurality of video frames from a sequence of frames of a video footage based on original images of the individual displayed in the sequence of frames, the computer program product comprising a computer usable tangible storage medium having computer readable program code thereon, the computer readable program code comprising:

a. program code for identifying a plurality of control points with respect to the individual for characterizing the shape of the individual;

b. program code for characterizing at least one feature of the individual in each frame of the video footage by a shape vector, the shape vector comprising x and y image coordinate positions of the plurality of control points;

c. program code for warping each shape vector of each frame to a common standard frame thereby generating an aligned shape vector and a transformed image for each frame;

d. program code for representing values of pixels of the transformed image as a texture vector;

e. program code for specifying a subset of the plurality of control points as belonging to a group of pose control points;

f. program code for creating a first model to predict a control expression shape vector from the group of “

pose”

control points;

g. program code for creating a further model to predict a texture vector from the plurality of control points;

h. program code for extracting a set of exemplar frames from among the plurality of video frames of video footage to serve as a basis for expression manipulation;

i. program code for representing each frame of the basis for expression manipulation as a difference between observed shape and texture vectors and the control expression shape and texture vectors predicted in steps (f) and (g) from the plurality of control points;

j. program code for transforming each shape vector for each frame in accordance with specified pose control points and weights applied to the said basis for expression manipulation in such a manner as to create new shape and new texture vectors; and

k. program code for reinserting the new shape and new texture vectors into the sequence of frames so as to create a graphical depiction of the individual bearing a modified expression.