Dynamic replacement of the face of an actor in a video movie

• US 7,697,787 B2
• Filed: 06/06/2003
• Issued: 04/13/2010
• Est. Priority Date: 06/06/2002
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services GmbH (Accenture PLC)
• Inventors: Illsley, Martin
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): ENTEZARI, MICHELLE M
• Application Number:
  US10/456,765
• Publication Number:
  US 20030228135A1
• Time in Patent Office: 2,503 Days
• Field of Search: 382/295
• US Class Current: 382/295
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

The present invention concerns a method for replacing the face of an actor in a video clip, with the one of a user (U) of an entertainment video system (4), consisting in: a preparation phase, taking one first fixed picture of the face of the user; building a 3D-model of that face; replacing a first video picture of the actor with a reproduction of the face of the user; and while the clip is going on, replacing the face of the actor in the successive pictures of the video clip with successive pictures reproducing the face of the user, the transitions of the face of the actor being followed by applying at least orientation, size and displacement vectors to the 3D-model of face of the user on the basis of orientation, size and displacement vectors calculated for the face of the actor in the clip.

Claims

1. A method for dynamically replacing in real time the face of an actor in a video clip, with the one of a user of an entertainment video system, comprising the steps of:

in a preparation phase;

taking at least one first fixed picture of the face of the user;

building a 3D-model of that face;

replacing a first video picture of the face of the actor with a reproduction of the face of the user after matching their respective size and orientation; and

while the clip is going on;

taking periodically a current picture of the face of the user;

calculating current transformation vectors of the face of the user from a preceding picture of the face of the user to the current picture of the face of the user, using the 3D-model to determine said transformations; and

dynamically replacing the face of the actor in successive pictures of the video clip every 20 milliseconds with successive pictures reproducing the face of the user by applying at least orientation, size and displacement vectors to the 3D-model of the face of the user on the basis of orientation, size and displacement vectors calculated for the face of the actor in the clip and applying to a preceding picture of the user inserted in the clip the current transformation vectors of the face of the user, such that expressions in the face of the user and transitions of the face of the actor in the successive pictures of the video clip are followed.

2. Entertainment system for playing the part of an actor in a movie clip comprising:

a digital video reader (8);

a video screen (2);

a video camera (6) to take fixed pictures of an user;

a processing device (4) for replacing the face of a dedicated actor of the movie clip with the face of the user according to the method of claim 1.

3. The method of claim 1, further comprising:

calculating, based on the preceding picture of the face of the user and the current picture of the face of the user, expression transformation vectors for the 3D-model to reproduce a current expression of the user in the video clip.

4. The method of claim 3, where the expression transformation vectors comprise a shape-of-mouth transition vector.

5. The method of claim 3, where the expression transformation vectors comprise a position-of-eyes transition vector.

6. The method of claim 3, where replacing the face of the actor in successive pictures of the video clip with successive pictures reproducing the face of the user further comprises:

applying the expression transformation vectors to the 3D-model to generate for the successive video pictures the current user expressions in the current picture of the face of the user.

7. The method of claim 1 wherein the taking at least one first fixed picture of the user further comprises taking a picture of the user taking a pose corresponding to the first video picture of the actor.

8. The method of claim 1 wherein the preparation phase further comprises taking at least two additional pictures of the user taken from different angles.

9. The method of claim 1 further comprising:

replacing a voice of the actor with a voice of the user.

10. The method of claim 9 further comprising:

inserting written indication of the dialog such that the user'"'"'s voice is replaced dynamically.

11. The system of claim 2 further comprising:

a microphone for replacing dialog of the actor in the movie clip with a voice of the user.

12. The system of claim 11 wherein the microphone is linked to the processing device (4).