Method and system for the automatic computerized audio visual dubbing of movies

• US 7,109,993 B2
• Filed: 10/24/2002
• Issued: 09/19/2006
• Est. Priority Date: 10/08/1995
• Status: Expired due to Fees

• Current Assignee: Yissum Research Development Company of the Hebrew University of Jerusalem Ltd. (Hebrew University of Jerusalem)
• Original Assignee: Yissum Research Development Company of the Hebrew University of Jerusalem Ltd. (Hebrew University of Jerusalem)
• Inventors: Cohen, Ran, Peleg, Shmuel, Avnir, David
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US10/279,097
• Publication Number:
  US 20030085901A1
• Time in Patent Office: 1,426 Days
• Field of Search: 352/5, 352/12, 352/23, 352/25, 345/419, 345/630, 345/473, 345/474, 345/475, 345/949, 345/957, 375/88.13, 375/13, 375/88, 353/19
• US Class Current: 345/473
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G11B 27/028 with computer assistance
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: Y10S 345/949 Animation processing method

Abstract

A method for automated computerized audio visual dubbing of a movie comprises (i) generating a three-dimensional head model of an actor in a movie, the head model being representative of specific facial features of the actor, and (ii) generating a three-dimensional head model of a dubber making target sounds for the actor, the dubber head model being representative of specific facial features of the dubber, as the target sounds are made. The method also comprises modifying at least a portion of the specific facial features of the actor head model according to the dubber head model such that the actor appears to be producing target sounds made by the dubber.

Claims

1. A method for automated computerized audio visual dubbing of a movie, comprising:

(a) fitting a generic three-dimensional actor head model of an actor in a movie to a two dimensional head picture of the actor for at least one frame in said movie, wherein said actor head model is representative of specific facial features of said actor during said frame;

(b) fitting a generic three-dimensional dubber head model of a dubber making target sounds to a two dimensional head picture of the dubber for said at least one frame in said movie, wherein said dubber head model is representative of specific facial features of said dubber, as said target sounds are made, and

(c) modifying at least a portion of said specific facial features of said actor head model according to said dubber head model such that said actor appears to be producing said target sounds made by said dubber.

2. A method according to claim 1, further comprising modifying the face of the actor in said frame according to said actor head model as modified so as to obtain a frame wherein at least a portion of the specific facial features of the actor correspond to specific facial features of said dubber and replacing target sounds made by said actor with target sounds made by said dubber.

3. A method according to claim 1, wherein said specific facial features comprises the lips and mouth area.

4. A method according to claim 3, wherein said specific facial features further comprise secondary facial muscles including the cheeks and eyebrows.

5. A method according to claim 1, wherein said at least one frame comprises multiple frames of a movie.

6. A method according to claim 1, wherein said movie is selected from the group consisting of a photographed movie, an animated movie, or any combination thereof.

7. A method according to claim 1, wherein said fittings are accomplished by computer fitting a respective generic three-dimensional head model to a picture of the actor or the dubber using significant facial features of said actor or dubber.

8. A method according to claim 7, further comprising computer tracking specific facial features of said respective three-dimensional head models of said actor and said dubber through a plurality of frames in said movie so as to create a library of reference similarity frames.

9. A method according to claim 8, further comprising mapping, for a plurality of frames in the movie, the face of the actor to the three dimensional head model of said actor wherein said mapping employs a computerized texture mapping technique that uses said reference similarity frames.

10. A method according to claim 9, wherein said modifying comprises replacing, on a frame by frame basis, at least a portion of the specific facial features of the actor head model with those of the dubber head model.

11. A method according to claim 1, wherein the actor is a human, an animal, or any object made to appear to be speaking.

12. A method according to claim 1, wherein the dubber is a human, an animal, or any object made to appear to be speaking.

13. A method according to claim 1, wherein the target sounds comprise at least one spoken word, or at least one sound.

25. A computer-readable media containing instructions for controlling a computer system to implement the method of claim 1.

14. A method for automated computerized audio visual dubbing of a movie, comprising:

(a) providing a three-dimensional graphical model of a head;

(b) providing a video sequence of a speaker having at least one frame;

(c) tracking features of said speaker through said video sequence, and extracting changing head parameters of said speaker;

(d) applying the extracted head parameters to a three-dimensional head model of the speaker;

(e) applying said changing head parameters from the three-dimensional head model of the speaker to said three-dimensional graphical head model so as to create a speaking three-dimensional head model; and

(f) generating a video sequence, having at least one frame, of said speaking three-dimensional head model using computer graphic methods.

15. A method according to claim 14, wherein the head parameters comprise the lips and mouth area.

16. A method according to claim 15, wherein the head parameters further comprise secondary facial muscles including the cheeks and the eyebrows.

17. A method according to claim 14 wherein a sound track is added to said video sequence of said three-dimensional head model.

18. A method according to claim 14, wherein the speaker is a human, an animal, or any object made to appear to be speaking.

26. A computer-readable media containing instructions for controlling a computer system to implement the method of claim 14.

19. A system for the automated computerized audio visual dubbing of a movie, comprising;

(a) means for providing a three-dimensional graphical model of a head;

(b) means for tracking features of a speaker through a video sequence, and for extracting changing head parameters of said speaker through said video sequence;

(c) means for applying the extracted head parameters to a three-dimensional head model of the speaker;

(d) means for applying said changing head parameters from the three-dimensional head model of the speaker to said three-dimensional graphical head model so as to create a speaking three-dimensional head model and for generating a video sequence, having at least one frame, of said speaking three-dimensional head model using computer graphic methods; and

(e) means for adding a sound track to said video sequence of said three dimensional head model.

20. A system for the automated computerized audio visual dubbing of a movie, comprising;

means for fitting a generic three-dimensional actor head model of an actor in a movie to a two dimensional head picture of the actor for at least one frame in said movie, wherein said actor head model is representative of specific facial features of said actor during said frame;

means for fitting a generic three-dimensional dubber head model of a dubber making target sounds to a two dimensional head picture of the dubber for said at least one frame in said movie, wherein said dubber head model is representative specific facial features of said dubber, as said target sounds are made, and;

means for modifying at least a portion of said specific facial features of said actor head model according to said dubber head model such that said actor appears to be producing said target sounds made by said dubber.

21. Method for computerized audio visual dubbing of a movie, the method comprising using mouth parameters of an original actor for locating similar mouth parameters in a library of reference similarity frames and replacing the mouth area of the original actor in at least one frame of a movie by the mouth area of an actor taken from said library.

22. Method according to claim 21 wherein the library of reference similarity frames comprises a compiled picture library of the original actor.

23. Method according to claim 21 wherein the wherein the library of reference similarity frames comprises a compiled picture library of at least one picture of an actor other then the original actor.

27. A computer-readable media containing instructions for controlling a computer system to implement the method of claim 21.

24. A method for automated computerized audio visual dubbing of a movie, comprising;

(a) providing a three-dimensional head model of an actor in a movie for at least one frame in said movie, wherein said actor head model is representative of specific facial features of said actor during said frame;

(b) providing a three-dimensional head model of a dubber making target sounds for said at least one frame in said movie, wherein said dubber head model is representative of specific facial features of said dubber, as said target sounds are made; and

(c) modifying at least a portion of said specific facial features of said actor head model according to said dubber head model such that said actor appears to be producing said target sounds made by said dubber.

28. A computer-readable media containing instructions for controlling a computer system to implement the method of claim 24.