Film language

• US 6,778,252 B2
• Filed: 12/20/2001
• Issued: 08/17/2004
• Est. Priority Date: 12/22/2000
• Status: Active Grant

• Current Assignee: BabelOn LLC
• Original Assignee: Film Language
• Inventors: Bryant, Andrew, Wolff, Steven, Moulton, William Scott, Schumacher, Rod, Hamilton, Strath, Taschner, Dana, Hamilton, Marcy
• Primary Examiner(s): FULLER, RODNEY EVAN
• Application Number:
  US10/027,191
• Publication Number:
  US 20020097380A1
• Time in Patent Office: 971 Days
• Field of Search: 352/5, 352/12, 352/23, 352/25
• US Class Current: 352/12
• CPC Class Codes: G03B 31/00 Associated working of cameras or projectors with sound-recording or sound-reproducing means
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G11B 2220/90 Tape-like record carriers
• CPC Class Codes: G11B 27/022 Electronic editing of analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/024 on tapes G11B27/028, G11B27/029 take precedence
• CPC Class Codes: G11B 27/028 with computer assistance
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel

Abstract

A method for modifying an audio visual recording originally produced with an original audio track of an original speaker, using a second audio dub track of a second speaker, comprising analyzing the original audio track to convert it into a continuous time coded facial and acoustic symbol stream to identify corresponding visual facial motions of the original speaker to create continuous data sets of facial motion corresponding to speech utterance states and transformations, storing these continuous data sets in a database, analyzing the second audio dub track to convert it to a continuous tune coded facial and acoustic symbol stream, using the second audio dub track'"'"'s continuous time coded facial and acoustic symbol stream to animate the original speaker'"'"'s face, synchronized to the second audio dub track to create natural continuous facial speech expression by the original speaker of the second dub audio track.

Claims

1. A method for modifying an audio visual recording originally produced with an original audio track of an original speaker, using a second audio dub track of a second speaker, to produce a new transformed audio visual recording with synchronized audio to facial expressive speech of the second audio dub track spoken by the original speaker, comprising analyzing the original audio track to convert it into a continuous time coded facial and acoustic symbol stream to identify corresponding visual facial motions of the original speaker to create continuous data sets of facial motion corresponding to speech phoneme utterance states and transformations, storing these continuous data sets in a database, analyzing the second audio dub track to convert it to a continuous time coded facial and acoustic symbol stream, using the second audio dub track'"'"'s continuous time coded facial and acoustic symbol stream to animate the original speaker'"'"'s face, synchronized to the second audio dub track to create natural continuous facial speech expression by the original speaker of the second dub audio track.

2. The method of claim 1 wherein said second audio dub track is spoken in a language different from that of the original speaker.

3. The method of claim 1 wherein said continuous time coded facial and acoustic symbols comprise diphones and visemes.

4. The method of claim 1 wherein said continuous time coded facial and acoustic symbols comprise triphones and visemes.

5. The method of claim 1 comprising using a set affixed facial reference points to track the facial transformation from one continuous time coded facial and acoustic symbol to another continuous time coded facial and acoustic symbol.

6. The method of claim 5 comprising using a computer motion tracking system to record the optical flow path for each fixed reference point.

7. The method of claim 1 further comprising using a set of fixed facial reference points to track the facial transformation from one continuous time coded facial and acoustic symbol to another continuous time coded facial and acoustic symbol.

8. The method of claim 7 further comprising accumulating a database of recorded mouth transformation optical flow paths associated with continuous time coded facial and acoustic symbols.

9. The method of claim 5 further comprising adding an emotional elicitation process by using a computer motion tracking system to record a number of facial control points for each emotion.

10. The method of claim 9 in which the facial control points comprise, but are not limited to, the chin, teeth, inner lips, outer lips, cheeks and jowls.

11. The method of claim 7 comprising accumulating a database of visemes by recording the facial control points corresponding to each continuous time coded facial and acoustic symbol.

12. The method of claim 11 comprising accumulating a database of muzzle patches by mapping the facial control points for each viseme.

13. The method of claim 12 comprising selecting muzzle patches from the speaker'"'"'s viseme database based on the second dub audio track and continuous time coded facial and acoustic symbol to viseme sequence and applying the selected muzzle patches onto a three dimensional facial muzzle model.

14. The method of claim 13 further comprising also collecting a set of patches of light, color and texture from the sampled first speaker'"'"'s muzzle patches.

15. The method of claim 1 in which the fixed reference points for both speakers are mapped using radar.

16. The method of claim 15 in which the dub audio track radar measurements are referenced to a particular continuous time coded facial and acoustic symbol or symbol to symbol transitions.

28. The methods of claim 1 or 17 comprising using head modeling to aid in positioning three dimensional visemes.

29. The methods of claim 1 or 17 comprising modeling multiple floating point control vertices relative to a generic head shape and relative on a variety of speech viseme standards.

30. The methods of claim 1 or 17 comprising texture sampling target footage to texture match visemes.

17. A method for modifying an audio visual recording originally produced with an original audio track of an original screen actor, using a second audio dub track of a second screen actor, to produce a new transformed audio visual recording with synchronized audio to facial expressive speech of the second audio dub track spoken by the original screen actor, comprising analyzing the original audio track to convert it into a continuous time coded facial and acoustic symbol stream to identify corresponding visual facial motions of the original speaker to create continuous data sets of facial motion corresponding to speech utterance states and transformations, storing these continuous data sets in a database, analyzing the second audio dub track to convert it to phonemes as a time coded phoneme stream a continuous time coded facial and acoustic symbol stream, using the second audio dub track'"'"'s continuous time coded facial and acoustic symbol stream to animate the original screen actor'"'"'s face, synchronized to the second audio dub track to create natural continuous facial speech expression by the original screen actor of the second dub audio track.

18. The method of claim 17 comprising using a computer vision system to track and record a database of visemes of the actors'"'"' head position, facial motions of the jaw, and the lip motion during speech for each continuous time coded facial and acoustic symbol.

19. The method of claim 18 comprising tracking fixed reference control points on the head, jaw and lips.

20. The method of claim 19 in which the reference control points comprise the outside edge of the lips, the inside edge of the lips, the edge of the chin, the tip of the tongue, the bottom edge of the upper teeth, the upper edge of the lower teeth, the cheeks and jowls, the nose and the eyes.

21. The method of claim 20 comprising accumulating a database of muzzle patches by mapping the facial control points for each viseme.

22. The method of claim 21 comprising selecting muzzle patches from the speaker'"'"'s viseme database based on the second dub audio track and continuous time coded facial and acoustic symbol to viseme sequence and applying the selected muzzle patches onto a three dimensional facial muzzle model.

23. The method of claim 22 further comprising also collecting a set of patches of light, color and texture from the sampled first speaker'"'"'s muzzle patches and applying these patches based on the second dub audio track.

24. The method of claim 19 in which the fixed reference points for both actors are mapped using radar.

25. The method of claim 24 in which the dub audio track radar measurements are referenced to a particular phoneme or phoneme to phoneme transition in time continuous time coded facial and acoustic symbol or symbol to symbol transition.

26. The method of claim 24 in which the radar mapping information is transferred to an animation control mixer.

27. The method of claim 24 in which the viseme database information is transferred to an animation control mixer.

31. A method for modifying an audio visual recording originally produced with an original audio track of an original screen actor, using a second audio dub track of a second screen actor, to produce a transformed audio visual recording with synchronized audio to facial expressive speech of the second audio dub track spoken by the original screen actor, comprising analyzing the original audio track to convert it into a continuous time coded facial and acoustic symbol stream, identifying corresponding visemes of the original screen actor, using radar to measure a set of facial reference points corresponding to speech continuous time coded facial and acoustic symbol utterance states and transformations, storing the data obtained in a database, analyzing the second audio dub track to convert it to phonemes a continuous time coded facial and acoustic symbol stream, identifying corresponding visemes of the second screen actor, using radar to measure a set of facial reference points corresponding to speech facial and acoustic symbol utterance states and transformations, storing the data obtained in a database, using the second audio dub track time-coded continuous time coded facial and acoustic symbol stream and the actors visemes to animate the original screen actor'"'"'s face, synchronized to the second audio dub track to create natural continuous facial speech expression by the original screen actor of the second dub audio track.

32. The method of claim 31 in which the database resides in an animation control mixer.

33. The method of claim 31 in which the facial reference points comprise the outside edge of the lips, the inside edge of the lips, the edge of the chin, the tip of the tongue, the bottom edge of the upper teeth, the upper edge of the lower teeth, the cheeks and jowls, the nose and the eyes.

34. The method of claim 31 comprising texture sampling target footage to texture match visemes.