Method and system of presenting foreign films in a native language

• US 8,655,152 B2
• Filed: 01/11/2013
• Issued: 02/18/2014
• Est. Priority Date: 01/31/2012
• Status: Active Grant

• Current Assignee: Golden Monkey Entertainment
• Original Assignee: Golden Monkey Entertainment
• Inventors: Marks, Darryl, McIntyre, Raymond Jr.
• Primary Examiner(s): Dang, Hung
• Assistant Examiner(s): Park, Sunghyoun
• Application Number:
  US13/739,993
• Publication Number:
  US 20130195428A1
• Time in Patent Office: 403 Days
• Field of Search: None
• US Class Current: 386/278
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 5/93 Regeneration of the television signal or of selected parts thereof

Abstract

A process for presenting live action foreign language feature films in a native language by replacing the physical mouth positions of the original actors to match a newly recorded audio track in a different language with the original and/or replacement actors keeping the essence of the original dialect, while achieving the illusion that the content was originally filmed in the new voice over language. This process applies an artistic, multifaceted approach rather than a purely procedural approach, and utilizes a combination of visual effects techniques to achieve realistic lip sync animation to be convincing to the viewing audience.

Claims

1. A method for converting a foreign film into a native language, comprising:

a. for every segment of an original recording having a speaking line and associated facial expressions recorded at a specific perspective view of an original actor in the foreign film;

i. audio recording a voice over actor speaking the speaking line in the native language to create a native language audio recording;

ii. video recording the voice over actor during the audio recording from a face-on camera angle to create a face-on video recording;

iii. video recording the voice over actor during the audio recording from a matched camera angle to create a matched-angle video recording;

iv. photographing the voice over actor from the face-on camera angle to create a face-on photograph;

v. photographing the voice over actor from the matched camera angle to create a matched-angle photograph;

vi. adjusting a delivery of the speaking line of the voice over actor, wherein adjusting the delivery of the speaking line comprises;

1. adjusting facial expressions of the voice over actor reciting the speaking line in the native language to best-fit the facial expressions of the original actor reciting the speaking line (“

original speaking actor”

),

2. adjusting a duration of speech of the voice over actor to recite the speaking line in the native language to match a duration of speech of the original speaking actor,

vii. using an imagery selected from the group consisting of the face-on video recording, the matched-angle video recording, the face-on photograph, and the matched-angle photograph as a visual guide to permit a digital artist to recreate native language visemes;

b. overlaying and comparing one of the segments of the original recording containing the speaking line with a corresponding segment of the matched-angle video recording of the voice over actor to create an exposure sheet to determine a viseme position for the native language audio recording;

c. checking the exposure sheet with a waveform of the native language audio recording to map out the native language visemes of the native language audio recording, wherein the exposure sheet comprises a frame number and a native language phoneme;

d. creating a still frame of the original speaking actor at rest that is a neutral face position of the original actor while the original actor is not speaking or emoting;

e. choosing a best methodology to change original visemes of the original speaking actor, the best methodology selected from the group consisting of a 2D technique and a 3D technique, wherein

i. the 2D technique uses existing pixels within a photographed plate of the original recording, wherein existing pixels of lip and facial image elements are taken from other frames within the original recording in which the original speaking actor'"'"'s lip and facial image elements matches the voice over actor'"'"'s lip and facial image elements at a different frame, and wherein

ii. the 3D technique comprises

1. creating a digital geometry to match the original actor'"'"'s facial structure at a particular frame,

2. projecting a skin texture of the original actor back onto the digital geometry, and

3. integrating the digital geometry back into the original recording;

f. executing work to change a mouth position of the original actor to match the voice over actor reciting the speaking line to create a modified original speaking actor;

g. performing quality control for an overall look, feel, and accuracy of a conversion to the native language, wherein special attention is given to a blending of any newly created elements with the original actor'"'"'s face, the newly created elements selected from the group consisting of colors, shadows, light, motion blur, focus, grain, and noise;

h. comparing a resulting facial imagery of the modified original speaking actor and the voice over actor by compositing a picture-in-picture of the recorded video of the voice over actor to determine accuracy of the conversion; and

i. audio mixing to replace a voice track of the original actor with a voice track of the voice over actor, and integrate the voice over actor'"'"'s voice into an audio track for the original recording.

2. The method of claim 1, wherein the 2D technique is selected from the group consisting of warping, painting, pixel tracking, planar tracking, rotoscoping, replacing with similar frames from the original actor, and compositing.

3. The method of claim 2, wherein using the 2D technique comprises the steps of:

a. tracking the original actor of the speaking line iteratively, frame by frame, using a computer by picking a first facial feature of the original actor, and determining and following a position of the first facial feature of the original actor on a three-dimensional coordinate relative to an original camera, wherein at least two facial features are tracked and used to calculate rotation and skew;

b. tracking the voice over actor of the speaking line iteratively, frame by frame, using the computer by picking a first facial feature of the voice over actor, and determining and following a position of the first facial feature of the voice over actor on a three-dimensional coordinate relative to a voice over camera;

c. assigning the voice over actor to the original actor, wherein the voice over actor becomes a desired lip and facial expression for the original actor; and

d. using visemes from the voice over actor as a guide to modify a mouth movement of the original actor.

4. The method of claim 2, wherein the original actor'"'"'s sequence of mouth movements from a first segment is replaced with the original actor'"'"'s sequence of mouth movements from a second segment, wherein the original actor'"'"'s sequence of mouth movements from the second segment matches the voice over actor'"'"'s sequence of mouth movements speaking the speaking line in the native language in the first segment.

5. The method of claim 4, wherein the original actor'"'"'s mouth movements from the second segment is modified from its original sequence, wherein the modification is selected from the group consisting of running the sequence faster, running the sequence slower, and running the sequence in reverse.

6. The method of claim 2, wherein the original actor'"'"'s mouth movements is replaced with the still frame.

7. The method of claim 1, wherein when the original actor'"'"'s movements involve significant perspective changes, at least one 3D technique selected from the group consisting of modeling, texturing and lighting of the original actor'"'"'s facial features is used.

8. The method of claim 7, comprising the steps of:

a. tracking the original actor of the speaking line iteratively, frame by frame, using a computer by picking a first facial feature of the original actor, and determining and following a position of the first facial feature on a three-dimensional coordinate relative to an original camera, wherein at least two facial features are tracked and used to calculate rotation and skew;

b. tracking the voice over actor of the speaking line iteratively, frame by frame, using the computer by picking a first facial feature of the voice over actor, and determining and following a position of the first facial feature of the voice over actor on a three-dimensional coordinate relative to a voice over camera;

c. assigning the voice over actor to the original actor, wherein the voice over actor becomes a desired lip and facial expression for the original actor;

d. calculating a position, a movement, and a lens information of the original camera to recreate a new camera motion for the speaking line; and

e. performing a 3D match-move of the original actor'"'"'s face to determine the original actor'"'"'s position in the scene relative to the camera.

9. The method of claim 8, further comprising the step of creating a digital model of the shape and motion of a head of the original actor.

10. The method of claim 9, wherein the digital model is created using more than one image of the original actor from the original recording and creating a digital wire frame of a face of the original actor.

11. The method of claim 10, wherein the digital wire frame is animated to match the mouth movements of the voice over actor reciting the speaking line in the native language.

12. The method of claim 11, further comprising:

a. rotoscoping to extract the hair, skin, color and lighting from the original actor as captured in the original recording and projected onto the digital wire frame; and

b. rendering to match the new skin, lighting, and shadows to the original actor.

13. The method of claim 12, further comprising:

a. compositing the digital model with an original frame; and

b. utilizing a 2D technique to blend into the original actor'"'"'s face.

14. The method of claim 1, further comprising compositing a picture-in-picture of each frame of the segment of the recorded video of the voice over actor with each frame of the corresponding segment of the original actor to determine accuracy of the lip synchronization.

15. A method for converting a foreign film into a native language, comprising:

a. for every segment of an original recording having a speaking line and associated facial expressions recorded at a specific perspective for every original actor in the foreign film;

i. audio recording a voice over actor speaking the speaking line in a native language to create a native language audio recording;

ii. video recording the voice over actor during the audio recording from a face-on camera angle to create a face-on video recording;

iii. video recording the voice over actor during the audio recording from a matched camera angle to create a matched-angle video recording;

iv. using an imagery selected from the group consisting of the face-on video recording and the matched-angle video recording as a visual guide to permit a digital artist to recreate native language visemes;

b. overlaying and comparing a segment of the original recording containing the speaking line with a corresponding segment of the matched-angle video recording of the voice over actor to create an exposure sheet to determine a viseme position for the native language audio recording;

c. choosing a best methodology to change an actively speaking original actor'"'"'s visemes, the best methodology selected from the group consisting of a 2D technique and a 3D technique;

d. executing a conversion to change a mouth position of the actively speaking original actor to match the voice over actor reciting the speaking line to create a modified original actor;

e. performing quality control for an overall look, feel, and accuracy of the conversion to the native language;

f. comparing a resulting facial imagery of the modified original actor with the voice over actor to determine accuracy of the conversion; and

g. audio mixing to replace a voice track of the original actor with a voice track of the voice over actor, and integrate the voice over actor'"'"'s voice track into an audio track for the original recording.

16. A method for converting a foreign film into a native language, comprising:

a. transcribing an original dialogue of an original actor having a speaking line in an original segment into a native language to create a transcribed speaking line;

b. recording a voice-over actor performing the transcribed speaking line at a current camera angle matching an original camera angle of the speaking line shown in the original segment, wherein the voice-over actor matches a timed-length of speaking of the original actor to create a transcribed segment;

c. creating a first set of exposure sheets comprising placement of original phonemes and corresponding visemes per exposed frame of the original segment to create a time code for frames as it relates to the original phonemes;

d. creating a second exposure sheet based on a waveform of the voice over actor'"'"'s recording to map out new phonemes and corresponding visemes of the transcribed speaking line;

e. creating animated visemes to replace visemes of the original actor, wherein the technique used to create the animated visemes is selected from the group consisting of 2D methodology and 3D methodology;

f. blending a new element with existing faces to match discrepancies; and

g. replacing an original audio track with the speaking line with a new audio track with the transcribed speaking line;

h. comparing the original segment with the transcribed segment by using a technique selected from the group consisting of picture-in-picture, split screen, or ghost image overlay; and

i. using the comparison to check phoneme positions of the speaking line and determining a best lip-sync for the transcribed speaking line.

17. The method of claim 16, further comprising creating still frames of neutral facial expression of the original actor from the foreign film, in which the original actor is not speaking and is not making any facial expressions for every shot in which the original actor appears.

18. The method of claim 17, further comprising creating additional still frames if a perspective view of the original actor changes within the original segment.

19. The method of claim 16, wherein creating animated visemes comprises original visemes of the original actor shifted from an original frame to a different frame within the foreign film.