Apparatus and method for lip-synching animation

• US 4,913,539 A
• Filed: 04/04/1988
• Issued: 04/03/1990
• Est. Priority Date: 04/04/1988
• Status: Expired due to Fees

• Current Assignee: New York Institute of Technology Incorporated
• Original Assignee: New York Institute of Technology Incorporated
• Inventors: Lewis, John P.
• Primary Examiner(s): Hayes, Monroe H.
• Application Number:
  US07/176,843
• Time in Patent Office: 729 Days
• Field of Search: 352/87, 352/50, 352/51, 352/52, 352/54, 352/5
• US Class Current: 352/87
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G09B 19/04 Speaking with audible presentation of the material to be studied G09B5/04
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

The disclosed invention takes advantage of the fact that the speech information and processing needed for successful animation lip-synching is actually much less than the information and processing which is needed for identifying individual words, as in the traditional speech recognition task, The disclosed embodiment utilizes linear prediction to obtain speech parameters which can be used to identify phonemes from a limited set corresponding to visually distinctive mouth positions. In the disclosed embodiment there is set forth an apparatus for producing lip-synching for a spoken sound track. Means are provided for storing, for each of a number of mouth positions, an unvoiced phonetic information representation associated with a mouth position. Means, responsive to samples of sound from the sound track, are provided for generating unvoiced phonetic information signals for the samples. Further means are provided for comparing the unvoiced phonetic information signals with the stored unvoiced phonetic information representations to determine which of the stored representations most closely matches the unvoiced phonetic information signals. The determined representations are then stored in time correspondence with the sound track.

Claims

1. Apparatus for producing lip-synching for a spoken sound track, comprising:

means for storing, for each of a number of mouth positions, an unvoiced phonetic information representation associated with a mouth position;

means responsive to samples of sound from the sound track for generating unvoiced phonetic information signals for the samples;

means for comparing said unvoiced phonetic information signals with the stored unvoiced phonetic information representations to determine which of the stored representations most closely matches the unvoiced phonetic information signals; and

means for storing the determined representations in time correspondence with the sound track.

2. Apparatus as defined by claim 1, wherein said means responsive to samples of sound is operative to generate said unvoiced phonetic information signals as linear predictive coding signals representative of a vocal tract filter.

3. Apparatus as defined by claim 2, wherein said stored unvoiced phonetic information representations are also in the form of linear predictive coding signals.

4. Apparatus as defined by claim 1, wherein said means for storing unvoiced phonetic information representations comprises means for storing unvoiced phonetic information from a number of training utterances which are respectively associated with said mouth positions.

5. Apparatus as defined by claim 2, wherein said means for storing unvoiced phonetic information representations comprises means for, storing unvoiced phonetic information from a number of training utterances which are respectively associated with said mouth positions.

6. Apparatus as defined by claim 1, further comprising means responsive to the determined representations for producing video mouth positions in conjunction with the sound track.

7. Apparatus as defined by claim 3, further comprising means responsive to the determined representations for producing video mouth positions in conjunction with the sound track.

8. Apparatus as defined by claim 5, further comprising means responsive to the determined representations for producing video mouth positions in conjunction with the sound track.

9. A method for producing lip-synching for a spoken sound track, comprising the steps of:

storing, for each of a number of mouth positions, an unvoiced phonetic information representation associated with a mouth position;

generating, in response to samples of sound from the sound track, unvoiced phonetic information signals for the samples;

comparing the unvoiced phonetic information signals with the stored unvoiced phonetic information representations to determine which of the stored representations most closely matches the unvoiced phonetic information signals; and

storing the determined representations in time correspondence with the sound track.

10. The method as defined by claim 9, wherein said generating step comprises generating said unvoiced phonetic information signals as linear predictive coding signals representative of a vocal tract filter.

11. The method as defined by claim 9, wherein the step of storing unvoiced phonetic information representations comprises storing unvoiced phonetic information from a number of training utterances which are respectively associated with said mouth positions.

12. The method as defined by claim 10, wherein the step of storing unvoiced phonetic information representations comprises storing unvoiced phonetic information from a number of training utterances which are respectively associated with said mouth positions.

13. The method as defined by claim 9, further comprising the step of producing, in response to the determined representations, video mouth positions in conjunction with the sound track.

14. The method as defined by claim 10, further comprising the step of producing, in response to the determined representations, video mouth positions in conjunction with the sound track.

15. The method as defined by claim 12, further comprising the step of producing, in response to the determined representations, video mouth positions in conjunction with the sound track.

16. A method for receiving input training utterances for each of a number of different mouth positions, and for subsequently lip-synching a spoken sound track, comprising the steps of:

(a) storing, for each training utterance, an unvoiced phonetic information representation of a sample of sound from the utterance;

(b) generating, for a sample of the sound track, unvoiced phonetic information signals;

(c) comparing the unvoiced phonetic information signals with the stored unvoiced phonetic information representation for each training utterance, and determining which training utterance provides the best match;

(d) storing, in conjunction with the time of the sample of the soundtrack, an indication of the mouth position corresponding to the training utterance which provided the best match; and

repeating steps (b), (c) and (d) for subsequent samples of the sound track.

17. The method as defined by claim 16, wherein said generating step (b) comprises generating said unvoiced phonetic information signals as linear predictive coding signals representative of a vocal tract filter.

18. The method as defined by claim 16, further comprising the step of producing, in response to the determined indications of step (d), video mouth positions in conjunction with the sound track.

19. The method as defined by claim 17, further comprising the step of producing, in response to the determined indications of step (d), video mouth positions in conjunction with the sound track.