Automated speech alignment for image synthesis

• US 5,884,267 A
• Filed: 02/24/1997
• Issued: 03/16/1999
• Est. Priority Date: 02/24/1997
• Status: Expired due to Term

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Digital Equipment Corporation (HP Inc.)
• Inventors: Van Thong, Jean-Manuel, Waters, Keith, Goldenthal, William D.
• Primary Examiner(s): Hudspeth, David R.
• Assistant Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US08/804,761
• Time in Patent Office: 750 Days
• Field of Search: 764/200, 764/276, 764/243, 764/245, 764/254, 764/260, 764/235, 764/250, 370/389, 370/392, 379/90.01, 379/93.01, 704/270
• US Class Current: 704/270.1
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

In a computerized method, speech signals are analyzed using statistical trajectory modeling to produce time aligned acoustic-phonetic units. There is one acoustic-phonetic unit for each portion of the speech signal determined to be phonetically distinct. The acoustic-phonetic units are translated to corresponding time aligned image units representative of the acoustic-phonetic units. An image including the time aligned image units is displayed. The display of the time aligned image units is synchronized to a replaying of the digitized natural speech signal.

Claims

1. A computerized method for synchronizing audio signals to computer generated visual images comprising the steps of:

analyzing an audio signal to produce a stream of time aligned acoustic-phonetic units, there is one acoustic-phonetic unit for each portion of audio signal determined to be phonetically distinct, each acoustic phonetic unit having a starting time and an ending time of the phonetically distinct portion of the audio signal;

translating each acoustic-phonetic unit to a corresponding time aligned image unit representative of the acoustic-phonetic unit; and

displaying an image including the time aligned image units while synchronizing to the audio signal.

2. The method of claim 1 wherein the audio signal is a continuous natural speech signal and wherein the analyzing further includes converting the audio signal to a digitized speech signal before analyzing the continuous natural speech signal.

3. The method of claim 1 wherein the acoustic-phonetic units have variable durations.

4. The method of claim 1 wherein the acoustic-phonetic units can be interpreted as fundamental linguistic elements.

5. The method of claim 1 further comprising the steps of:

partitioning the audio signal into a sequence of frames; and

processing the frames by a pattern classifier and phonetic recognizer.

6. The method of claim 5 further comprising:

applying statistical trajectory models while processing the frames.

7. The method of claim 1 wherein the visemes correspond to facial gestures.

8. The method of claim 1 further comprising the steps of:

acquiring the audio signal by a first client computer system; and

rendering the audio signal and the image in a second client computer system.

9. The method of claim 8 further comprising:

communicating phonetic records between the first and second client computer systems, each phonetic record including an identity of a particular acoustic-phonetic unit, and the starting and ending time of the acoustic phonetic unit.

10. The method of claim 9 further comprising the steps of:

formatting the audio signal in an audio data file; and

appending the phonetic records to the audio data file.

11. The method of claim 9 wherein the first and second client computers are connected by a network, and further comprising:

analyzing the audio signal in a server computer system connected to the network.

12. The method of claim 1 further comprising:

performing the analyzing, translating, and displaying steps synchronously in real-time.

13. A method for synchronizing audio signals to computer generated visual images, the method comprising the steps of:

analyzing an audio signal to produce time aligned acoustic-phonetic units;

translating each one of the time aligned acoustic-phonetic units to a time aligned image unit representative of the each one to produce a stream of time aligned image units; and

displaying the stream of time aligned image units wherein the stream of time aligned image units is synchronized to the audio signal.

14. The method of claim 13, wherein one of a set of distinct acoustic-phonetic units is produced for each portion of the audio signal determined by the analyzing to be phonetically distinct.

15. The method of claim 14, wherein the each one of the time aligned acoustic-phonetic units has a starting time and an ending time.

16. The method of claim 15, wherein the audio signal is a continuous analog natural speech signal.

17. The method of claim 13 wherein the audio signal is a continuous analog natural speech signal and the analyzing includes converting the continuous analog natural speech signal to a digitized speech signal before analysis.

18. The method of claim 13 wherein each one of the acoustic-phonetic units has a variable duration.

19. The method of claim 13 wherein the each one of the acoustic-phonetic units represents a fundamental linguistic element.

20. The method of claim 13 wherein the analyzing further includes:

partitioning the audio signal into a sequence of frames; and

processing the sequence of frames by a pattern classifier and phonetic recognizer.

21. The method of claim 20 wherein the processing includes applying statistical trajectory models to the sequence of frames.

22. The method of claim 20 wherein each one of the time aligned image units corresponds to a facial gesture.

23. The method of claim 22, wherein the facial gesture corresponds to a fundamental linguistic element.

24. The method of claim 13 further comprising the steps of:

acquiring the audio signal from a first client processor; and

playing back the audio signal and rendering the stream of time aligned image units in a second client processor connected to the first client processor, wherein the stream of time aligned image units is synchronized to the audio signal.

25. The method of claim 24 wherein the playing back and rendering further includes communicating phonetic records between the first and second client processors, each phonetic record including an identity of a particular acoustic-phonetic unit, and the starting and ending time of the particular acoustic phonetic unit.

26. The method of claim 25 wherein the acquiring includes formatting the audio signal in an audio data file and appending the phonetic records to the audio data file before the communicating.

27. The method of claim 26 wherein the first and second client processors are connected by a network, and wherein the analyzing is performed on a server processor connected to the network.

28. The method of claim 27 wherein the analyzing, and associating steps are performed synchronously in real-time.

29. The method of claim 28 wherein the acquiring, and playback and rendering steps are further performed synchronously in real-time.

30. A computing device for synchronizing audio signals to computer generated visual images, the computing device comprising:

an audio signal; and

a processor coupled to the audio signal, the processor configured to;

analyze the audio signal to produce time aligned acoustic-phonetic units;

to translate each one of the time aligned acoustic-phonetic units with a time aligned image unit representative of the each one to produce a stream of time aligned image units; and

to display the stream of time aligned image units wherein the stream of time aligned image units is synchronized to the audio signal.

31. The computing device of claim 30, wherein one of a set of distinct acoustic-phonetic units is produced for each portion of the audio signal determined by the analyzing to be phonetically distinct.

32. The computing device of claim 31, wherein the each one of the time aligned acoustic-phonetic units has a starting time and an ending time.

33. The computing device of claim 32, wherein the audio signal is a continuous analog natural speech signal.

34. The computing device of claim 30, wherein the audio signal is a continuous analog natural speech signal and the analyzing includes converting the continuous analog natural speech signal to a digitized speech signal before analysis.

35. The computing device of claim 30, wherein each one of the acoustic-phonetic units has a variable duration.

36. The computing device of claim 30, wherein the each one of the acoustic-phonetic units represents a fundamental linguistic element.

37. The computing device of claim 30, wherein the processor is further configured to partition the audio signal into a sequence of frames and to process the sequence of frames by a pattern classifier and phonetic recognizer.

38. The computing device of claim 37, wherein the processor is further configured to apply statistical trajectory models to the sequence of frames.

39. The computing device of claim 30, wherein each one of the time aligned image units corresponds to a facial gesture.

40. The computing device of claim 39, wherein the facial gesture corresponds to a fundamental linguistic element.

41. The computing device of claim 30 further comprising:

a first client processor; and

a second client processor connected to the first client processor, wherein the first client processor is configured to acquire the audio signal from an input device and to communicate the audio signal to the second client processor and wherein the second client processor is configured to play back the communicated audio signal and render the stream of time aligned image units in the second processor wherein the stream of time aligned image units is synchronized to the communicated audio file.

42. The computing device of claim 41 wherein the first client processor is further configured to communicate phonetic records between the first and the second client processors, each phonetic record including an identity of a particular acoustic-phonetic unit, and the starting and ending time of the particular acoustic phonetic unit.

43. The computing device of claim 42 wherein the first client processor is further configured to format the audio signal in an audio data file and appending the phonetic records to the audio data file before communicating the audio signal to the second client processor.

44. The computing device of claim 43 wherein the first and second client processors are connected by a network, and further comprising a server processor connected to the network, the server processor configured to:

analyze the audio signal acquired from the first client processor to produce time aligned acoustic-phonetic units;

generate a set of phonetic records from the time aligned acoustic-phonetic units;

append the set of phonetic records to the audio data file; and

communicate the audio data file to the second client processor, wherein the second client processor is configured to play back the audio file and render the stream of time aligned image units using the appended set of phonetic records to synchronize the stream of time aligned image units to the audio signal.