Sound-synchronized video system

• US 5,608,839 A
• Filed: 06/21/1994
• Issued: 03/04/1997
• Est. Priority Date: 03/18/1994
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Chen, Homer H.
• Primary Examiner(s): Hafiz, Tariq R.
• Application Number:
  US08/263,271
• Time in Patent Office: 987 Days
• Field of Search: 381/36-41, 381/51, 395/2, 395/2.67, 395/2.69, 395/2.81, 395/2.79, 395/2.4, 395/2.44, 353/30
• US Class Current: 704/235
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G11B 2220/90 Tape-like record carriers
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59

Abstract

A stream of unsynchronized audio signal, representing speech and video signal of a speaker, is processed by decoding the signal, memorizing a plurality of visemes corresponding to phonemes in the audio signal, fetching from the plurality visemes corresponding to phonemes in the audio signal, and imparting a synchronism to the video signal and audio signal by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream. According to an embodiment, the fetching step includes fetching visemes of the lip movement. The system is suitable for use in a videophone.

Claims

1. A method of processing a stream of unsynchronized audio and video signals of a speaker, comprising the steps of:

decoding the audio and video signals;

memorizing a plurality of visemes corresponding to phonemes in the decoded audio signal;

fetching visemes corresponding to phonemes in the audio signal; and

imparting a synchronism to the decoded video and audio signals by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.

2. A method as in claim 1, wherein the memorizing step includes continuously updating the plurality of visemes corresponding to phonemes by continuously extracting visemes from the decoded video signal during the decoding of the audio and video signals and updating the memory of the visemes with the extracted visemes during the decoding of the audio and video signals.

3. A method as in claim 1, wherein the fetching step includes fetching visemes of the lip movement.

4. A communication method, comprising the steps of:

transmitting and receiving streams of synchronized video and audio signals at a multiplicity of stations;

at each of said stations, decoding said signals;

at a plurality of said multiplicity of stations, the steps of;

memorizing a plurality of visemes corresponding to phonemes in the decoded audio signal;

fetching from the plurality visemes corresponding to phonemes in the audio signal; and

imparting a synchronism to the video signal and audio signal by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.

5. A method as in claim 4, wherein the memorizing step includes continuously updating the plurality of visemes corresponding to phonemes by continuously extracting visemes from the decoded video signal during the decoding of the audio and video signals and updating the memory of the visemes with the extracted visemes during the decoding of the audio and video signals.

6. A method as in claim 4, wherein the fetching step includes fetching visemes of lip movement.

7. An apparatus for processing a stream of audio and video signals of a speaker, comprising:

a signal decoder responsive to the audio and video signals;

a visime memory responsive to a plurality of visemes corresponding to phonemes in the audio signal from the signal decoder;

a video signal and audio signal synchronizer coupled to said visime memory and responsive to the plurality visemes corresponding to phonemes in the audio signal and to the unsynchronized video signal of the stream so as to impart synchronism of the video signal with corresponding phonemes in the audio signal of the stream.

8. An apparatus as in claim 7, wherein the signal decoder includes an audio output for outputting the audio signal from the stream and a video output for outputting the video signal from the stream.

9. An apparatus as in claim 7, wherein the synchronizer responds to visemes of lip movement.

10. An apparatus as in claim 7, wherein said visime memory is responsive continuously to new visemes corresponding to said phonemes during decoding.

14. An apparatus as in claim 10, wherein said visime memory is responsive continuously to new visemes corresponding to said phonemes during decording.

11. A communication system, comprising:

a plurality of stations each having means for transmitting and receiving video and audio signals;

a communications network linking said stations;

a signal decoder at each of said stations;

at a plurality of said stations;

a visime memory coupled to said decoding means and responsive to a plurality of visemes corresponding to phonemes in the audio signal;

a video signal and audio signal synchronizer coupled to said visime memory and responsive to the plurality visemes corresponding to phonemes in the audio signal and to the unsynchronized video signal of the stream so as to impart synchronism of the video signal with corresponding phonemes in the audio signal of the stream.

12. An apparatus as in claim 11, wherein said signal decoder includes an audio output for outputting the audio signal from the stream and a video output for outputting the video signal from the stream.

13. An apparatus as in claim 11, wherein the synchronizer responds to visemes of lip movement.