Low bit rate audio-visual communication system having integrated perceptual speech and video coding

• US 5,550,581 A
• Filed: 05/31/1995
• Issued: 08/27/1996
• Est. Priority Date: 04/06/1994
• Status: Expired due to Term

• Current Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Zhou, Yong
• Primary Examiner(s): Kostak, Victor R.
• Application Number:
  US08/455,449
• Time in Patent Office: 454 Days
• Field of Search: 348/14, 348/17, 348/20, 348/77, 348/78, 348/484, 348/152
• US Class Current: 348/14.12
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G10L 19/012 Comfort noise or silence coding
• CPC Class Codes: H04N 19/10 using adaptive coding
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/62 by frequency transforming in three dimensions H04N19/63 takes precedence
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

Disclosed is a low bit rate audio and video communication system which employs an integrated encoding system that dynamically allocates available bits among the audio and video signals to be encoded based on the content of the audio and video information and the manner in which the audio and video information will be perceived by a viewer. A dynamic bit allocation and encoding process will evaluate the current content of the audio and video information and allocate the available bits among the audio and video signals to be encoded. In addition, an appropriate audio encoding technique is dynamically selected based on the current content of the audio signal. A face location detection subroutine will detect and model the location of faces in each video frame, in order that the facial regions may be more accurately encoded than other portions of the video frame. A lip motion detection subroutine will detect the location and movement of the lips of a person present in a video scene, in order to determine when a person is speaking and to encode the lip regions more accurately. The audio and video signals generated by a second party to a communication are monitored to determine if the second party is paying attention to the audio and video information transmitted by the first party to the communication.

Claims

1. A method of encoding a first audio and a first video signal in a transmitter of an audio-visual communication system associated with a first party for transmission, utilizing a predetermined number of available bits, to a receiver associated with a second party, said receiver decoding said first audio and video signals for presentation to said second party, said encoding method comprising the steps of:

detecting whether said second party is talking; and

allocating a minimal number of said predetermined number of available bits for encoding said first audio signal if said detecting step determines that said second party is talking.

3. The method of claim 1, wherein said detecting step further comprises the step of:

analyzing a video signal of said second party to determine if said second party is talking.

4. The method of claim 3, wherein the step of determining if said second party is talking further comprises determining if said second party'"'"'s lips are moving, and detecting if there is audio activity while said second party'"'"'s lips are moving.

5. The method of claim 1 further comprising the step of initializing a speaker data base including data on each speaker participating in a videoconference.

6. The method of claim 5 further comprising the step of setting a speaker talking flag in said speaker data base upon determining said second speaker is talking.

7. The method of claim 6, wherein an image of said second party is captured in a video signal having a plurality of frames;

said frames are tested to determine if said second party is talking; and

the speaker talking flag is updated each time a change in the status of whether said second party is talking is detected.

8. The method of claim 1, wherein said first party has a receiver and said second party has a transmitter, and the method further comprises:

detecting whether said first party is talking; and

allocating a minimal number of a predetermined number of bits availabile to the transmitter of said second party for encoding an audio signal to be transmitted from the second party to the first party, if it is detected that the first party is talking.

9. The method of claim 1, wherein said step of detecting further comprises the step of modelling the second speaker'"'"'s face as an ellipse.

10. The method of claim 9, wherein said step of detecting further comprises the step of modelling the second speaker'"'"'s mouth region as a second ellipse located within the first ellipse which models the second speaker'"'"'s face.

11. The method of claim 1, wherein said detecting step comprises the step of differentiating between speech and other background noise by performing a lip motion subroutine to detect lip movement and to confirm that a detected audio signal coincides with lip movement.

2. A method of encoding a first audio and a first video signal in a transmitter of an audio-visual communication system associated with a first party for transmission to a receiver associated with a second party, said receiver decoding said first audio and video signals for presentation to said second party, said encoding method comprising the steps of:

detecting whether said second party is talking; and

allocating a minimal number of said available bits for encoding said first audio signal if said detecting step determines that said second party is talking;

said detecting step comprising the steps of;

analyzing a second audio signal associated with said second party to determine if said second party is generating audio activity;

analyzing a second video signal which includes a representation of said second party, said second video signal including a representation of the face and lips of said second party, said analysis of said second video signal determining if said lips of said second party are moving; and

concluding said second party is talking if said analyzing steps determine that there is audio activity and said lips of said second party are moving.